Method for detecting part missing through image comparison, equipment inspection method and system, terminal and medium

Abstract

The invention provides a method for detecting part missing through image comparison, an equipment inspection method and system, a terminal and a medium, and the method for detecting part missing through image comparison comprises the steps: extracting each standard part frame based on a template image; extracting each inspection component frame based on the inspection image; performing matching based on each standard component frame and each inspection component frame to confirm whether each component is missing or not; wherein for any standard component frame, the matching of each inspection component frame comprises the following steps: mapping the standard component frame into the inspection image to serve as a mapping component frame; calculating the maximum coincidence degree of the mapping component frame and each inspection component frame; if the standard component frame meets the component overlapping requirement, the component corresponding to the standard component frame is not lost; otherwise, the corresponding part is missing. According to the method, the standard component frames are mapped into the inspection image, so that based on the coincidence degree of each mapping component frame and each inspection component frame, the accuracy degree is relatively high, and a better inspection effect can be achieved.

Description

Technical Field

[0001] This application belongs to the field of computer vision and relates to an equipment inspection technology, particularly to a method for detecting missing components through image comparison, an equipment inspection method, and a system, terminal, and medium. Background Technology

[0002] Equipment inspection refers to comparing the various components of an equipment with their standard state to check for any deficiencies, in order to monitor the equipment's health status. To achieve efficient and accurate automated equipment inspection, inspection images of the equipment in its current state are collected, and computer vision technology is used to compare them with template images in the standard state, thereby realizing equipment inspection.

[0003] Currently, the comparison between inspection images and template images is typically achieved through feature point matching or object detection methods. Feature point matching extracts feature points from both the template and inspection images using feature point extraction algorithms, matching and aligning the two images to compare whether the inspection image is missing relative to the template image. Object detection uses a trained object detection model to extract components from both the template and inspection images and marks them with bounding boxes for comparison. However, feature point matching aligns inspection and template images at the pixel level, lacking semantic information and failing to extract individual components, thus unable to analyze the absence of any components. While object detection can identify and mark components, it cannot automatically match the bounding boxes between the template and inspection images, or the matching accuracy is low, especially when components are densely packed or similar, making matching difficult and resulting in low image comparison accuracy.

[0004] Therefore, there is currently no image comparison method that can achieve high-precision alignment and matching of each component and automatically reflect the missing parts of each component. Summary of the Invention

[0005] The purpose of this application is to provide a method, equipment inspection method, system, terminal and medium for detecting missing components in image comparison, in order to solve the problem that there is no existing inspection method that can achieve high-precision alignment and matching of each component and automatically reflect the missing status of each component.

[0006] In a first aspect, this application provides a method for detecting missing components by comparing corresponding template images and inspection images to detect the missing status of each component, including: extracting standard component boxes based on the template image; extracting inspection component boxes based on the inspection image; and matching each inspection component box based on each standard component box to confirm whether each component is missing; wherein, for any standard component box, matching each inspection component box includes: mapping the standard component box to the inspection image as a mapped component box; calculating the maximum overlap between the mapped component box and each inspection component box; if the maximum overlap meets the component overlap requirement, then the component corresponding to the standard component box is not missing; otherwise, the component corresponding to the standard component box is missing.

[0007] In one embodiment of this application, for any of the standard component frames, mapping the standard component frame to the inspection image as a mapped component frame includes: obtaining a projection mapping relationship from the template image to the inspection image based on the template image and the inspection image; mapping each vertex of the standard component frame to the inspection image based on the projection mapping relationship as each mapped vertex; and obtaining the mapped component frame based on each mapped vertex.

[0008] In one embodiment of this application, the projection mapping relationship is a homography matrix; the step of obtaining the projection mapping relationship from the template image to the inspection image based on the template image and the inspection image includes: obtaining a number of feature points and matching them based on the template image and the inspection image to obtain a number of matching feature point pairs; obtaining each of the standard component boxes, and obtaining the weights of each set of matching feature point pairs based on each of the standard component boxes, so as to filter and weight each set of matching feature point pairs; calculating the homography matrix based on the matching feature point constraint rules and the weights of each set of matching feature point pairs, as the projection mapping relationship from the template image to the inspection image; wherein, the matching feature point constraint rules are: for two feature points in the same set of matching feature point pairs, the feature points in the template image are mapped to the inspection image as mapped feature points, the homogeneous vectors corresponding to the mapped feature points are obtained, and the homogeneous vectors corresponding to the feature points in the inspection image are obtained, wherein the homogeneous vectors corresponding to the mapped feature points and the homogeneous vectors of the feature points in the inspection image are parallel.

[0009] In one embodiment of this application, for any standard component frame, each pair of matching feature points within the frame has the same weight; the method for obtaining the weight of any pair of matching feature points includes: calculating the area of ​​each standard component frame based on each vertex of each standard component frame; obtaining the standard component frame corresponding to the pair of matching feature points, and normalizing it based on the reciprocal of the area of ​​the corresponding standard component frame as the weight of the pair of matching feature points.

[0010] In one embodiment of this application, the maximum overlap is the maximum area overlap rate between the mapped component frame and each of the inspection component frames; the component overlap requirement is that the maximum area overlap rate is not less than a preset overlap threshold; calculating the maximum overlap between the mapped component frame and each of the inspection component frames includes: calculating the area overlap rate between the mapped component frame and each of the inspection component frames based on each vertex of the mapped component frame and each of the inspection component frames; and taking the maximum value of these area overlap rates as the maximum area overlap rate.

[0011] In one embodiment of this application, the method for extracting the bounding boxes of each inspection component is the same as the method for extracting the bounding boxes of each standard component, including: obtaining the bounding boxes of each standard component based on the template image using a trained target detection model; or, including: selecting each component based on the template image using an image annotation tool, and manually adding the corresponding name of each component to serve as the bounding boxes of each standard component.

[0012] Secondly, this application provides a device inspection method, comprising: acquiring at least one set of template images and inspection images based on a preset acquisition location; and for any set of template images and inspection images, performing the image comparison detection method for missing components as described above.

[0013] Thirdly, this application provides an image comparison system for detecting missing components, used to compare corresponding template images and inspection images to detect the missing status of each component, including a component box extraction module and a component box matching module; the component box extraction module is used to extract standard component boxes based on the template image; and extract inspection component boxes based on the inspection image; the component box matching module is used to match each inspection component box based on each standard component box to confirm whether each component is missing; wherein, for any standard component box, matching each inspection component box includes: mapping the standard component box to the inspection image as a mapped component box; calculating the maximum overlap between the mapped component box and each inspection component box; if the maximum overlap meets the component overlap requirement, then the component corresponding to the standard component box is not missing; otherwise, the component corresponding to the standard component box is missing.

[0014] Fourthly, this application provides a terminal, including: a processor and a memory, wherein the memory and the processor are communicatively connected;

[0015] The memory is used to store computer programs, and the processor is used to execute the computer programs stored in the memory, so that the terminal performs the image comparison detection component missing method or the equipment inspection method as described above.

[0016] Fifthly, this application provides a computer storage medium storing a computer program, which, when executed by a processor, implements the image comparison detection component missing method or equipment inspection method as described above.

[0017] As described above, this application provides a method for detecting missing components through image comparison, an equipment inspection method, and a system, terminal, and medium. By mapping standard component frames to inspection images, it is possible to analyze whether each component is missing based on the degree of overlap between the mapped component frames and the inspection component frames. This method not only has high accuracy but also allows for the correspondence between components in the template image and components in the inspection image, and marks the missing components. This enables staff to quickly and efficiently obtain information on missing equipment components, complete equipment inspections, improve inspection efficiency, and achieve better inspection results. Attached Figure Description

[0018] Figure 1 The diagram shown is a flowchart illustrating a method for detecting missing components in image comparison according to an embodiment of this application.

[0019] Figure 2 The diagram shown is a flowchart illustrating another method for detecting missing components in image comparison according to an embodiment of this application.

[0020] Figure 3 The diagram shows a flowchart illustrating a matching method between a standard component frame and an inspection component frame as described in an embodiment of this application.

[0021] Figure 4 The diagram shown is a flowchart illustrating a method for obtaining a homography matrix as described in an embodiment of this application.

[0022] Figure 5 The diagram shown is a flowchart illustrating a method for obtaining weights of matching feature points as described in an embodiment of this application.

[0023] Figure 6 The diagram shown is a flowchart illustrating a standard component frame mapping method as described in an embodiment of this application.

[0024] Figure 7The diagram shown is a flowchart illustrating a method for obtaining the maximum area overlap rate as described in an embodiment of this application.

[0025] Figure 8 The diagram shown is a flowchart illustrating an equipment inspection method according to an embodiment of this application.

[0026] Figure 9 The diagram shown is a structural schematic of an image comparison and detection component missing system according to an embodiment of this application.

[0027] Figure 10 The diagram shown is a structural schematic of a terminal as described in an embodiment of this application.

[0028] Explanation of reference numerals in the attached figures

[0029] 31: Component box extraction module; 32: Component box matching module; 40: Terminal; 41: Processor; 42: Memory; 421: Operating system; 422: Application program; 43: User interface; 44: Network interface; 45: Bus system. Detailed Implementation

[0030] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0031] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. Therefore, the drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0032] Existing equipment inspection methods often involve collecting images of the equipment in its standard and current states, and then comparing these images using computer vision technology to analyze whether any parts are missing. However, current computer vision technology is not yet capable of performing high-precision matching of individual components in the images from the standard and current states. This makes analyzing the missing components difficult and less accurate, thus affecting the effectiveness of equipment inspection.

[0033] To address the technical problems existing in the prior art, the following embodiments of this application provide a method for detecting missing components through image comparison, an equipment inspection method, and a system, terminal, and medium. First, each component is extracted from the template image and the inspection image, and each component in the template image is mapped to the inspection image to analyze whether each component is missing, so as to obtain the missing component status of each component of the current equipment relative to the standard state. The accuracy is high, thereby effectively improving the reliability and practicality of the equipment inspection results.

[0034] The following embodiments of this application provide a method for detecting missing components through image comparison, an equipment inspection method, and a system, terminal, and medium, including but not limited to applications in inspection scenarios such as industrial equipment, power systems, rail systems, photovoltaic arrays, and warehouse goods. This application does not specifically limit these applications. The following description will use industrial equipment inspection as an example.

[0035] The technical solutions in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0036] like Figure 1 and 2 As shown, this embodiment provides a method for detecting missing components by comparing images acquired during equipment inspection, thereby analyzing the missing status of various components in the equipment based on the acquired images. Specifically, as... Figure 2 As shown, methods for detecting missing parts through image comparison include:

[0037] S1: Extract the bounding boxes of each standard component based on the template image; extract the bounding boxes of each inspected component based on the inspection image.

[0038] The template image is an image captured when the device is in its standard state, such as an image captured during initial use or before use after the device has completed its pre-use preparation. It should be noted that a single captured template image covers all components of the device. For example, if a single template image is captured, that image covers all components of the device; or, if multiple template images are captured, such as one from the left side and one from the right side of the device, then all components in these template images cover all components of the device.

[0039] The inspection images are images captured during the equipment inspection process and are used to represent the current state of the equipment. It should be noted that the equipment is inspected once every preset inspection time interval; that is, images of the equipment are captured at preset inspection time intervals to serve as inspection images. Those skilled in the art should know how to set the specific inspection time intervals; this embodiment does not impose specific limitations here.

[0040] For example, images can be acquired using an aircraft or a ground-based mobile robot equipped with a camera or video camera.

[0041] Furthermore, for large or complex equipment, the equipment can be divided into multiple areas, and the presence of missing components in each area can be checked separately. This avoids the problem of too many or too complex components appearing in a single image, thus improving the accuracy of equipment inspection. Specifically, an aircraft or ground-based mobile robot equipped with a camera moves along a preset inspection trajectory and captures images at each preset navigation point to obtain template images or inspection images. The template image corresponding to each area covers all components within that area, and each preset navigation point is a location on the preset inspection trajectory. The image captured at each navigation point corresponds to the equipment image for that area. It should be noted that those skilled in the art can design specific inspection trajectories and navigation points based on actual conditions; this embodiment does not impose specific limitations.

[0042] Furthermore, when performing image comparison, the template image and inspection image corresponding to the same area of ​​the equipment are used as corresponding images for comparison of each component.

[0043] It should be noted that, in order to compare the various components, detection boxes for each component are extracted from the image. For example, the detection boxes are usually rectangular boxes, and the component names of each component are labeled, such as screws, gears, etc., to increase the semantic information of each detection box, so as to facilitate the subsequent matching and acquisition of the missing components.

[0044] Specifically, based on the template image, the bounding boxes of each standard component are extracted, and based on the inspection image, the bounding boxes of each inspected component are extracted.

[0045] Among them, each standard component bounding box is used to represent the detection box corresponding to each component extracted from the template image; each inspection component bounding box is used to represent the detection box corresponding to each component extracted from the inspection image.

[0046] In some optional implementations, the methods for extracting components from inspection images and extracting components from template images are the same. The following explanation will use the extraction of components from template images as an example.

[0047] For example, a trained object detection model is used to extract each component from the template image and label the component names to identify and add semantic information. The object detection model can be any one of the trained Faster R-CNN, Mask R-CNN, YOLO series, YOLO-Nano, SSD, RetinaNet, DETR, Deformable DETR, or MobileNet-SSD; this embodiment does not impose specific limitations.

[0048] Furthermore, the training method of the object detection model includes: acquiring images of each component labeled as training samples, inputting them into the object detection model to be trained for training, so that the object detection model can extract each component, and based on the labeling of these samples, providing feedback on the extraction accuracy of the object detection model, and optimizing the object detection model.

[0049] Training an object detection model requires a large number of samples. For example, each image is labeled with its components using the labelImg software to serve as training samples.

[0050] In some alternative implementations, when the number of samples is small and the target detection model has not been trained, for example, when only a few inspection images are collected after obtaining the template image, the labelImg software is used to extract each component from the template image or inspection image, and the corresponding names of each component, such as screws and gears, are manually added by the staff to increase semantic information.

[0051] Based on this, using template images and inspection images, the components of the equipment in both standard and current states are identified, so that subsequent analysis of the equipment's component missingness can be performed based on each component.

[0052] S2, based on each standard component frame, matches each inspection component frame to confirm whether any component is missing.

[0053] For any standard component frame, each inspection component frame is matched. If the standard component frame cannot be matched with any of the inspection component frames, then the component corresponding to the standard component frame is missing.

[0054] Based on this, the missing parts of each component can be obtained, enabling equipment inspection.

[0055] Specifically, such as Figure 3 As shown, for any standard component frame, the methods for matching it with each inspection component frame include:

[0056] S10: Map the standard component frame to the inspection image as the mapped component frame.

[0057] Mapping refers to the process of obtaining the position of the standard component bounding box in the inspection image based on the projection mapping relationship between the template image and the inspection image. It should be noted that since the corresponding template image and inspection image are images collected by the same or similar aircraft or ground mobile robots based on the same navigation point, their corresponding actual spatial ranges are the same. That is, the actual spatial ranges of the template image and the inspection image correspond to each other. Based on this, each mapped component bounding box can be mapped into the inspection image.

[0058] Specifically, based on the spatial geometric relationship between the template image and the inspection image, the projection mapping relationship from the template image to the inspection image is obtained, so as to map the standard component box into the inspection image, and based on the mapped component box, the actual situation of the component corresponding to the standard component box can be analyzed.

[0059] In some optional implementations, the correspondence between the actual spatial ranges of the template image and the standard component image is used as a projection mapping relationship to project the standard component frame. Specifically, based on the position of the standard component frame in the template image, its position in the actual spatial range is obtained, and further based on its position in the actual spatial range, it is mapped onto the inspection image to obtain the corresponding mapped component frame.

[0060] In some alternative implementations, the projection mapping relationship from the template image to the inspection image is a homography matrix between the template image and the inspection image. The homography matrix is ​​a 3x3 matrix. Specifically, as shown... Figure 4 As shown, the method for obtaining the homography matrix between the template image and the inspection image includes:

[0061] S11: Based on the template image and the inspection image, obtain several feature points and perform matching to obtain several sets of matching feature point pairs.

[0062] Specifically, a feature point matching model is used to extract and match feature points from the template image and the inspection image, and these matching feature points are used as matching feature point pairs.

[0063] For example, the feature point matching model includes SuperPoint (feature extractor) and LightGlue (feature matcher). SuperPoint is used to extract several feature points from the template image and the inspection image respectively, and LightGlue is used to match the feature points of the template image with the feature points of the inspection image one by one to form matching feature point pairs. Specifically, those skilled in the art should know the working principle and specific settings of the feature point matching model including SuperPoint and LightGlue, which will not be specifically explained in this embodiment.

[0064] Furthermore, for each matching feature point pair, the RANSAC algorithm is used for filtering. Based on the principle that the projection mapping relationship between each matching feature point pair should be consistent, matching feature point pairs with erroneous matching relationships are removed, thereby improving the quality of the matching feature point pairs and enhancing the accuracy of the subsequently obtained projection mapping relationship. For example, in the RANSAC algorithm, the reprojection error threshold is set to 3 pixels, the maximum number of iterations is set to 3000, and the confidence threshold is set to 0.995.

[0065] S12, obtain the standard component bounding boxes, and based on the standard component bounding boxes, obtain the weights of each set of matching feature point pairs.

[0066] The method for obtaining each standard component frame is described in step S1 above, and will not be repeated here in this embodiment.

[0067] The weights are used to filter whether each set of matching feature points is located inside each standard component box, and to characterize the importance of the standard component boxes corresponding to each set of matching feature points.

[0068] Specifically, based on the vertex positions of each standard component box, feature points located on the template image in each set of matching feature points are filtered. If a feature point on the template image is located inside any standard component box, the feature point is retained; otherwise, it is deleted. Simultaneously, weight values ​​are set for each set of matching feature points based on their importance within their corresponding standard component boxes. For ease of understanding by those skilled in the art, an image coordinate system is constructed with the pixel in the first row and first column of the top-left corner of the image as the origin, the horizontal direction to the right as the positive x-axis, and the vertical direction downwards as the positive y-axis. Please refer to the following formula:

[0069]

[0070]

[0071] in, For the first Feature points located on the template image in a group matching feature point pair; For the first Weights corresponding to group-matched feature point pairs; Used to characterize the The importance of each standard component frame; This is used to filter whether feature points on a template image are inside any standard component bounding box. Specifically, For the first The minimum x-coordinate of a standard component frame For the first The maximum x-coordinate of a standard component frame. For the first The minimum y-coordinate of a standard component frame. For the first The maximum y-coordinate of a standard component frame; This represents the total number of standard component frames.

[0072] Furthermore, the importance of each standard component box can be set based on the category confidence score output by the object detection model or the position of each standard component box.

[0073] In some optional implementations, the importance of each standard component frame is determined by the size of its area. This is because, when calculating the projection mapping relationship from the template image to the inspection image, standard component frames with smaller areas are more prone to shifting. If a standard component frame with a smaller area does not shift after mapping, then a standard component frame with a larger area will also not shift after mapping, thereby improving the accuracy of the obtained projection mapping relationship and thus improving the accuracy of subsequent analysis of component missingness.

[0074] Specifically, please see Figure 5 The importance of each standard component bounding box is determined by its area. Based on this, the weights of each set of matching feature point pairs are obtained through the following methods:

[0075] S121, Calculate the area of ​​each standard component frame based on each vertex of each standard component frame.

[0076] Specifically, for the calculation method of the area of ​​each standard component frame, please refer to the following formula:

[0077]

[0078] in, For the first The area value of a standard component frame.

[0079] S122, obtain the standard component boxes corresponding to the matching feature point pairs, and normalize them based on the reciprocal of the area of ​​the corresponding standard component boxes, which is used as the weight of the matching feature point pairs.

[0080] Since a smaller standard component box area reflects the accuracy of the projection mapping relationship more accurately, i.e., a higher importance, in this embodiment, the weight is obtained based on the reciprocal of the area of ​​the corresponding standard component box for the current matching feature point. Of course, in some alternative implementations, the weight can also be obtained based on a power of the reciprocal of the area of ​​the corresponding standard component box; this embodiment does not specifically limit this.

[0081] Specifically, please refer to the aforementioned method for filtering whether feature points on a template image are inside any standard component frame. Based on this This allows us to obtain the standard component bounding box corresponding to the matching feature point pair, which will not be elaborated upon in this embodiment.

[0082] Furthermore, to avoid the weight values ​​being affected by the size of the area of ​​each standard component frame, becoming too large or too small, which would affect subsequent calculations, normalization was also performed in this embodiment.

[0083] Specifically, please refer to the following formula, where the weights of each pair of matched feature points are:

[0084]

[0085] in, For the first The reciprocal of the area of ​​a standard component frame. ; The average of the reciprocals of the areas of all standard component frames. .

[0086] S13. Based on the constraint rules of matching feature points and the weights of each pair of matching feature points, calculate the homography matrix as the projection mapping relationship from the template image to the inspection image.

[0087] The constraint rule for matching feature points is as follows: for two feature points in the same set of matching feature point pairs, the feature points in the template image are mapped to the inspection image as mapped feature points. The homogeneous vectors corresponding to the mapped feature points are obtained, and the homogeneous vectors of the corresponding feature points in the inspection image are also obtained. The homogeneous vectors corresponding to the mapped feature points are parallel to the homogeneous vectors of the feature points in the inspection image.

[0088] It should be noted that the process of mapping feature points from the template image to the inspection image is actually a coordinate transformation process using the homography matrix. Based on this, and since mathematical laws dictate that the cross product of two parallel vectors is 0, a coordinate system is constructed with the pixel in the first row and first column of the top-left corner of the image as the origin, the horizontal direction to the right as the positive x-axis, and the vertical direction downwards as the positive y-axis. The constraint law for matching feature points can be expressed as follows:

[0089]

[0090] in, It is a homography matrix, a 3x3 matrix; The homogeneous vector used to characterize the feature points located in the template image in the matching feature point pair, specifically, the coordinates of the feature points located in the template image are... Then its corresponding homogeneous coordinates are This yields the homogeneous vectors of the feature points located in the template image; The homogeneous vector used to characterize the feature points located in the inspection image in the i-th set of matched feature point pairs is, as mentioned earlier, the coordinates of the feature points located in the inspection image are... Then its corresponding homogeneous coordinates are This yields the homogeneous vectors of the feature points located in the inspection image.

[0091] Based on this, the homography matrix can be represented as: Therefore, for each pair of matched feature points, there exists a linear constraint:

[0092]

[0093]

[0094]

[0095] in, This is used to characterize the vectorization of the homography matrix to obtain the corresponding one-dimensional vector.

[0096] Based on the aforementioned matching feature point constraints, and combining the coordinates of each set of matching feature points, the homography matrix can be calculated to represent the projection mapping relationship from the template image to the inspection image. Specifically, please refer to the following formula:

[0097]

[0098]

[0099] in, The total number of sets of matching feature point pairs.

[0100] It should be noted that, in this embodiment, in order to improve the accuracy of the obtained projection mapping relationship, a weighted processing is performed based on the importance of each standard component box to calculate the homography matrix.

[0101] For example, a weight matrix is ​​constructed by assigning weights to each standard component frame based on its area:

[0102]

[0103] in, For the first Weights corresponding to group-matched feature point pairs; Used to characterize the construction of diagonal matrices.

[0104] Based on the weight matrix, the weighted processing yields:

[0105]

[0106] Furthermore, the singular value decomposition method is used to solve for the homography matrix. Specifically, for Decompose:

[0107]

[0108] in, It is a left singular vector matrix; It is a right singular vector matrix; It is a singular value matrix.

[0109] For example, in order to avoid The meaningless solution, set constraints. At this point, the right singular vector matrix... The last column is used as a one-dimensional vector. That is, the optimal solution is,

[0110]

[0111] in, Used to characterize a one-dimensional vector The optimal solution; Used to characterize constraints At that time, obtaining makes The solution that yields the minimum value; Used to characterize the right singular vector matrix The last column.

[0112] Based on the obtained optimal solution This is converted to a 3x3 matrix, and norm normalization is performed to obtain the homography matrix, i.e.:

[0113]

[0114]

[0115] in, Used to characterize the optimal solution Restored to a 3x3 matrix, which is equivalent to The inverse operation; Norm values ​​are used to characterize matrices; Used to characterize the assignment update, that is, the homography matrix after norm normalization is used as the final homography matrix.

[0116] Based on the above steps S11 to S13, the homography matrix can be obtained. This serves as the projection mapping relationship between the template image and the inspection image.

[0117] Furthermore, based on the acquired projection mapping relationship, the standard component boxes are mapped onto the inspection image as mapped component boxes, so as to facilitate the analysis of the actual condition of each component based on the mapped component boxes. In some optional implementations, such as Figure 6 As shown, the mapping process for any standard component box includes:

[0118] S101, Based on the template image and the inspection image, obtain the projection mapping relationship from the template image to the inspection image.

[0119] Specifically, please refer to the aforementioned steps S11 to S13, which will not be repeated here in this embodiment.

[0120] S102, based on the projection mapping relationship, maps each vertex of the standard component box to the inspection image as each mapped vertex.

[0121] Specifically, each vertex of the standard component frame is obtained, and each vertex is mapped based on the projection mapping relationship to obtain each mapped vertex.

[0122] For example, in this embodiment, the standard component frame is a rectangle with four vertices. The coordinates of these four vertices are obtained, and the four vertices are mapped based on the homography matrix. Specifically, the image coordinate system is constructed with the pixel in the first row and first column of the top left corner of the image as the origin, the horizontal direction to the right as the positive x-direction, and the vertical downward direction as the positive y-axis. If the coordinates of the four vertices of the standard component frame are:

[0123]

[0124] in, Used to characterize the top left corner of the standard component frame; Used to characterize the upper right corner of this standard component frame; Used to characterize the lower right corner of this standard component frame; Used to characterize the lower left corner of this standard component frame; The minimum x-coordinate of the standard component frame. This represents the maximum x-coordinate of the standard component frame. This represents the minimum y-coordinate of the standard component frame. This represents the maximum y-coordinate of the standard component frame.

[0125] Convert the coordinates of each vertex to homogeneous coordinates:

[0126]

[0127] in, Used to characterize the One vertex, Used to characterize the The x-coordinates of each vertex. Used to characterize the The y-coordinate values ​​of each vertex, for example... of for , of for , of for , of for And so on.

[0128] Based on the homography matrix obtained in steps S11 to S13 Perform mapping:

[0129]

[0130] in, Used to characterize the Homogeneous coordinates after mapping each vertex Used to characterize the The x-coordinate value of the homogeneous coordinates after mapping Used to characterize the The y-coordinate value of the homogeneous coordinates after mapping. Used to characterize the The scaling factor of homogeneous coordinates after mapping.

[0131] Based on this, the homogeneous coordinates of the four vertices of the standard component box mapped to the inspection image are obtained. To facilitate determining the position of each vertex in the inspection image for subsequent operations, the homogeneous coordinates are restored to the coordinates of the image coordinate system.

[0132]

[0133] in, Used to characterize the The coordinates of each vertex after mapping to the image coordinate system; Used to characterize the The x-coordinate values ​​of each vertex after mapping to the image coordinate system; Used to characterize the Each vertex is mapped to its y-coordinate value in the image coordinate system.

[0134] S103, obtain the mapping component box based on each mapping vertex.

[0135] Specifically, the rectangle formed by these vertices is identified based on the vertices, which is the mapped component box.

[0136] Perform the mapping process of steps S101 to S103 above on each standard component frame to map each standard component frame into the inspection image.

[0137] S20, calculate the maximum overlap between the mapped component frame and each inspected component frame; if the maximum overlap meets the component overlap requirements, then the component corresponding to the standard component frame is not missing; otherwise, the component corresponding to the standard component frame is missing.

[0138] The maximum overlap refers to the maximum degree of overlap between the mapped component frame and each inspected component frame.

[0139] The degree of overlap between the mapped component frame and each inspection component frame is used to reflect the size of the overlapping area between the mapped component frame and each inspection component frame. For example, the degree of overlap between the mapped component frame and each inspection component frame can be any one of the overlapping area value, area overlap rate, and center point distance value.

[0140] When the overlap between the mapped component frame and any inspected component frame meets the component overlap requirement, the inspected component frame is the component image of the corresponding component in the inspection image. Based on this, the maximum overlap between the mapped component frame and each inspected component frame is obtained. When it meets the component overlap requirement, the component corresponding to the mapped component frame is not missing, and its corresponding position in the inspection image is the inspected component frame with the highest overlap with the mapped component frame. When it does not meet the component overlap requirement, the component corresponding to the mapped component frame is missing, and the standard component frame corresponding to the mapped component frame is marked in the template image to show the missing component in the standard component frame.

[0141] Based on this, this embodiment maps standard component frames to inspection images. Based on the overlap between each mapped component frame and each inspection component frame, it analyzes whether each component is missing. This not only has a high degree of accuracy, but also finds the component positions in the template image and marks the missing components in the template image. This makes it easier for staff to inspect the equipment, improves inspection efficiency, and achieves better inspection results.

[0142] In some optional implementations, the area overlap rate is used to characterize the degree of overlap between the mapped component frame and each inspection component frame. That is, the maximum overlap rate is the maximum area overlap rate between the mapped component frame and each inspection component frame. In this case, the component overlap requirement is set to the maximum area overlap rate not being less than a preset overlap threshold. For example, the preset overlap threshold is 0.5. Of course, those skilled in the art can also set the specific data of the overlap threshold according to actual needs. This embodiment does not make specific limitations here.

[0143] Based on this, such as Figure 7 As shown, for any mapped component frame, the maximum overlap between it and each inspected component frame, i.e., the maximum area overlap rate, is obtained through the following methods:

[0144] S21. Based on each vertex of the mapped component box and each vertex of each inspected component box, calculate the area overlap rate between the mapped component box and each inspected component box.

[0145] Among them, the area overlap rate between the mapped component frame and each inspection component frame is the area intersection-union ratio between the mapped component frame and each inspection component frame.

[0146] Specifically, for any mapped component frame and any inspected component frame, please refer to the following formula for calculating their area overlap rate:

[0147]

[0148]

[0149]

[0150] in, Used to characterize the mapped component box; Used to characterize the inspection component frame; Used to characterize the maximum x-coordinate of the mapped component box; Used to characterize the maximum x-coordinate of the inspection component frame; Used to characterize the minimum x-coordinate of the mapped component box; The minimum x-coordinate of the inspected component frame is used to characterize the inspected component frame. Used to characterize the maximum y-coordinate of the mapped component box; Used to characterize the maximum y-coordinate of the inspection component frame; Used to characterize the minimum y-coordinate of the mapped component box; The minimum y-coordinate of the inspected component frame is used to characterize the component frame. Used to represent taking the maximum value; Used to represent taking the minimum value; This is used to characterize the area overlap ratio of the mapped component frame and the inspected component frame, i.e., the area coincidence rate.

[0151] Calculate the area intersection ratio between the current mapped component frame and each inspected component frame, and use it as the area overlap rate between the mapped component frame and each inspected component frame.

[0152] S22, take the maximum value of these area overlap rates as the maximum area overlap rate.

[0153] Specifically, the maximum value of each area overlap rate is obtained and used as the maximum area overlap rate.

[0154] If the maximum area overlap rate is not less than the preset overlap threshold, then the component corresponding to the mapped component box is not missing, and the component image in the inspection component box corresponding to the maximum area overlap rate is used as the component image corresponding to the component in the standard component box of the mapped component box in the inspection image; if the maximum area overlap rate is less than the preset overlap threshold, then the component corresponding to the mapped component box is missing, and the component in the standard component box in the template image is marked as missing.

[0155] Based on this, the missing parts of the equipment during inspection can be accurately analyzed, and the missing parts can be marked in the template image and matched with the missing parts in the inspection image, so as to achieve better inspection results.

[0156] On the other hand, this application also provides an equipment inspection method to achieve accurate and efficient equipment inspection by utilizing the aforementioned image comparison method for detecting missing components. Specifically, as... Figure 8 As shown, the equipment inspection methods include:

[0157] S1': Based on the preset acquisition location, acquire at least one set of template images and inspection images.

[0158] As mentioned earlier, the template image is an image captured by the device in its standard state, while the inspection image is an image captured by the device during the inspection process. For example, an aircraft or ground mobile robot equipped with a camera or video camera is used to perform inspections along a preset inspection trajectory at preset inspection time intervals to collect images for use as inspection images. Those skilled in the art should know how to set specific inspection time intervals; this embodiment does not impose specific limitations here.

[0159] Furthermore, for large or complex equipment, the equipment is divided into multiple areas, and images are acquired from each area separately. For example, for aircraft or ground mobile robots, images are acquired at each preset navigation point to obtain template images or inspection images corresponding to each area. In this case, the template images and inspection images in the same group represent images acquired from the same area of ​​the equipment, thus improving the accuracy of equipment inspection.

[0160] Based on this, by comparing the collected inspection images and template images, the missing parts of each component of the equipment can be analyzed.

[0161] S2', for any set of template images and inspection images, perform the image comparison detection method for missing components as described above, in order to analyze the missing status of each component in the equipment.

[0162] Specifically, for each set of template images and inspection images, the image comparison method for detecting missing components, as described above, is executed to analyze the missing status of each component in each area of ​​the equipment, thereby achieving equipment inspection. For details on image comparison between the template images and inspection images, please refer to the aforementioned method for detecting missing components through image comparison; this embodiment will not repeat it here.

[0163] Based on this, this embodiment collects inspection images and template images of the equipment and uses image comparison to detect missing parts. This enables the analysis of missing parts of the equipment, thereby timely detection of missing parts, identification of potential hazards, and prevention of equipment malfunctions and dangers. This not only ensures the safety of personnel and assets but also improves the operational reliability and production continuity of the equipment.

[0164] Furthermore, to facilitate understanding of the equipment inspection method described in this application by those skilled in the art, a specific example will be given below for illustration.

[0165] For example, for urban rail transit trains such as subways, light rails, and suburban railways, it is necessary to inspect the train's components to prevent wear and tear on components during long-term operation from threatening the safety of train operation. Specifically, before the train departs, a wheeled robot will collect template images of the train's undercarriage and both sides based on preset navigation points to cover various components such as wheels, bolts, bearings, and gears on the train's undercarriage.

[0166] During the daily train return-to-depot maintenance window, the wheeled robot automatically navigates to the undercarriage and sides of the urban rail train according to preset navigation points to collect various inspection images. Based on these inspection images, it compares them with corresponding template images to match the components in the inspection images with the components in the template images, and marks the missing components in the template images. For details on the comparison method, please refer to the aforementioned content on the method of detecting missing components by image comparison, which will not be repeated here.

[0167] By comparing images to detect missing components and using the results of train inspections, the status of missing components on the train can be obtained. This helps prevent trains with missing components from departing the next day, reminds staff to repair the trains in a timely manner, and effectively improves the safety and reliability of train operations.

[0168] like Figure 9 As shown, this embodiment provides an image comparison and detection system for missing components, including a component box extraction module 31 and a component box matching module 32.

[0169] Among them, the component box extraction module 31 is used to extract the standard component boxes based on the template image and to extract the inspection component boxes based on the inspection image.

[0170] The component frame matching module 32 is used to match each inspection component frame with each standard component frame to confirm whether any component is missing.

[0171] For any standard component frame, the process involves matching each inspected component frame, including:

[0172] Map the standard component bounding box to the inspection image to obtain the mapped component bounding box;

[0173] Calculate the maximum overlap between the mapped component frame and each inspected component frame; if the maximum overlap meets the component overlap requirements, then the component corresponding to the standard component frame is not missing; otherwise, the component corresponding to the standard component frame is missing.

[0174] Based on the same technical concept, the image comparison detection component missing method or equipment inspection method provided in the embodiments of the present invention can be implemented on the terminal side or the server side.

[0175] like Figure 10 The diagram shown illustrates an optional hardware structure of a terminal according to an embodiment of the present invention. The terminal 40 can be an inspection robot, an aircraft, a mobile phone, a computer device, a tablet device, a personal digital processing device, a factory back-end processing device, etc. The terminal 40 includes at least one processor 41, a memory 42, at least one network interface 44, and a user interface 43. The various components in the device are coupled together via a bus system 45. It is understood that the bus system 45 is used to realize communication between these components. In addition to a data bus, the bus system 45 also includes a power bus, a control bus, and a status signal bus.

[0176] The user interface 43 may include a monitor, keyboard, mouse, trackball, clicker, button, touchpad, or touch screen.

[0177] It is understood that memory 42 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM) or programmable read-only memory (PROM), used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM) and synchronous static random access memory (SSRAM). The memory characterized in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable categories of memory.

[0178] In this embodiment of the invention, the memory 42 is used to store various types of data to support the operation of the terminal. Examples of this data include: any executable program for operation on the terminal 40, such as the operating system 421 and application programs 422; the operating system 421 contains various system programs, such as the framework layer, core library layer, driver layer, etc., for implementing various basic services and handling hardware-based tasks. The application program 422 may contain various applications, such as media players, browsers, etc., for implementing various application services. The method for detecting missing components for image comparison or the device inspection method provided in this embodiment of the invention may be included in the application program 422.

[0179] The methods disclosed in the above embodiments of the present invention can be applied to processor 41, or implemented by processor 41. Processor 41 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 41 or by instructions in the form of software. The processor mentioned above may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 41 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present invention. Processor 41 may be a microprocessor or any conventional processor, etc. The steps of the accessory optimization method provided in the embodiments of the present invention can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium, which is located in a memory. The processor reads the information in the memory and combines it with its hardware to complete the steps of the aforementioned method.

[0180] In an exemplary embodiment, terminal 40 may be used by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), or complex programmable logic devices (CPLDs) to execute the aforementioned method.

[0181] This invention also provides a computer-readable storage medium storing a computer program that, when called by a processor, implements the method for detecting missing image comparison components or the equipment inspection method provided by this invention.

[0182] Computer-readable storage media can be tangible devices capable of holding and storing instructions used by an instruction execution device. Computer-readable storage media can be, for example, (but not limited to) electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, and mechanical encoding devices.

[0183] The computer-readable program represented herein can be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network, to an external computer or external storage device. A network adapter card or network interface in each computing / processing device receives computer-readable program instructions from the network and forwards these instructions to the computer-readable storage medium in the respective computing / processing device.

[0184] In summary, this application uses image comparison to detect missing components by collecting inspection images and template images of the equipment. This method accurately and efficiently obtains information on the missing components of the equipment, enabling timely detection of missing components and preventing equipment from malfunctioning or even posing a danger due to missing components. This effectively ensures the safety of personnel and assets and improves the operational reliability and production continuity of the equipment.

[0185] The descriptions of the processes or structures corresponding to the above figures each have their own emphasis. For parts of a process or structure that are not described in detail, please refer to the relevant descriptions of other processes or structures.

[0186] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A method of image comparison for detecting missing parts of components by comparing a corresponding template image and an inspection image, characterized in that, include: Based on the template image, extract the bounding boxes of each standard component; Based on the inspection images, extract the bounding boxes of each inspection component; Based on each of the standard component frames, each of the inspection component frames is matched to confirm whether any component is missing; Specifically, for any of the standard component frames, matching is performed on each of the inspection component frames, including: The standard component frame is mapped onto the inspection image to form the mapped component frame; Calculate the maximum overlap between the mapped component frame and each of the inspection component frames; if the maximum overlap meets the component overlap requirements, then the component corresponding to the standard component frame is not missing; otherwise, the component corresponding to the standard component frame is missing.

2. The method of claim 1, wherein, For any of the aforementioned standard component frames, mapping the standard component frame to the inspection image as a mapped component frame includes: Based on the template image and the inspection image, obtain the projection mapping relationship from the template image to the inspection image; Based on the projection mapping relationship, each vertex of the standard component frame is mapped to the inspection image, serving as each mapped vertex; Based on each of the mapped vertices, the mapped component box is obtained.

3. The method according to claim 2, characterized in that, The projection mapping relationship is a homography matrix; obtaining the projection mapping relationship from the template image to the inspection image based on the template image and the inspection image includes: Based on the template image and the inspection image, several feature points are obtained and matched to obtain several sets of matching feature point pairs. Obtain each of the standard component frames, and based on each of the standard component frames, obtain the weights of each group of matching feature point pairs, so as to filter and weight each group of matching feature point pairs; Based on the matching feature point constraint rules and the weights of each set of matching feature point pairs, the homography matrix is ​​calculated as the projection mapping relationship from the template image to the inspection image. The constraint rule for the matching feature points is as follows: for two feature points in the same set of matching feature point pairs, the feature points in the template image are mapped to the inspection image as mapped feature points, the homogeneous vectors corresponding to the mapped feature points are obtained, and the homogeneous vectors corresponding to the feature points in the inspection image are obtained. The homogeneous vectors corresponding to the mapped feature points and the homogeneous vectors of the feature points in the inspection image are parallel.

4. The method according to claim 3, characterized in that, For any of the standard component frames, each pair of matching feature points within the frame has the same weight; the weight of any pair of matching feature points is obtained by means of: Calculate the area of ​​each of the standard component frames based on each vertex of each of the standard component frames; Obtain the standard component box corresponding to the matching feature point pair, and normalize it based on the reciprocal of the area of ​​the corresponding standard component box, which is used as the weight of the matching feature point pair.

5. The method according to claim 1, characterized in that, The maximum overlap is the maximum area overlap rate between the mapped component frame and each of the inspection component frames; the component overlap requirement is that the maximum area overlap rate is not less than a preset overlap threshold. The calculation of the maximum overlap between the mapped component frame and each of the inspection component frames includes: Based on each vertex of the mapped component frame and each vertex of each of the inspection component frames, calculate the area overlap rate between the mapped component frame and each of the inspection component frames; The maximum value of these area overlap rates is taken as the maximum area overlap rate.

6. The method according to claim 1, characterized in that, The method for extracting the bounding boxes of each of the inspection components is the same as the method for extracting the bounding boxes of each of the standard components, including: obtaining the bounding boxes of each of the standard components based on the template image and through a trained target detection model; Alternatively, it may include: based on the template image, selecting each component using an image annotation tool, and manually adding the corresponding name for each component to serve as the frame for each of the standard components.

7. A method for equipment inspection, comprising: Based on the preset acquisition locations, acquire at least one set of template images and inspection images; For any set of template images and inspection images, perform the image comparison detection component missing method as described in any one of claims 1 to 6.

8. An image comparison and detection system for missing components, used to compare corresponding template images and inspection images, characterized in that, Includes a component box extraction module and a component box matching module; The component frame extraction module is used to extract standard component frames based on the template image and to extract inspection component frames based on the inspection image. The component frame matching module is used to match each of the inspection component frames based on each of the standard component frames in order to confirm whether each component is missing. Specifically, for any of the standard component frames, matching is performed on each of the inspection component frames, including: The standard component frame is mapped onto the inspection image to form the mapped component frame; Calculate the maximum overlap between the mapped component frame and each of the inspection component frames; if the maximum overlap meets the component overlap requirements, then the component corresponding to the standard component frame is not missing; otherwise, the component corresponding to the standard component frame is missing.

9. A terminal, characterized in that, include: A processor and a memory, wherein the memory and the processor are communicatively connected; The memory is used to store computer programs, and the processor is used to execute the computer programs stored in the memory to cause the terminal to perform the image comparison detection component missing method as described in any one of claims 1 to 6 or the device inspection method as described in claim 7.

10. A computer storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the method for detecting missing image comparison components as described in any one of claims 1 to 6 or the equipment inspection method as described in claim 7.

Images