High-lock bolt detection method, device, electronic equipment and storage medium thereof

Abstract

The present invention discloses a high-lock bolt detection method, device, electronic device and storage medium thereof, including: obtaining a reference image of a high-lock bolt to be detected and an image to be detected, the reference image and the image to be detected respectively reflecting the state of the high-lock bolt to be detected at different times; determining the object-image ratios corresponding to the reference image and the image to be detected respectively based on the imaging focal lengths of the reference image and the image to be detected, the object-image ratio of the image reflecting the difference between the pixel distance in the image and the actual distance; converting the image to be detected and / or the reference image based on the object-image ratios corresponding to the reference image and the image to be detected respectively, obtaining an image pair to be detected, the object-image ratios corresponding to the two images in the image pair to be detected being consistent; determining the deformation of the high-lock bolt to be detected at the time corresponding to the image to be detected relative to the time corresponding to the reference image based on the image pair to be detected. The present invention realizes deformation detection based on image comparison, and realizes detection without direct mechanical contact.

Description

Technical Field

[0001] The present invention relates to the technical field of equipment detection, and in particular to a high-lock bolt detection method, device, electronic equipment and storage medium thereof. Background Art

[0002] In mechanical equipment such as aerospace equipment, railway equipment, and ship equipment, high-lock bolts are one of the most important components. The deformation of high-lock bolts often leads to uncontrollable dangers in mechanical equipment. Therefore, most of the deformation detection needs for high-lock bolts use traditional deformation detection methods. Traditional deformation detection methods mainly use mechanical detection, which relies on physical contact sensors such as strain gauges and pressure gauges to achieve detection. Although mechanical detection can provide relatively stable data, it requires direct mechanical contact with the object being tested, which may cause interference or damage to the object being tested itself, and it is also difficult to detect objects that are difficult to directly contact. Summary of the invention

[0003] In view of the above-mentioned problems, the present invention provides a high-lock bolt detection method, device, electronic device and storage medium thereof, so as to solve the defect of the prior art that direct mechanical contact with the object to be detected is required, and realize detection without direct mechanical contact.

[0004] In a first aspect, the present invention provides a high-lock bolt detection method, comprising:

[0005] Acquire a reference image of a high-lock bolt to be detected and an image to be detected, wherein the reference image and the image to be detected respectively reflect the states of the high-lock bolt to be detected at different times;

[0006] Determine the object-image ratios corresponding to the reference image and the image to be detected based on the imaging focal lengths of the reference image and the image to be detected, respectively, wherein the object-image ratio of the image reflects the difference between the pixel distance in the image and the actual distance;

[0007] Based on the object-image ratios respectively corresponding to the reference image and the image to be detected, the image to be detected and / or the reference image are converted to obtain an image pair to be detected, wherein the object-image ratios corresponding to the two images in the image pair to be detected are consistent;

[0008] Based on the pair of images to be detected, the deformation of the high-locking bolt to be detected at the time corresponding to the image to be detected relative to the time corresponding to the reference image is determined.

[0009] According to the high-lock bolt detection method provided by the present invention, based on the imaging focal lengths of the reference image and the image to be detected, determining the object-image ratios corresponding to the reference image and the image to be detected, respectively, includes:

[0010] Obtaining the object-image ratios corresponding to the reference image and the image to be detected respectively based on a preset formula;

[0011] The preset formula is:

[0012] in, is the object-image ratio corresponding to the image, dis is the distance between the camera and the high-lock bolt to be detected, and is the fitting coefficient corresponding to the imaging focal length of the image.

[0013] According to the high-lock bolt detection method provided by the present invention, the image to be detected and / or the reference image are converted based on the object-image ratios respectively corresponding to the reference image and the image to be detected to obtain the image pair to be detected, including:

[0014] Obtaining a reference object-image ratio, where the reference object-image ratio is the object-image ratio corresponding to the image to be detected, the object-image ratio corresponding to the reference image, or another preset object-image ratio;

[0015] Based on the reference object-image ratio and the object-image ratio of the image to be detected and / or the reference image, the image to be detected and / or the reference image are converted to obtain the image pair to be detected.

[0016] According to the high-lock bolt detection method provided by the present invention, the image to be detected and / or the reference image is converted based on the reference object-image ratio, the object-image ratio of the image to be detected and / or the reference image to obtain the image pair to be detected, including:

[0017] Obtaining a ratio of the object-image ratio of the image to be detected and / or the reference image to the reference object-image ratio;

[0018] The image content of the image to be detected and / or the reference image is scaled based on the ratio to obtain an image in the pair of images to be detected.

[0019] According to the high locking bolt detection method provided by the present invention, the step of determining the deformation of the high locking bolt to be detected at the time corresponding to the image to be detected relative to the time corresponding to the reference image based on the image pair to be detected includes:

[0020] Perform feature point detection on the two images in the image pair to be detected respectively to obtain a plurality of first feature points and a plurality of second feature points;

[0021] Matching the first feature point and the second feature point to obtain a feature point pair;

[0022] Based on the pixel positions of the feature point pairs, determining the deformation of the high-lock bolt to be detected at the time corresponding to the image to be detected relative to the time corresponding to the reference image;

[0023] The image pair to be detected is a first image pair, a second image pair or a third image pair, the first image pair includes two of the conversion images, the second image pair includes the conversion image and the reference image, and the third image pair includes the conversion image and the image to be detected.

[0024] According to the high-lock bolt detection method provided by the present invention, the feature point detection is performed on the two images in the image pair to be detected respectively to obtain a plurality of first feature points and a plurality of second feature points, including:

[0025] Inputting the images in the to-be-detected image pair into a trained feature point detection model respectively, and obtaining a feature point detection result output by the feature point detection model, wherein the feature point detection result includes the plurality of first feature points or the plurality of second feature points;

[0026] Among them, the feature point detection model includes a multi-scale feature module and a fusion module. The multi-scale feature module is used to construct a feature pyramid based on the image in the image pair to be detected, and output the corresponding intermediate feature points based on each layer of feature images of the feature pyramid. The fusion module is used to output the feature point detection results of the images in the image pair to be detected based on the intermediate feature points corresponding to each layer of the feature images.

[0027] In a second aspect, the present invention further provides a high-lock bolt detection device, comprising:

[0028] An image acquisition module, used to acquire a reference image of the high-lock bolt to be detected and an image to be detected, wherein the reference image and the image to be detected respectively reflect the states of the high-lock bolt to be detected at different times;

[0029] An object-image ratio determination module, used to determine the object-image ratios corresponding to the reference image and the image to be detected respectively based on the imaging focal lengths of the reference image and the image to be detected, wherein the object-image ratio of the image reflects the difference between the pixel distance in the image and the actual distance;

[0030] An image conversion module, configured to convert the image to be detected and / or the reference image based on the object-image ratios respectively corresponding to the reference image and the image to be detected, so as to obtain an image pair to be detected, wherein the object-image ratios corresponding to the two images in the image pair to be detected are consistent;

[0031] The deformation analysis module is used to determine the deformation of the high-locking bolt to be detected at the time corresponding to the image to be detected relative to the time corresponding to the reference image based on the image pair to be detected.

[0032] In a third aspect, the present invention further provides an electronic device, comprising: a memory for storing a computer software program; a processor for reading and executing the computer software program, thereby implementing the high-locking bolt detection method as described in the first aspect above.

[0033] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium, wherein the storage medium stores a computer software program, and when the computer software program is executed by a processor, the high-locking bolt detection method as described in the first aspect above is implemented.

[0034] In a fifth aspect, the present invention further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the high-locking bolt detection method as described in the first aspect above.

[0035] The high-locking bolt detection method provided by the present invention obtains a reference image of a high-locking bolt to be detected and an image to be detected, wherein the reference image and the image to be detected respectively reflect the state of the high-locking bolt to be detected at different times; based on the imaging focal lengths of the reference image and the image to be detected, the object-image ratios respectively corresponding to the reference image and the image to be detected are determined, and the object-image ratio of the image reflects the difference between the pixel distance in the image and the actual distance; based on the object-image ratios respectively corresponding to the reference image and the image to be detected, the image to be detected and / or the reference image are converted to obtain an image pair to be detected, wherein the object-image ratios corresponding to the two images in the image pair to be detected are consistent; based on the image pair to be detected, the deformation of the high-locking bolt to be detected at the time corresponding to the image to be detected relative to the time corresponding to the reference image is determined, therefore, by obtaining the reference image and the image to be detected respectively reflecting the state of the high-locking bolt to be detected at different times, determining the object-image ratio based on the imaging focal lengths of the reference image and the image to be detected, and converting the reference image and the image to be detected based on the object-image ratio, two images with consistent object-image ratios reflecting the state of the high-locking bolt to be detected at different times can be obtained, deformation detection can be achieved based on image comparison, and detection without direct mechanical contact is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0037] Figure 1 It is a schematic flow chart of the high-lock bolt detection method provided by the present invention;

[0038] Figure 2It is a structural schematic diagram of a high-lock bolt detection device provided by the present invention;

[0039] Figure 3 A schematic diagram of an electronic device according to an embodiment of the present invention;

[0040] Figure 4 A schematic diagram of an embodiment of a computer-readable storage medium provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0042] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0043] In the description of the present invention, the term "for example" is used to mean "used as an example, illustration or explanation". Any embodiment described as "for example" in the present invention is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is given to enable any technician in the field to implement and use the present invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present invention can be implemented without using these specific details. In other examples, well-known structures and processes will not be elaborated in detail to avoid obscuring the description of the present invention with unnecessary details. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in the present invention.

[0044] Optional, see Figure 1 , Figure 1 : is a flow chart of a high lock bolt detection method provided by the present invention, and the high lock bolt detection method comprises the following steps:

[0045] Step 10, obtaining a reference image of the high-lock bolt to be detected and an image to be detected, wherein the reference image and the image to be detected respectively reflect the states of the high-lock bolt to be detected at different times;

[0046] Step 20: Determine the object-image ratios corresponding to the reference image and the image to be detected based on the imaging focal lengths of the reference image and the image to be detected, respectively. The object-image ratio of the image reflects the difference between the pixel distance in the image and the actual distance.

[0047] Step 30: based on the object-image ratios of the reference image and the image to be detected, respectively, convert the image to be detected and / or the reference image to obtain an image pair to be detected, wherein the object-image ratios of the two images in the image pair to be detected are consistent;

[0048] Step 40: Determine, based on the pair of images to be detected, the deformation of the high-lock bolt to be detected at the time corresponding to the image to be detected relative to the time corresponding to the reference image.

[0049] The method provided by the present invention obtains a reference image and an image to be detected which respectively reflect the states of the high-locking bolt to be detected at different times, determines the object-image ratio based on the imaging focal lengths of the reference image and the image to be detected, and converts the reference image and the image to be detected based on the object-image ratio, so as to obtain two images with consistent object-image ratios reflecting the states of the high-locking bolt to be detected at different times. Deformation detection can be achieved based on image comparison, thereby achieving detection without direct mechanical contact.

[0050] The images of the high-locking bolt to be tested at different times can reflect the state of the high-locking bolt to be tested at different times. Therefore, deformation detection can be achieved by comparing the images of the high-locking bolt to be tested at different times. In the method provided by the present invention, a reference image of the high-locking bolt to be tested and an image to be tested whose deformation is to be detected are first obtained. The reference image and the image to be tested are images obtained by imaging the high-locking bolt to be tested at different times. That is to say, the reference image and the image to be tested reflect the state of the high-locking bolt to be tested at different times. Based on the image to be tested and the reference image, the deformation of the high-locking bolt to be tested at the time corresponding to the image to be tested relative to the time corresponding to the reference image can be obtained.

[0051] When imaging the high-lock bolt to be inspected to obtain the reference image and the image to be inspected, shooting can be performed at multiple angles or positions to ensure that sufficient data is obtained to cover the entire deformation detection area. A shooting device that can shoot within a wide focal length range can be used. The configuration should include a high-stability bracket and a precise focal length adjustment mechanism to ensure shooting stability and repeatability at different focal lengths.

[0052] There is a difference between the pixel size in the image and the actual size, and this difference will also change due to different shooting focal lengths. This difference can be measured by the object-image ratio, and the object-image ratio of the image reflects the difference between the pixel distance in the image and the actual distance. Therefore, ideally, if the deformation detection of the high-locking bolt to be detected is performed directly based on the image to be detected and the reference image, the two images need to be shot at the same focal length. However, affected by the actual situation, this same focal length shooting is often not possible. For example, the high-locking bolt to be detected may be in different positions at two times (for example, before and after the installation of the component), or the high-locking bolt to be detected has a large volume or high deformation requirements, and it is necessary to determine its deformation at different scales and precisions. In these cases, imaging the high-locking bolt to be detected may require imaging from multiple focal lengths to achieve better imaging effects, such as selecting a shorter focal length for small or detailed objects, and selecting a longer focal length for larger objects or objects at a longer distance. Therefore, the image to be detected and the reference image need to be further processed to obtain two images with the same object-image ratio before deformation detection.

[0053] In one possible implementation, before the image to be detected and the reference image are converted to have consistent object-image ratio, they may be preprocessed, including image cropping, contrast adjustment, and sharpening, to reduce the impact of other factors such as lighting on the image and optimize the subsequent deformation analysis process.

[0054] Specifically, based on the imaging focal lengths of the reference image and the image to be detected, determining the object-image ratios corresponding to the reference image and the image to be detected respectively includes:

[0055] Obtaining the object-image ratios corresponding to the reference image and the image to be detected respectively based on a preset formula;

[0056] The default formula is:

[0057] It refers to the object-image ratio of the image obtained when the focal length is f and the distance between the camera and the high-lock bolt to be detected is dis. The unit is mm / pixcel. dis is the distance between the camera and the high-lock bolt to be detected, and the unit is mm. and is the fitting coefficient corresponding to the imaging focal length f of the image, that is, for the same imaging focal length f, the corresponding fitting coefficient and is a fixed constant.

[0058] The fitting coefficient can be obtained through image acquisition experiments. Specifically, in the experiment, a high-resolution video camera with a variable-focus lens can be selected. The device can shoot in a wide range of focal lengths to adapt to detection objects of different sizes and distances, and includes a high-stability bracket and a precise focal length adjustment mechanism to ensure shooting stability and repeatability at different focal lengths. The video camera is connected to a computer system, and the shooting parameters are controlled by dedicated software, and image data is transmitted.

[0059] In the experimental preparation stage, select the shooting object. You can choose a black and white checkerboard calibration plate of known specifications as the shooting object, so that you can choose the corner points of the calibration plate to count the pixel distance, which is more accurate. Set the camera focal length that needs to determine the corresponding fitting coefficient, shoot the shooting object at different distances to obtain multiple images (for example, five images) and ensure that the black and white checkerboard calibration plate appears completely in the camera field of view, and use a laser rangefinder to detect and record the distance between the camera and the center of the calibration plate. For each image obtained by shooting, randomly extract the corner points of the calibration plate pattern to form a point pair, and record the pixel coordinates of the point pair (for example, randomly extract 10 groups), and calculate the average value of the ratio of the object spacing of multiple corresponding corner points to the pixel spacing. Take the average value and the distance value detected by the laser rangefinder as independent variables, and use the least squares method to fit the fitting coefficient in the preset formula. For the fitting coefficients corresponding to other focal length values, similar steps can be used to obtain.

[0060] Through the preset formula in the method provided by the present invention, the object-image ratios corresponding to the reference image and the image to be detected can be calculated, so that the reference image and / or the image to be detected can be converted to obtain an image pair to be detected, which includes two images with the same object-image ratio, thereby realizing automatic adjustment and conversion of images at different focal lengths to ensure the consistency and comparability of the data, thereby ensuring the accuracy of the image-based detection results.

[0061] Based on the object-image ratios corresponding to the reference image and the image to be detected, respectively, the image to be detected and / or the reference image are converted to obtain an image pair to be detected, including:

[0062] Obtaining a reference object-image ratio, where the reference object-image ratio is the object-image ratio corresponding to the image to be detected, the object-image ratio corresponding to the reference image, or another preset object-image ratio;

[0063] Based on the reference object-image ratio and the object-image ratio of the image to be detected and / or the reference image, the image to be detected and / or the reference image is converted to obtain an image pair to be detected.

[0064] The reference object-image ratio may be the object-image ratio corresponding to the image to be detected, the object-image ratio corresponding to the reference image, or other preset object-image ratios. In practical applications, using the object-image ratio corresponding to the reference image as the reference object-image ratio can improve conversion efficiency.

[0065] Based on the reference object-image ratio and the object-image ratio of the image to be detected and / or the reference image, the image to be detected and / or the reference image is converted to obtain an image pair to be detected, including:

[0066] Obtaining a ratio of the object-image ratio of the image to be detected and / or the reference image to the reference object-image ratio;

[0067] The image content of the image to be detected and / or the reference image is scaled based on the ratio to obtain an image in the pair of images to be detected.

[0068] When the reference object-image ratio is the object-image ratio of the image to be detected or the reference image, it is only necessary to convert the image without the reference object-image ratio. That is, when the reference object-image ratio is the object-image ratio of the reference image, it is only necessary to convert the image to be detected. At this time, the image in the image pair to be detected is the reference image and the image obtained after converting the image to be detected.

[0069] Specifically, when the image to be detected or the reference image is converted, the ratio of the corresponding object-image ratio to the reference object-image ratio is obtained, and the image to be detected or the reference image is scaled based on the ratio. For example, if object A occupies 15 pixels in the image, and it needs to be reduced by 1.5 times according to the object-image ratio function, then object A is reduced as a whole so that object A occupies 10 pixels in the image.

[0070] After obtaining the image pair to be detected, since the object-image ratio of the images in the image pair to be detected is consistent, the corresponding deformation can be obtained based on the pixel distance between the pixel points corresponding to the same point of the high-locking bolt to be detected between the images in the image pair to be detected, thereby realizing deformation detection. Specifically, determining the deformation of the high-locking bolt to be detected at the time corresponding to the image to be detected relative to the time corresponding to the reference image based on the image pair to be detected includes:

[0071] Perform feature point detection on the two images in the image pair to be detected respectively to obtain a plurality of first feature points and a plurality of second feature points;

[0072] Matching the first feature point and the second feature point to obtain a feature point pair;

[0073] Based on the pixel positions of the feature point pairs, determining the deformation of the high-lock bolt to be detected at the time corresponding to the image to be detected relative to the time corresponding to the reference image;

[0074] The image pair to be detected is a first image pair, a second image pair or a third image pair. The first image pair includes two converted images, the second image pair includes a converted image and a reference image, and the third image pair includes a converted image and an image to be detected.

[0075] The method provided by the present invention realizes the matching of pixel points in the image pair to be detected by performing feature point detection on the image pair to be detected, that is, identifying the pixel points corresponding to the same actual point from the image pair to be detected.

[0076] Perform feature point detection on the two images in the image pair to be detected, respectively, to obtain a plurality of first feature points and a plurality of second feature points, including:

[0077] Inputting the images in the image pair to be detected into the trained feature point detection model respectively, and obtaining the feature point detection results output by the feature point detection model, wherein the feature point detection results include a plurality of first feature points or a plurality of second feature points;

[0078] Among them, the feature point detection model includes a multi-scale feature module and a fusion module. The multi-scale feature module is used to construct a feature pyramid based on the image in the image pair to be detected, and output the corresponding intermediate feature points based on each layer of feature images in the feature pyramid. The fusion module is used to output the feature point detection results of the image in the image pair to be detected based on the intermediate feature points corresponding to each layer of feature images.

[0079] In the method provided by the present invention, a multi-scale processing mechanism is adopted to work simultaneously on images of different resolutions, effectively extract feature points of various scales, and fuse the feature points detected at various scales, which can effectively improve the accuracy of the detected feature points, ensure that the detected feature points are representative key points, and thus improve the accuracy of deformation detection.

[0080] Each image in the image pair to be detected can be used as a target image. The target image is input into the trained feature point detection model. First, a feature pyramid is constructed through the multi-scale feature module in the feature point detection model. The pyramid includes multiple layers of feature images, and the resolution of each layer of feature images is different. Specifically, the multi-scale feature module includes multiple detection network layers, each layer includes an image feature extraction module and a feature point head. The image feature extraction module is used to extract the feature image of the corresponding resolution of the layer, and the feature point head is used to perform feature point detection based on the feature image of the layer, and output the intermediate feature points corresponding to the feature image of the layer.

[0081] Furthermore, after outputting the intermediate feature points corresponding to each layer of feature images, each intermediate feature point is fused to obtain the feature point detection result of the target image. Specifically, the fusion of each intermediate feature point can be achieved based on the confidence of each intermediate feature point. The confidence of the intermediate feature point is also output at the same time as the intermediate detection point is output at the feature point head. The intermediate feature points corresponding to each feature image and the confidence of the intermediate feature point are input into the fusion module to obtain the feature point detection result of the target image output by the fusion module, which specifically includes:

[0082] Through the fusion module, perform the following operations:

[0083] Map the intermediate feature points to the same scale;

[0084] Select the middle feature point with the largest confidence as the feature point to be processed, and obtain the overlap between the middle feature point and the feature point to be processed in the neighborhood of the feature point to be processed, where the overlap reflects the similarity between the middle feature point and the feature point to be processed;

[0085] Delete the intermediate feature points whose overlap is higher than the preset threshold;

[0086] Select the intermediate feature point with the largest confidence from the intermediate feature points after removing the feature point to be processed as the new feature point to be processed, and repeat the steps of obtaining the overlap between the intermediate feature points in the neighborhood of the feature point to be processed and the feature point to be processed, until all the intermediate feature points are traversed;

[0087] The retained intermediate feature points are used as the feature point detection results of the target image.

[0088] As can be seen from the previous description, the multi-scale feature module outputs the feature point detection results at multiple resolutions, and maps the intermediate feature points at each resolution to the same scale, that is, converts the position coordinates of the intermediate feature points at each resolution to the image coordinates at the same resolution. For example, based on the feature image with a resolution of a*b, the coordinates of the intermediate feature point A are detected as (x1, y1). When it is converted to the image coordinate system with a resolution of 2a*2b, the coordinates of the intermediate feature point A will change accordingly, so that the actual content corresponding to the position coordinates of the intermediate feature point at two scales is consistent.

[0089] For each intermediate feature point converted to the same scale, first select the intermediate feature point with the largest confidence as the feature point to be processed, and search for the intermediate feature point in the neighborhood of the feature point to be processed. The neighborhood of the feature point to be processed refers to a domain composed of pixels whose pixel distance to the feature point to be processed is within a preset range (for example, 4 pixel distances). The intermediate feature points found in the field of the feature point to be processed are respectively calculated for overlap with the feature point to be processed. The overlap reflects the similarity between the intermediate feature point and the feature point to be processed. The higher the overlap, the greater the possibility that the intermediate feature point and the feature point to be processed are actually detecting the same position point. The method provided by the present invention can effectively reduce the redundancy of feature points, optimize the representation of feature points, and improve the stability and robustness of feature points in deformation analysis by screening the intermediate feature points in the field of intermediate feature points with higher confidence using the overlap.

[0090] Specifically, obtaining the overlap between the intermediate feature point in the neighborhood of the feature point to be processed and the feature point to be processed includes:

[0091] Obtain feature descriptors of intermediate feature points and feature points to be processed respectively;

[0092] The overlap between the intermediate feature points and the feature points to be processed is determined based on the feature descriptors of the intermediate feature points and the feature points to be processed.

[0093] The feature descriptor of the intermediate feature point can be output together when detecting the intermediate feature point. The feature descriptor of the intermediate feature point describes the intermediate feature point. In the method provided by the present invention, if the similarity between the intermediate feature point and the feature descriptor of the feature point to be processed reaches a preset similarity threshold, then it is determined that the overlap between the intermediate feature point and the feature point to be processed is higher than the preset threshold, and the intermediate feature point with the overlap higher than the preset threshold is deleted. After that, the intermediate feature point with the highest confidence is continued to be selected as the new feature point to be processed from the intermediate feature points after removing the current feature point to be processed, and the steps of searching for the intermediate feature point in the field of the feature point to be processed and processing it as described above are repeated until all the intermediate feature points (excluding the deleted intermediate feature points) are used as feature points to be processed once. Finally, the intermediate feature points that have not been deleted are used as the feature point detection results of the target image.

[0094] The feature point detection model can be trained based on multiple sets of training data, each set of training data includes a sample image and a feature point detection result corresponding to the sample image. The training data can be obtained based on an existing feature point annotation dataset, such as the MS-COCO dataset.

[0095] After obtaining each first feature point and the second feature point, the first feature point and the second feature point are matched. Feature point matching can adopt an existing feature point matching method, such as a matching algorithm based on Euclidean distance, a descriptor-based method (such as FLANN, Fast Library for Approximate Nearest Neighbors) or other effective matching algorithms. In the matching process, the RANSAC algorithm can be combined to eliminate false matches and improve the accuracy of matching.

[0096] The first feature point and the second feature point that are successfully matched are considered to correspond to the same point on the high-lock bolt to be detected. The change in coordinate position between the first feature point and the second feature point can reflect the change in position of the same point on the high-lock bolt to be detected at different times, that is, the deformation. Therefore, according to the change in the spatial position of the feature point after matching, the deformation data of the high-lock bolt to be detected can be calculated using a geometric change model (such as affine transformation or perspective transformation).

[0097] The high locking bolt detection device provided by an embodiment of the present invention is described below. The high locking bolt detection device described below and the high locking bolt detection method described above can be referenced to each other.

[0098] See also Figure 2 , Figure 2 A schematic diagram of the structure of a high-lock bolt detection device provided in an embodiment of the present invention, the high-lock bolt detection device comprises:

[0099] An image acquisition module 210 is used to acquire a reference image of the high-lock bolt to be detected and an image to be detected, wherein the reference image and the image to be detected respectively reflect the states of the high-lock bolt to be detected at different times;

[0100] The object-image ratio determination module 220 is used to determine the object-image ratios corresponding to the reference image and the image to be detected based on the imaging focal lengths of the reference image and the image to be detected, respectively. The object-image ratio of the image reflects the difference between the pixel distance in the image and the actual distance;

[0101] An image conversion module 230 is used to convert the image to be detected and / or the reference image based on the object-image ratios corresponding to the reference image and the image to be detected, respectively, to obtain an image pair to be detected, wherein the object-image ratios corresponding to the two images in the image pair to be detected are consistent;

[0102] The deformation analysis module 240 is used to determine the deformation of the high-lock bolt to be detected at the time corresponding to the image to be detected relative to the time corresponding to the reference image based on the image pair to be detected.

[0103] The high-locking bolt detection device provided by the present invention obtains a reference image of a high-locking bolt to be detected and an image to be detected, wherein the reference image and the image to be detected respectively reflect the state of the high-locking bolt to be detected at different times; based on the imaging focal lengths of the reference image and the image to be detected, the object-image ratios corresponding to the reference image and the image to be detected are determined, and the object-image ratios of the images reflect the difference between the pixel distance in the image and the actual distance; based on the object-image ratios corresponding to the reference image and the image to be detected, the image to be detected and / or the reference image are converted to obtain an image pair to be detected, wherein the object-image ratios corresponding to the two images in the image pair to be detected are consistent; based on the image pair to be detected, the deformation of the high-locking bolt to be detected at the time corresponding to the image to be detected relative to the time corresponding to the reference image is determined, therefore, by obtaining the reference image and the image to be detected respectively reflecting the state of the high-locking bolt to be detected at different times, determining the object-image ratio based on the imaging focal lengths of the reference image and the image to be detected, and converting the reference image and the image to be detected based on the object-image ratio, two images with consistent object-image ratios reflecting the state of the high-locking bolt to be detected at different times can be obtained, deformation detection can be achieved based on image comparison, and detection without direct mechanical contact is achieved.

[0104] In one embodiment, the object-image ratio determination module 220 is further configured to:

[0105] Obtaining the object-image ratios corresponding to the reference image and the image to be detected respectively based on a preset formula;

[0106] The preset formula is:

[0107] in, is the object-image ratio corresponding to the image, dis is the distance between the camera and the high-lock bolt to be detected, and is the fitting coefficient corresponding to the imaging focal length of the image.

[0108] In one embodiment, the image conversion module 230 is further configured to:

[0109] Obtaining a reference object-image ratio, where the reference object-image ratio is the object-image ratio corresponding to the image to be detected, the object-image ratio corresponding to the reference image, or another preset object-image ratio;

[0110] Based on the reference object-image ratio and the object-image ratio of the image to be detected and / or the reference image, the image to be detected and / or the reference image are converted to obtain the image pair to be detected.

[0111] In one embodiment, the image conversion module 230 is further configured to:

[0112] Obtaining a ratio of the object-image ratio of the image to be detected and / or the reference image to the reference object-image ratio;

[0113] The image content of the image to be detected and / or the reference image is scaled based on the ratio to obtain an image in the pair of images to be detected.

[0114] In one embodiment, the deformation analysis module 240 is further configured to:

[0115] Perform feature point detection on the two images in the image pair to be detected respectively to obtain a plurality of first feature points and a plurality of second feature points;

[0116] Matching the first feature point and the second feature point to obtain a feature point pair;

[0117] Based on the pixel positions of the feature point pairs, determining the deformation of the high-lock bolt to be detected at the time corresponding to the image to be detected relative to the time corresponding to the reference image;

[0118] The image pair to be detected is a first image pair, a second image pair or a third image pair, the first image pair includes two of the conversion images, the second image pair includes the conversion image and the reference image, and the third image pair includes the conversion image and the image to be detected.

[0119] In one embodiment, the deformation analysis module 240 is further configured to:

[0120] Inputting the images in the to-be-detected image pair into a trained feature point detection model respectively, and obtaining a feature point detection result output by the feature point detection model, wherein the feature point detection result includes the plurality of first feature points or the plurality of second feature points;

[0121] Among them, the feature point detection model includes a multi-scale feature module and a fusion module. The multi-scale feature module is used to construct a feature pyramid based on the image in the image pair to be detected, and output the corresponding intermediate feature points based on each layer of feature images of the feature pyramid. The fusion module is used to output the feature point detection results of the images in the image pair to be detected based on the intermediate feature points corresponding to each layer of the feature images.

[0122] See also Figure 3 , Figure 3 Schematic diagram of an electronic device provided by an embodiment of the present invention. Figure 3 As shown, an embodiment of the present invention provides an electronic device 300, including a memory 310, a processor 320, and a computer program 311 stored in the memory 310 and executable on the processor 320. When the processor 320 executes the computer program 311, the following steps are implemented:

[0123] Acquire a reference image of a high-lock bolt to be detected and an image to be detected, wherein the reference image and the image to be detected respectively reflect the states of the high-lock bolt to be detected at different times;

[0124] Determine the object-image ratios corresponding to the reference image and the image to be detected based on the imaging focal lengths of the reference image and the image to be detected, respectively, wherein the object-image ratio of the image reflects the difference between the pixel distance in the image and the actual distance;

[0125] Based on the object-image ratios respectively corresponding to the reference image and the image to be detected, the image to be detected and / or the reference image are converted to obtain an image pair to be detected, wherein the object-image ratios corresponding to the two images in the image pair to be detected are consistent;

[0126] Based on the pair of images to be detected, the deformation of the high-locking bolt to be detected at the time corresponding to the image to be detected relative to the time corresponding to the reference image is determined.

[0127] See also Figure 4 , Figure 4 Schematic diagram of an embodiment of a computer-readable storage medium provided in an embodiment of the present invention. Figure 4 As shown, this embodiment provides a computer-readable storage medium 400, on which a computer program 411 is stored. When the computer program 411 is executed by a processor, the following steps are implemented:

[0128] Acquire a reference image of a high-lock bolt to be detected and an image to be detected, wherein the reference image and the image to be detected respectively reflect the states of the high-lock bolt to be detected at different times;

[0129] Determine the object-image ratios corresponding to the reference image and the image to be detected based on the imaging focal lengths of the reference image and the image to be detected, respectively, wherein the object-image ratio of the image reflects the difference between the pixel distance in the image and the actual distance;

[0130] Based on the object-image ratios respectively corresponding to the reference image and the image to be detected, the image to be detected and / or the reference image are converted to obtain an image pair to be detected, wherein the object-image ratios corresponding to the two images in the image pair to be detected are consistent;

[0131] Based on the pair of images to be detected, the deformation of the high-locking bolt to be detected at the time corresponding to the image to be detected relative to the time corresponding to the reference image is determined.

[0132] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0133] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0134] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (apparatus), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0135] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0136] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0137] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0138] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A high lock bolt detection method, characterized in that: include: Acquire a reference image of the high-lock bolt to be detected and an image to be measured, wherein the reference image and the image to be measured respectively reflect the states of the high-lock bolt to be detected at different times; Determine the object-image ratios corresponding to the reference image and the image to be measured respectively based on the imaging focal lengths of the reference image and the image to be measured, wherein the object-image ratios of the images reflect the difference between the pixel distance in the image and the actual distance; Based on the object-image ratios respectively corresponding to the reference image and the image to be measured, the image to be measured and / or the reference image are converted to obtain an image pair to be detected, wherein the object-image ratios corresponding to the two images in the image pair to be detected are consistent; Based on the pair of images to be detected, determining the deformation of the high-lock bolt to be detected at the time corresponding to the image to be measured relative to the time corresponding to the reference image; Wherein, determining the object-image ratios respectively corresponding to the reference image and the image to be measured based on the imaging focal lengths of the reference image and the image to be measured includes: Obtaining the object-image ratios corresponding to the reference image and the image to be measured respectively based on a preset formula; The preset formula is: in, is the object-image ratio corresponding to the image, dis is the distance between the camera and the target object, and is the fitting coefficient corresponding to the imaging focal length of the image; The step of determining the deformation of the high-lock bolt to be detected at the time corresponding to the image to be measured relative to the time corresponding to the reference image based on the image pair to be detected includes: Perform feature point detection on the two images in the image pair to be detected respectively to obtain a plurality of first feature points and a plurality of second feature points; Matching the first feature point and the second feature point to obtain a feature point pair; Based on the pixel positions of the feature point pairs, determining the deformation of the high-lock bolt to be detected at the time corresponding to the image to be measured relative to the time corresponding to the reference image; The image pair to be detected is a first image pair, a second image pair or a third image pair. The first image pair includes two conversion images, the second image pair includes a conversion image and the reference image, and the third image pair includes a conversion image and the image to be measured.

2. The high locking bolt detection method according to claim 1, characterized in that: The converting the image to be measured and / or the reference image based on the object-image ratios respectively corresponding to the reference image and the image to be measured to obtain the image pair to be detected includes: Obtaining a reference object-image ratio, where the reference object-image ratio is the object-image ratio corresponding to the image to be measured, the object-image ratio corresponding to the reference image, or another preset object-image ratio; Based on the reference object-image ratio and the object-image ratio of the image to be measured and / or the reference image, the image to be measured and / or the reference image are converted to obtain the image pair to be detected.

3. The high locking bolt detection method according to claim 2, characterized in that: The converting the image to be measured and / or the reference image based on the reference object-image ratio and the object-image ratio of the image to be measured and / or the reference image to obtain the image pair to be detected includes: Acquire a ratio of the object-image ratio of the image to be measured and / or the reference image to the reference object-image ratio; The image content of the image to be measured and / or the reference image is scaled based on the ratio to obtain an image in the pair of images to be detected.

4. The high locking bolt detection method according to claim 1, characterized in that: The method of performing feature point detection on two images in the image pair to be detected to obtain a plurality of first feature points and a plurality of second feature points comprises: Inputting the images in the to-be-detected image pair into a trained feature point detection model respectively, and obtaining a feature point detection result output by the feature point detection model, wherein the feature point detection result includes the plurality of first feature points or the plurality of second feature points; Among them, the feature point detection model includes a multi-scale feature module and a fusion module. The multi-scale feature module is used to construct a feature pyramid based on the image in the image pair to be detected, and output the corresponding intermediate feature points based on each layer of feature images of the feature pyramid. The fusion module is used to output the feature point detection results of the images in the image pair to be detected based on the intermediate feature points corresponding to each layer of the feature images.

5. A high-lock bolt detection device, characterized in that: include: An image acquisition module, used to acquire a reference image of the high-lock bolt to be detected and an image to be measured, wherein the reference image and the image to be measured respectively reflect the states of the high-lock bolt to be detected at different times; An object-image ratio determination module, used to determine the object-image ratios corresponding to the reference image and the image to be measured respectively based on the imaging focal lengths of the reference image and the image to be measured, wherein the object-image ratio of the image reflects the difference between the pixel distance in the image and the actual distance; An image conversion module, configured to convert the image to be measured and / or the reference image based on the object-image ratios respectively corresponding to the reference image and the image to be measured, so as to obtain an image pair to be detected, wherein the object-image ratios corresponding to the two images in the image pair to be detected are consistent; A deformation analysis module, used for determining, based on the pair of images to be detected, the deformation of the high-lock bolt to be detected at a time corresponding to the image to be measured relative to a time corresponding to the reference image; Wherein, determining the object-image ratios respectively corresponding to the reference image and the image to be measured based on the imaging focal lengths of the reference image and the image to be measured includes: Obtaining the object-image ratios corresponding to the reference image and the image to be measured respectively based on a preset formula; The preset formula is: in, is the object-image ratio corresponding to the image, dis is the distance between the camera and the target object, and is the fitting coefficient corresponding to the imaging focal length of the image; The step of determining the deformation of the high-lock bolt to be detected at the time corresponding to the image to be measured relative to the time corresponding to the reference image based on the image pair to be detected includes: Perform feature point detection on the two images in the image pair to be detected respectively to obtain a plurality of first feature points and a plurality of second feature points; Matching the first feature point and the second feature point to obtain a feature point pair; Based on the pixel positions of the feature point pairs, determining the deformation of the high-lock bolt to be detected at the time corresponding to the image to be measured relative to the time corresponding to the reference image; The image pair to be detected is a first image pair, a second image pair or a third image pair. The first image pair includes two conversion images, the second image pair includes a conversion image and the reference image, and the third image pair includes a conversion image and the image to be measured.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the high-locking bolt detection method as described in any one of claims 1-4 is implemented.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the high-lock bolt detection method as described in any one of claims 1 to 4 is implemented.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the high-lock bolt detection method as described in any one of claims 1 to 4 is implemented.

Images