A device for calculating subpixel displacement based on image vision

Abstract

This invention discloses an image-based visual subpixel displacement calculation device, relating to the field of optical measurement technology. By fusing and analyzing grayscale gradient information, corner response information, and texture density information in continuous images, the block-level stability of image blocks is obtained. Based on this, automatic target region identification, connected region filtering, and progressive merging are performed, enabling accurate extraction of target regions in complex background environments. Through a feature point evaluation process, pixels with high structural stability within the target region are filtered, and non-maximum suppression and minimum spacing constraints are combined to achieve uniform distribution of feature points, effectively improving the stability and anti-interference capability of continuous frame matching. Subpixel coordinates of feature points are obtained by combining integer-pixel matching and subpixel interpolation, and the overall displacement is calculated using multi-feature-point weighted fusion, reducing measurement errors caused by mismatches of individual feature points and achieving high-precision calculation of target displacement.

Description

Technical Field

[0001] This invention relates to the field of optical measurement technology, specifically to a device for calculating subpixel displacement based on image vision. Background Technology

[0002] With the continuous expansion of infrastructure such as bridges, high-rise buildings, tunnels, transmission towers, and large industrial structures, they are susceptible to deflection, vibration, displacement, and structural response changes during their service life due to vehicle loads, wind loads, seismic effects, and environmental factors. Long-term continuous monitoring of structural displacement allows for timely understanding of the structural operating status, assessment of structural safety performance, and provides crucial information for structural maintenance, damage diagnosis, and life prediction. Therefore, high-precision displacement monitoring technology has become an important research direction in the field of structural health monitoring.

[0003] Currently, image-based non-contact displacement monitoring technology is increasingly being applied to structural health monitoring in fields such as bridges, buildings, and tunnels due to its advantages of flexible installation, large measurement range, and no need to contact the measured structure. However, most existing visual displacement monitoring systems adopt a processing mode that separates image acquisition from displacement calculation, requiring the transmission of a large amount of image data to backend equipment for analysis. This not only increases the data transmission burden but also easily leads to a decrease in the real-time performance of monitoring results, making it difficult to meet the real-time monitoring needs of dynamic structural displacement. Furthermore, existing technologies often rely on manual selection of target areas or direct identification using single features during the target area recognition process. Under environmental conditions such as complex backgrounds, changes in lighting, and partial occlusion, the target area positioning is prone to instability, thus affecting the reliability of subsequent displacement calculation results. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides an image visual subpixel displacement calculation device to solve the problems mentioned in the background section.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a device for calculating sub-pixel displacement based on image visual perception, comprising;

[0006] An image acquisition module is used to acquire continuous images of the target structure and preprocess the continuous images;

[0007] The identification and tracking module is used to construct block-level stability based on gray-level gradient information, corner response information and texture density information in continuous images, identify the target region based on the block-level stability, and establish the pixel planar motion trajectory of the target in continuous images.

[0008] The subpixel localization module is used to filter feature points within the target area, obtain the subpixel coordinates of the feature points through continuous frame matching, calculate the subpixel displacement of the target, and convert it into actual physical displacement.

[0009] Further, the recognition and tracking module divides the image into several image blocks, calculates the gray-level gradient magnitude and corner response mean of each pixel position in the image block, and calculates the average gradient intensity and corner response mean of each. Based on the gray-level gradient magnitude of each pixel position in the image block, the texture density of the graphic block is calculated, and then the average gradient intensity, average corner response, and texture density are fused according to a preset weight to obtain the block-level stability. Specifically, this includes:

[0010] For each pixel (x, y) in the image, calculate the horizontal and vertical gray-level gradients. and The gray-level gradient magnitude at pixel (x, y) is calculated based on the gray-level gradients in the horizontal and vertical directions. The calculation formula is:

[0011]

[0012] The image is divided into several small image blocks according to a set size. In this embodiment, the size is set to 32×32 pixels. The grayscale gradient magnitude of each pixel in the image block is counted and the average value is calculated to obtain the average gradient intensity of the image block.

[0013] For each pixel location in the image patch, the corner response value at that pixel location is calculated using the Harris corner response model. The calculation formula is as follows:

[0014]

[0015] Where M represents the gray-level autocorrelation matrix in the neighborhood of the pixel location, and its expression is: , and Let det(M) represent the horizontal and vertical grayscale changes at pixel position (x, y), det(M) represent the determinant of matrix M, trace(M) represent the trace of matrix M, and k is an empirical coefficient, usually taken as 0.04-0.06; count the corner response values ​​of each pixel in the image block, and select the corner response values ​​that are greater than zero to calculate the mean of the corner response of the image block.

[0016] When calculating the grayscale gradient magnitude in each image block, the gradient magnitude of each pixel in the image block is first normalized and mapped to the [0,1] interval. The number of pixels with a gradient magnitude greater than the gradient threshold is counted and used as the prominent pixels of the pixel block. In this implementation, the gradient threshold is set to 0.15. The total number of pixels in the pixel block is counted as the area of ​​the image block. The number of prominent pixels is counted and divided by the area to obtain the texture density of the image block. The larger the texture density, the richer the texture and the denser the features in the region of the image block, which is suitable for tracking.

[0017] The mean gradient intensity and mean corner response of the image patch are normalized, and then compared with...

[0018] The texture density is linearly weighted and summed according to preset weights to obtain the block-level stability of the image block. It should be noted that in this embodiment, the maximum and minimum gray-level gradient magnitudes in the image block are used to normalize the average gradient intensity. Similarly, the maximum and minimum gray corner response values ​​in the image block are used to normalize the average corner response. The weights are adjusted according to the specific scenario. In this embodiment, the weights of the average gradient intensity, the average corner response, and the texture density are set to 0.4, 0.4, and 0.2, respectively.

[0019] Furthermore, the recognition and tracking module divides the image block into foreground and background blocks based on the comparison result of block-level stability and a preset stability threshold, and constructs a block-level binary image based on the determination results of all image blocks; specifically including:

[0020] A preset stability threshold is used to distinguish between the target region and the background region. In this embodiment, the stability threshold is set to 0.6. The block-level stability of the image patch is compared with the stability threshold. If the block-level stability is greater than the stability threshold, the image patch is determined to be a foreground patch, that is, the image patch belongs to the target region. If the block-level stability is less than or equal to the stability threshold, the image patch is determined to be a background patch, that is, the image patch does not belong to the target region. Thus, a two-dimensional array Z with the same size as the image patch grid is constructed, and its expression is:

[0021]

[0022] Where S is the block-level stability of the image patch, S th The stability threshold is used to convert the image into a block-level binary image representing the stability state.

[0023] Furthermore, the identification and tracking module uses a connected component labeling algorithm to cluster foreground blocks in the block-level binary image to obtain multiple connected components, and filters the connected components according to a preset area range and a preset aspect ratio range, retaining connected components that satisfy the area range and the aspect ratio range; specifically including:

[0024] A two-pass scanning algorithm is used to process the two-dimensional label array: A 4-neighborhood algorithm defines the adjacency relationship between image blocks, meaning the current image block's neighboring blocks include its left, right, top, and bottom blocks; an 8-neighborhood algorithm, in addition to the 4-neighborhood, also includes the top-left, top-right, bottom-left, and bottom-right blocks. The first scan traverses each foreground block from top to bottom and left to right, checking the labels of its left and top blocks. If all scanned neighboring blocks are unlabeled, a new temporary label is assigned to the current block; if only one neighboring block has a label, the current block inherits that label; if multiple different labels exist, the current block inherits the smaller label, and the equivalence relationships between labels are recorded. After the first scan, all foreground blocks receive temporary labels, and an equivalence set of labels is established. During the second scan, the temporary labels are merged according to the equivalence relationships, assigning the same final label to blocks belonging to the same connected region, thus obtaining several independent connected regions.

[0025] The area and shape constraint analysis is performed on each connected region, specifically including: counting each connected image block and summing the area of ​​all image blocks to calculate the connected area. A preset area range is set. In this embodiment, considering that the size of a single image block is set to 32×32 pixels and the target usually occupies an area of ​​about 100×100 pixels in the image, the area range is preferably set to 4 to 20 image blocks. When the area of ​​the connected region is less than the preset minimum area threshold, it indicates that the region is insufficient in size and may be local noise or a false detection block. When the area of ​​the connected region is greater than the preset maximum area threshold, it indicates that the region exceeds the expected scale of the target and may be an abnormal region formed by multiple foreground blocks being mistakenly connected, adjacent irrelevant structures being included, or multiple candidate regions being merged. Therefore, it is removed. Only connected regions that meet the preset area range are retained to remove noise regions with too small an area and background regions with too large an area.

[0026] Based on the connected regions retained after area filtering, their corresponding minimum bounding rectangles are drawn. Specifically, the minimum bounding rectangle refers to the smallest axis-aligned rectangle that can completely enclose the connected region without rotating the image coordinate axes. The width and height of the minimum bounding rectangle of each connected region are extracted, and the width is divided by the height to obtain the aspect ratio. The aspect ratio is used to measure whether the shape of the region is close to the expected geometric shape of the target. A preset aspect ratio range is set. In this embodiment, considering that the imaging shape of the target in the image is usually close to a square or approximately rectangular, the aspect ratio range is preferably set to 0.5 to 2.0. Furthermore, among the connected regions that meet the preset area range, only the connected regions that meet the preset aspect ratio range are retained to ensure that the overall shape of the candidate region is consistent with the expected geometric features of the target.

[0027] Furthermore, the identification and tracking module calculates the center distance between every two connected regions retained after filtering, and performs region merging step by step in order of increasing center distance. Each time, it merges a pair of connected regions with the smallest center distance that meets a preset merging distance threshold, updates the merged connected region, and repeats the above process until no connected region pair meeting the merging distance threshold exists. The final connected region is then used as the target candidate region; specifically including:

[0028] Spatial merging is further performed on all connected regions that satisfy area and shape constraints. Specifically, the coordinates of the center points of the connected regions are extracted, and the Euclidean distance between the center points of the circumscribed rectangles of any two connected regions is calculated. This distance reflects the spatial proximity of the two connected regions on the image plane. The smaller the distance, the more likely the two regions are to originate from different parts of the same target segmented from the same target. To avoid merging irrelevant regions, a maximum allowable merging distance is set. The maximum allowable merging distance is set based on the target and its size, specifically by multiplying the merging coefficient by the target's reference size or expected scale. The merging coefficient is used to control the degree of leniency, and in this embodiment, it is set to 1.5. When the Euclidean distance between two connected regions is less than or equal to the maximum allowable merging distance, it indicates that the two regions have... The spatial premise for merging is to denote the two connected regions as candidate merge regions. Among all candidate merge regions that meet the maximum allowed merge distance, the region with the smallest Euclidean distance is selected for merging to obtain a new connected region. After merging, its minimum bounding rectangle and center point are redefined. After completing one merge, the Euclidean distance calculation, merge judgment, and minimum distance selection are re-executed for the remaining connected regions. That is, the distance between the new connected region and other connected regions is calculated again, and two pairs of connected regions that meet the maximum allowed merge distance and have the smallest Euclidean distance are searched again. The above merging process is repeated. This process continues to iterate until there are no regions in the set of connected regions that meet the merge distance threshold. The remaining region output at this time is the final target candidate region.

[0029] Furthermore, if there is only one target candidate region, it is directly determined as the final target region. If there are two or more target candidate regions, the identification and tracking module constructs a comprehensive matching score for the target candidate regions based on stability matching, area matching, shape matching, and position matching, and selects the target candidate region with the highest comprehensive matching score as the final target region. Specifically, this includes:

[0030] The overall matching score F is composed of stability matching items, area matching items, shape matching items, and position matching items, which can be specifically expressed as:

[0031]

[0032] in, Indicates the stability matching term. Indicates area matching items, Indicates shape matching items, This represents the location matching term; specifically: the stability matching term is calculated by averaging the block-level stability of all image patches contained within the target candidate region; the area matching term is calculated as follows:

[0033]

[0034] in, The reference area of ​​the target is determined by the target design dimensions or pre-calibration results;

[0035] Shape Matching Items The calculation is as follows:

[0036]

[0037] in, The reference aspect ratio of the target is determined by the target's design dimensions or pre-calibration results;

[0038] Location matching items The calculation is as follows:

[0039]

[0040] Where d represents the Euclidean distance between the target candidate region and the reference center point. This indicates the maximum allowable reference distance; the reference center point is the preset reference center point of the target, which is either the center of the initially manually selected target area or the center of the target area confirmed in the previous frame.

[0041] In each target candidate region, the target candidate region with the largest comprehensive matching score F is selected as the final target region. If multiple candidate regions have the same comprehensive matching score, the candidate region with the larger average stability matching term is selected first. If the stability matching term is still the same, the candidate region with the larger position matching term is selected first. If they still cannot be distinguished, the candidate region with the larger area matching term is selected. The minimum bounding rectangle boundary is recalculated for the final retained target candidate regions, and the bounding rectangle of the final target region is output. The center point coordinates of the bounding rectangle of the target region in each frame of the continuous image sequence are calculated and extracted. Thus, the sequence of center point coordinates of the target region in each frame of the image is obtained, which constitutes the motion trajectory of the target on the pixel plane.

[0042] Further, within the target area, the sub-pixel localization module calculates a feature evaluation value based on the corner response value and grayscale gradient magnitude of each pixel location, and filters candidate feature points based on the comparison result of the feature evaluation value and a preset feature scoring threshold; non-maximum suppression and minimum spacing constraint processing are performed on the candidate feature points to obtain feature points; specifically including:

[0043] Within the target area defined by the circumscribed rectangle, several feature points are selected as the objects of displacement calculation. Specifically, the pixel range covered by the circumscribed rectangle of the final target area is used as the feature extraction range. Within the feature extraction range, the corner response values ​​and grayscale gradient magnitudes at each pixel position are normalized and then weighted and summed according to a preset threshold to obtain the feature evaluation value. The sum of the weights is one. In this embodiment, the weights of the corner response values ​​and grayscale gradient magnitudes are set to 0.6 and 0.4, respectively. A preset feature scoring threshold is used to remove pixels with low stable feature evaluation values ​​and unclear local structures to avoid flat or noisy areas being mistakenly selected as feature points. In this embodiment, it is set to 0.65. Pixel positions that satisfy the feature evaluation value greater than the feature scoring threshold are recorded as candidate feature points. After obtaining the candidate feature points, non-maximum suppression processing is performed on the candidate feature points, that is, within the local neighborhood of each candidate feature point, Only the pixel position with the largest stable feature evaluation value is retained as the local optimum, and its neighboring points with smaller scores are deleted, thereby avoiding multiple duplicate feature points in the same local area. To eliminate overly dense candidate feature points, a minimum spacing constraint is also performed on the point set after non-maximum suppression. Specifically, the Euclidean distance between any two candidate feature points is calculated, and the distance is less than a preset minimum spacing threshold. In this embodiment, the minimum spacing threshold is preferably set to 4 pixels, or one-quarter of the side length of a single image block, to ensure that the retained feature points are evenly distributed in the target area and to avoid feature points being concentrated near local edges or the same corner. At this time, only the candidate feature point with the larger stable feature evaluation value is retained, and the candidate feature points with smaller evaluation values ​​are deleted, thereby avoiding excessive aggregation of feature points in the local area. Finally, the pixel position after threshold filtering, non-maximum suppression, and minimum spacing constraint is used as the feature point.

[0044] Furthermore, the subpixel localization module uses local image blocks centered on each feature point in the previous frame as matching templates. Within the search window of the current frame, it employs a normalized cross-correlation matching method to determine the integer pixel position that best matches each matching template, which is then used as the integer pixel matching position of the corresponding feature point; specifically, this includes:

[0045] Let the stable features selected from the t-th frame be expressed as: , where n represents the number of feature points participating in sub-pixel calculation, Let represent the pixel coordinates of the q-th feature point in frame t, where q represents the index of the feature point. The position of the target in the current frame is predicted using the coordinates of the target center point in the previous frame and the aforementioned pixel planar motion trajectory, resulting in the predicted center point of the target. A search window is constructed centered on this predicted center point, with a size of w×w, where w is a preset search side length, specifically determined based on the target's maximum displacement amplitude, sampling frame rate, and monitoring distance. Local image patches surrounding the corresponding feature point in the previous frame are used as templates to match candidate positions within the search window of the current frame. In this embodiment, normalized cross-correlation is used to calculate the similarity between the template and the candidate positions, yielding the correlation response value. Where u and v represent the candidate coordinates in the search window and the relevant response values. The larger the value, the more similar the candidate region of the current frame is to the template of the previous frame, and the more likely it is to be the corresponding location of the same physical feature point in the current frame; specifically, the formula for calculating the correlation response value is expressed as:

[0046]

[0047] in, Let represent the grayscale value at pixel (x, y) of the image in frame t-1. Let represent the grayscale value of the pixel at the candidate position (u,v) in the t-th frame image. template Average grayscale value of internal pixels This represents the average grayscale value of pixels within the candidate region of the current frame. By calculating the relevant response value for all candidate positions within the search window, the integer pixel position with the largest relevant response value is taken as the integer pixel matching position of the q-th feature point.

[0048] Furthermore, the subpixel localization module constructs a local response surface in the local neighborhood of the integer pixel matching position of each feature point based on the response values ​​of the relevant response peak points and their adjacent pixels. It calculates the subpixel offsets in the horizontal and vertical directions by interpolation, and superimposes the subpixel offsets with the integer pixel matching positions to obtain the subpixel coordinates of the feature point.

[0049] Furthermore, the subpixel localization module determines the weight of each feature point based on the block-level stability of the image block in which each feature point is located, performs a weighted average fusion of the displacement results of each feature point to obtain the overall displacement of the target on the pixel plane, and then converts the overall displacement into the actual physical displacement according to the pre-calibrated pixel scale coefficient.

[0050] Furthermore, it also includes an output display module, used to receive the displacement results output by the subpixel positioning module, and output the real-time displacement information of the target in the form of numerical display, curve display, and status prompts; specifically including:

[0051] The system receives the real-time displacement value of the target output from the sub-pixel positioning module and establishes displacement time-series data according to the image acquisition time sequence. The displacement time-series data includes at least the acquisition time, target center point coordinates, horizontal displacement, vertical displacement, and composite displacement value. The system displays the displacement result at the current moment in real time, including the target's current displacement value, displacement direction, and displacement rate. Simultaneously, it dynamically plots the displacement change curve according to the time sequence, allowing monitoring personnel to intuitively observe the structural displacement change trend. The system calculates the displacement rate based on the displacement change amount and time interval between two adjacent sampling moments and displays the displacement rate synchronously.

[0052] A pre-set structural safety displacement threshold is established, and the real-time displacement value is compared with the safety displacement threshold. When the real-time displacement value exceeds the safety displacement threshold, an over-limit alarm is generated. Specifically, the over-limit alarm information includes the alarm time, the current displacement value, the over-limit magnitude (the difference between the real-time displacement value and the safety displacement threshold), and the corresponding image frame number. At the same time, an audible and visual alarm, information push, or remote monitoring platform alarm is triggered to remind monitoring personnel to handle the situation in a timely manner.

[0053] The present invention has the following beneficial effects:

[0054] This invention provides a subpixel displacement calculation device based on image vision. By fusing and analyzing grayscale gradient information, corner response information, and texture density information in continuous images, the block-level stability of image blocks is obtained. Based on the block-level stability, automatic target region identification, connected region filtering, and progressive merging are performed, thereby accurately extracting target regions in complex background environments. Furthermore, a feature point evaluation process is used to filter pixels with high structural stability within the target region, and non-maximum suppression and minimum spacing constraints are combined to achieve uniform distribution of feature points, effectively improving the stability and anti-interference ability of continuous frame matching. Simultaneously, subpixel coordinates of feature points are obtained by combining integer pixel matching and subpixel interpolation, and the overall displacement is calculated by multi-feature point weighted fusion, reducing measurement errors caused by mismatches of individual feature points and achieving high-precision calculation of target displacement. This invention not only improves the reliability of target identification and tracking but also effectively enhances the accuracy of micro-displacement measurement and real-time monitoring capabilities, making it suitable for non-contact dynamic displacement monitoring of engineering structures such as bridges, high-rise buildings, and tunnels. Attached Figure Description

[0055] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This is a schematic diagram of a device for calculating subpixel displacement based on image vision. Detailed Implementation

[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] Please see Figure 1 The present invention provides a technical solution: an image visual subpixel displacement calculation device, comprising an image acquisition module, a recognition and tracking module, a subpixel positioning module and an output display module;

[0059] The image acquisition module acquires high-quality, low-noise continuous raw images and improves image quality through preprocessing. Specifically, in this embodiment, a CMOS global shutter industrial camera is used, and the resolution is selected according to the monitoring distance and accuracy requirements. An 850nm narrowband filter is installed in front of the lens, which, together with an 850nm LED array on the target, effectively suppresses ambient stray light. The camera automatically adjusts the exposure to ensure that the grayscale value of the target area is within the optimal dynamic range. The image is then denoised, contrast stretched, and sharpened, and an enhanced continuous image sequence is output. It should be noted that in this embodiment, denoising uses fast median filtering or bilateral filtering to eliminate shot noise and thermal noise. Contrast stretching uses adaptive histogram equalization to limit noise amplification and enhance the target edges. Sharpening uses the Laplacian operator or an unsharpened mask to enhance target feature points.

[0060] The recognition and tracking module locates the target region in continuous images, extracts stable features, and outputs the target's motion trajectory on the pixel plane, specifically including:

[0061] For each pixel (x, y) in the image, calculate the horizontal and vertical gray-level gradients. and The gray-level gradient magnitude at pixel (x, y) is calculated based on the gray-level gradients in the horizontal and vertical directions. The calculation formula is:

[0062]

[0063] It should be noted that regions with large gradients usually have rich textures and stable edges, making them suitable as displacement monitoring features. The image is divided into several small image blocks according to a set size. In this embodiment, the size is set to 32×32 pixels. The gray-level gradient amplitude of each pixel in the image block is counted, and the average value is calculated to obtain the average gradient intensity of the image block.

[0064] For each pixel location in the image patch, the corner response value at that pixel location is calculated using the Harris corner response model. The calculation formula is as follows:

[0065]

[0066] Where M represents the gray-level autocorrelation matrix in the neighborhood of the pixel location, and its expression is: , and Let R(x, y) represent the horizontal and vertical grayscale changes at pixel position (x, y), det(M) represent the determinant of matrix M, trace(M) represent the trace of matrix M, and k is an empirical coefficient, usually taken as 0.04-0.06. The larger the corner response value R(x, y), the more stable the local structure near the pixel, and the more suitable it is for feature extraction and tracking. The corner response values ​​of each pixel in the image block are counted, and the corner response values ​​with a value greater than zero are selected to calculate the mean corner response value of the image block. It should be noted that positive responses usually correspond to corners or highly stable structures, while negative responses are often not suitable as stable features.

[0067] When calculating the grayscale gradient magnitude in each image block, the gradient magnitude of each pixel within the image block is first normalized, mapping the gradient magnitude to the [0,1] interval. The number of pixels with a gradient magnitude greater than the gradient threshold is counted and designated as prominent pixels of the pixel block. In this implementation, the gradient threshold is set to 0.15, specifically based on the noise level of the image acquired by the industrial camera, the edge sharpness of the target, and the size of the image block. The total number of pixels in the pixel block is counted as the area of ​​the image block. The number of prominent pixels is counted and divided by the area to obtain the texture density of the image block. The higher the texture density, the richer the texture and the denser the features within the image block, making it suitable for tracking.

[0068] The mean gradient intensity and mean corner response of the image patch are normalized, and then compared with...

[0069] The block-level stability of the image block is obtained by linearly weighting the texture density according to preset weights. It should be noted that in this embodiment, the average gradient intensity is normalized using the maximum and minimum gray-level gradient magnitudes in the image block. Similarly, the average corner response is normalized using the maximum and minimum gray-corner response values ​​in the image block. The weights are adjusted according to the specific scenario: when the target edge is clear, the weight of the average gradient intensity is increased; when the target has obvious corners, the weight of the average corner response is increased; when the target surface texture is rich, the weight of texture density is increased. In this embodiment, the weights of the average gradient intensity, the average corner response, and the texture density are set to 0.4, 0.4, and 0.2, respectively. Through this weighted fusion method, the block-level stability can simultaneously reflect three types of information: edge, corner, and texture, avoiding misjudgment caused by a single indicator.

[0070] A preset stability threshold is used to distinguish between target regions and background regions. Target regions typically possess high edge strength, prominent corner structures, and rich texture features, resulting in generally high block-level stability. Background regions, on the other hand, often have lower block-level stability due to less texture, less distinct edges, or a lack of stable corners. To balance target detection rate and background suppression capability, the stability threshold is set to 0.6 in this embodiment. The block-level stability of an image patch is compared with the stability threshold. If the block-level stability is greater than the threshold, the image patch is determined to be a foreground patch, meaning it belongs to the target region. If the block-level stability is less than or equal to the threshold, the image patch is determined to be a background patch, meaning it does not belong to the target region. A two-dimensional array Z with the same grid size as the image patch is constructed, expressed as:

[0071]

[0072] Where S is the block-level stability of the image patch, S th To establish a stability threshold, the image is converted into a block-level binary image representing the stability state. Through the above steps, three types of features—gradient, corner points, and texture—are first extracted from the pixel-level image, then fused into a block-level stability value. Finally, the image blocks are divided into foreground and background blocks based on the threshold, thus automatically selecting image blocks with high stability suitable for forming the target region. Since the target usually spans multiple adjacent image blocks, connected component labeling and region merging are performed based on the foreground blocks to obtain the complete target region.

[0073] Since the target block typically spans multiple adjacent image blocks, connected component labeling is required for these foreground blocks. A two-pass scanning algorithm is used to process the two-dimensional label array: Using 4-neighborhood to define the adjacency relationship between image blocks, the adjacent blocks of the current image block include its left, right, top, and bottom blocks; using 8-neighborhood, it includes the top-left, top-right, bottom-left, and bottom-right blocks in addition to the 4-neighborhood. The first scan traverses each foreground block from top to bottom and left to right, checking the labels of its left and top blocks. If all scanned adjacent blocks are unlabeled, a new temporary label is assigned to the current block; if only one adjacent block has a label, the current block inherits that label; if multiple different labels exist, the current block inherits the smaller label, and the equivalence relationship between labels is recorded. After the first scan, all foreground blocks obtain temporary labels, and an equivalence set of labels is established. During the second scan, the temporary labels are merged according to the equivalence relationship, assigning the same final label to blocks belonging to the same connected component, thus obtaining several independent connected components.

[0074] The area and shape constraint analysis is performed on each connected region, specifically including: counting each connected image block and summing the area of ​​all image blocks to calculate the connected area. A preset area range is set. In this embodiment, considering that the size of a single image block is set to 32×32 pixels and the target usually occupies an area of ​​about 100×100 pixels in the image, the area range is preferably set to 4 to 20 image blocks. When the area of ​​the connected region is less than the preset minimum area threshold, it indicates that the region is insufficient in size and may be local noise or a false detection block. When the area of ​​the connected region is greater than the preset maximum area threshold, it indicates that the region exceeds the expected scale of the target and may be an abnormal region formed by multiple foreground blocks being mistakenly connected, adjacent irrelevant structures being included, or multiple candidate regions being merged. Therefore, it is removed. Only connected regions that meet the preset area range are retained to remove noise regions with too small an area and background regions with too large an area.

[0075] Based on the connected regions retained after area filtering, their corresponding minimum bounding rectangles are drawn. Specifically, the minimum bounding rectangle refers to the smallest axis-aligned rectangle that can completely enclose the connected region without rotating the image coordinate axes. The width and height of the minimum bounding rectangle of each connected region are extracted, and the width is divided by the height to obtain the aspect ratio. The aspect ratio is used to measure whether the shape of the region is close to the expected geometric shape of the target. A preset aspect ratio range is set. In this embodiment, considering that the imaging shape of the target in the image is usually close to a square or approximately rectangular, the aspect ratio range is preferably set to 0.5 to 2.0. Furthermore, among the connected regions that meet the preset area range, only the connected regions that meet the preset aspect ratio range are retained to ensure that the overall shape of the candidate region is consistent with the expected geometric features of the target.

[0076] In real-world scenarios, a target may be segmented into multiple adjacent small connected regions due to changes in lighting, local occlusion, or threshold segmentation errors. Therefore, spatial merging is further performed on all connected regions that meet area and shape constraints. Specifically, the coordinates of the center points of the connected regions are extracted, and the Euclidean distance between the center points of the circumscribed rectangles of any two connected regions is calculated. This distance reflects the spatial proximity of the two connected regions on the image plane; the smaller the distance, the more likely the two regions are to originate from different parts of the same segmented target. To avoid merging irrelevant regions, a maximum allowable merging distance is set. This maximum allowable merging distance is based on the target and its size, specifically obtained by multiplying the merging coefficient by the target's reference size or expected scale. The merging coefficient controls the degree of leniency; in this embodiment, it is set to 1.5. When the Euclidean distance between two connected regions is less than or equal to the maximum allowable merging distance, it indicates that these two regions meet the spatial requirements for merging, and these two connected regions are recorded as candidate regions for merging. Among all candidate regions that meet the maximum allowable merging distance, [the following is selected]. The regions with the smallest Euclidean distance are merged to obtain a new connected region. After merging, the minimum bounding rectangle and center point are redefined. After one merge, the Euclidean distance calculation, merging judgment, and minimum distance selection are re-executed for the remaining connected regions. That is, the distance between the new connected region and other connected regions is calculated again, and two pairs of connected regions with the smallest Euclidean distance that meet the maximum allowed merging distance are found again. The above merging process is repeated. This process continues to iterate until there are no regions in the connected region set that meet the merging distance threshold. The remaining region output at this time is the final target candidate region. If the number of target candidate regions is one, it is directly determined as the target region. If the number of target candidate regions is greater than or equal to two, the target region is selected from them. Specifically, in order to determine the final target region from multiple candidate regions, a comprehensive matching score of the candidate regions is constructed. In this embodiment, the comprehensive matching score F is composed of stability matching item, area matching item, shape matching item, and position matching item, which can be specifically expressed as:

[0077]

[0078] in, Indicates the stability matching term. Indicates area matching items, Indicates shape matching items, This represents the location matching term; specifically: the stability matching term is calculated by averaging the block-level stability of all image patches contained within the target candidate region; the area matching term is calculated as follows:

[0079]

[0080] in, The reference area of ​​the target is determined by the target design dimensions or pre-calibration results;

[0081] Shape Matching Items The calculation is as follows:

[0082]

[0083] in, The reference aspect ratio of the target is determined by the target's design dimensions or pre-calibration results;

[0084] Location matching items The calculation is as follows:

[0085]

[0086] Where d represents the Euclidean distance between the target candidate region and the reference center point. This indicates the maximum allowable reference distance; the reference center point is the preset reference center point of the target, which is either the center of the initially manually selected target area or the center of the target area confirmed in the previous frame.

[0087] In each target candidate region, the target candidate region with the largest comprehensive matching score F is selected as the final target region. If multiple candidate regions have the same comprehensive matching score, the candidate region with the larger average stability matching term is selected first. If the stability matching term is still the same, the candidate region with the larger position matching term is selected first. If they still cannot be distinguished, the candidate region with the larger area matching term is selected. The minimum bounding rectangle boundary is calculated again for the final retained target candidate regions, and the bounding rectangle of the final target region is output. The center point coordinates of the bounding rectangle of the target region in each frame of the continuous image sequence are calculated and extracted. Thus, the center point coordinate sequence of the target region in each frame of the image is obtained, which constitutes the motion trajectory of the target on the pixel plane.

[0088] The sub-pixel localization module performs sub-pixel-level solutions based on the target's motion trajectory on the pixel plane, and further converts the pixel displacements into actual physical displacements; specifically including:

[0089] Within the target area defined by the circumscribed rectangle, several feature points are selected as the objects of displacement calculation. Specifically, the pixel range covered by the circumscribed rectangle of the final target area is used as the feature extraction range. Within the feature extraction range, the corner response values ​​and grayscale gradient magnitudes at each pixel position are normalized and weighted and summed according to a preset threshold to obtain the feature evaluation value. The sum of the weights is one. In this embodiment, the weights of the corner response values ​​and grayscale gradient magnitudes are set to 0.6 and 0.4, respectively. A preset feature scoring threshold is used to remove pixels with low stable feature evaluation values ​​and unclear local structures to avoid flat or noisy areas being mistakenly selected as feature points. In this embodiment, it is set to 0.65. Pixel positions that meet the feature evaluation value greater than the feature scoring threshold are recorded as candidate feature points. After obtaining the candidate feature points, non-maximum suppression processing is performed on the candidate feature points. That is, in the local neighborhood of each candidate feature point, only the pixel position with the largest stable feature evaluation value is retained as the local optimum, and neighboring points with smaller scores in its neighborhood are deleted, thereby avoiding the occurrence of multiple pixels in the same local area. To eliminate overly dense candidate feature points, a minimum spacing constraint is applied to the point set after non-maximum suppression. Specifically, the Euclidean distance between any two candidate feature points is calculated, and the distance is less than a preset minimum spacing threshold. In this embodiment, the minimum spacing threshold is preferably set to 4 pixels, or one-quarter of the side length of a single image block, to ensure that the retained feature points are evenly distributed within the target area and to avoid feature points concentrating near local edges or the same corner. At this point, only the candidate feature point with the larger stable feature evaluation value is retained, while the candidate feature point with the smaller feature evaluation value is deleted, thereby avoiding excessive clustering of feature points in local areas. Finally, the pixel positions after threshold filtering, non-maximum suppression, and minimum spacing constraint are used as feature points. These feature points are used to establish feature correspondences between subsequent consecutive frames and to calculate sub-pixel displacements. Through the combined processing of non-maximum suppression and minimum spacing constraint, the retained pixel positions are evenly distributed within the target area, ensuring the stability of the feature points while avoiding redundancy in subsequent matching caused by dense feature points.

[0090] Let the stable features selected from the t-th frame be expressed as: , where n represents the number of feature points participating in sub-pixel calculation, Let represent the pixel coordinates of the q-th feature point in frame t, where q represents the index of the feature point. The position of the target in the current frame is predicted using the coordinates of the target center point in the previous frame and the aforementioned pixel planar motion trajectory, resulting in the predicted center point of the target. A search window is constructed centered on this predicted center point, with a size of w×w, where w is a preset search side length, specifically determined based on the target's maximum displacement amplitude, sampling frame rate, and monitoring distance. Local image patches surrounding the corresponding feature point in the previous frame are used as templates to match candidate positions within the search window of the current frame. In this embodiment, normalized cross-correlation is used to calculate the similarity between the template and the candidate positions, yielding the correlation response value. Where u and v represent the candidate coordinates in the search window and the relevant response values. The larger the value, the more similar the candidate region of the current frame is to the template of the previous frame, and the more likely it is to be the corresponding location of the same physical feature point in the current frame; specifically, the formula for calculating the correlation response value is expressed as:

[0091]

[0092] in, Let represent the grayscale value at pixel (x, y) of the image in frame t-1. Let represent the grayscale value of the pixel at the candidate position (u,v) in the t-th frame image. template Average grayscale value of internal pixels This represents the average grayscale value of pixels within the candidate region of the current frame; by calculating the relevant response value for all candidate positions within the search window, the integer pixel position with the largest relevant response value is taken as the integer pixel matching position of the qth feature point;

[0093] Since the integer pixel matching results are still limited by pixel discretization, to further improve the accuracy of displacement measurement, a local response surface is constructed around the integer pixel matching position, and sub-pixel interpolation is performed on the relevant response peaks. Specifically, the response values ​​of the relevant response peak points and their left and right adjacent positions are extracted, and the sub-pixel offset in the horizontal direction is calculated based on the trend of response value changes. At the same time, the response values ​​of the peak points and their upper and lower adjacent positions are extracted, and the sub-pixel offset in the vertical direction is calculated based on the trend of response value changes. The sub-pixel offsets are superimposed with the integer pixel matching positions to obtain the sub-pixel coordinates of the feature points in the current frame. The sub-pixel coordinates of the feature points in the current frame are compared with the sub-pixel coordinates of the corresponding feature points in the previous frame to obtain the sub-pixel displacement of the feature points between two consecutive sampling times. Since the target area usually contains multiple feature points, to reduce the impact of mismatch of a single feature point, the displacement results of all feature points are fused. Specifically, the feature point weights are determined according to the block-level stability of the image block where the feature point is located, and the displacement of the target as a whole in the pixel plane is calculated by weighted averaging, thereby obtaining the sub-pixel displacement results of the target in the horizontal and vertical directions.

[0094] Camera calibration is performed beforehand to obtain pixel scale coefficients. Pixel scale coefficients represent the actual physical length corresponding to a unit pixel. The overall sub-pixel displacement of the target is converted to the pixel scale coefficients in the corresponding directions to obtain the horizontal and vertical displacements of the target in actual space. Further, the horizontal and vertical displacements are synthesized to obtain the actual displacement value of the target. Through these steps, a high-precision conversion from the pixel-planar motion trajectory of the target to its actual physical displacement is achieved, providing sub-pixel-level displacement measurement capabilities. This meets the real-time monitoring needs of minute displacement changes in engineering structures such as bridges, high-rise buildings, and tunnels.

[0095] The output display module receives the displacement results from the subpixel positioning module and outputs the real-time displacement information of the target in the form of numerical display, curve display, and status prompts; specifically including:

[0096] The system receives the real-time displacement values ​​of the target from the sub-pixel positioning module and establishes displacement time-series data according to the image acquisition time sequence. The displacement time-series data includes at least the acquisition time, target center point coordinates, horizontal displacement, vertical displacement, and composite displacement value. The system displays the displacement results at the current moment in real time, including the target's current displacement value, displacement direction, and displacement rate. Simultaneously, it dynamically plots displacement change curves in time sequence, allowing monitoring personnel to intuitively observe the structural displacement change trend. The system calculates the displacement rate based on the displacement change and time interval between two adjacent sampling moments and displays the displacement rate synchronously. When the structure vibrates, it calculates dynamic response parameters such as vibration period, vibration frequency, and vibration amplitude based on continuous displacement data for structural operation status analysis.

[0097] A pre-set structural safety displacement threshold is established, and the real-time displacement value is compared with the safety displacement threshold. When the real-time displacement value exceeds the safety displacement threshold, an over-limit alarm is generated. Specifically, the over-limit alarm information includes the alarm time, the current displacement value, the over-limit magnitude (the difference between the real-time displacement value and the safety displacement threshold), and the corresponding image frame number. At the same time, an audible and visual alarm, information push, or remote monitoring platform alarm is triggered to remind monitoring personnel to handle the situation in a timely manner.

[0098] Displacement time-series data, alarm records, and corresponding image data are synchronously stored in a database to form a historical monitoring archive. This archive is used for subsequent structural operation trend analysis, anomaly tracing, and structural health status assessment. Through these steps, real-time display, dynamic analysis, over-limit early warning, and historical data storage of target structural displacement results are achieved, providing reliable data support for structural health monitoring and safety assessment.

[0099] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A device for calculating subpixel displacement based on image vision, comprising an image acquisition module, wherein the image acquisition module is used to acquire continuous images of a target structure and preprocess the continuous images; characterized in that, Also includes: The recognition and tracking module constructs block-level stability based on grayscale gradient information, corner response information, and texture density information in continuous images. Based on this block-level stability, it identifies the target region and establishes the pixel-plane motion trajectory of the target in the continuous images. Specifically, the recognition and tracking module divides the image into several image blocks, calculates the grayscale gradient magnitude and mean corner response of each pixel position in each image block, and then averages these values ​​to obtain the mean gradient intensity and mean corner response. Based on the grayscale gradient magnitude of each pixel position in the image block, it calculates the texture density of the image block, and then fuses the mean gradient intensity, mean corner response, and texture density according to preset weights to obtain the block-level stability. The subpixel localization module filters feature points within the target area, obtains the subpixel coordinates of the feature points through continuous frame matching, calculates the subpixel displacement of the target, and converts it into actual physical displacement.

2. The image visual subpixel displacement calculation device according to claim 1, characterized in that: The recognition and tracking module divides the image block into foreground and background blocks based on the comparison between block-level stability and a preset stability threshold, and constructs a block-level binary image based on the determination results of all image blocks.

3. A device for calculating sub-pixel displacement based on image vision according to claim 2, characterized in that: The identification and tracking module uses a connected component labeling algorithm to cluster the foreground blocks in the block-level binary image to obtain multiple connected components. The connected components are then filtered according to a preset area range and a preset aspect ratio range, retaining the connected components that satisfy the area range and aspect ratio range.

4. A device for calculating sub-pixel displacement based on image vision according to claim 3, characterized in that, The identification and tracking module calculates the center distance between each pair of connected regions after filtering and retaining each connected region, and performs region merging step by step in order of increasing center distance. Each time, it merges a pair of connected regions with the smallest center distance that meets the preset merging distance threshold, updates the merged connected regions, and repeats the above process until there are no more connected region pairs that meet the merging distance threshold. The finally obtained connected regions are used as target candidate regions.

5. A device for calculating sub-pixel displacement based on image vision according to claim 4, characterized in that: If there is only one target candidate region, it is directly determined as the final target region. If there are two or more target candidate regions, the identification and tracking module constructs a comprehensive matching score for the target candidate regions based on stability matching, area matching, shape matching, and position matching, and selects the target candidate region with the largest comprehensive matching score as the final target region.

6. A device for calculating sub-pixel displacement based on image vision according to claim 1, characterized in that: The subpixel localization module calculates a feature evaluation value within the target area based on the corner response value and grayscale gradient magnitude of each pixel position, and filters candidate feature points based on the comparison result of the feature evaluation value and the preset feature scoring threshold; it then performs non-maximum suppression and minimum spacing constraint processing on the candidate feature points to obtain the feature points.

7. A device for calculating sub-pixel displacement based on image vision according to claim 6, characterized in that: The subpixel localization module uses local image blocks centered on each feature point in the previous frame as matching templates. Within the search window of the current frame, it uses a normalized cross-correlation matching method to determine the integer pixel position that best matches each matching template, which is then used as the integer pixel matching position of the corresponding feature point.

8. A device for calculating sub-pixel displacement based on image vision according to claim 7, characterized in that: The subpixel localization module constructs a local response surface in the local neighborhood of the integer pixel matching position of each feature point based on the response values ​​of the relevant response peak points and their adjacent pixels. It calculates the subpixel offsets in the horizontal and vertical directions by interpolation, and superimposes the subpixel offsets with the integer pixel matching positions to obtain the subpixel coordinates of the feature point.

9. A device for calculating sub-pixel displacement based on image vision according to claim 1, characterized in that: The subpixel localization module determines the weight of each feature point based on the block-level stability of the image block in which each feature point is located, performs a weighted average fusion of the displacement results of each feature point to obtain the overall displacement of the target on the pixel plane, and then converts the overall displacement into the actual physical displacement according to the pixel scale coefficient obtained by pre-calibration.

Images