Image processing method and device
By constructing scale space in image stitching, detecting feature points and using pipeline circuits to perform descriptive sub-matching operations, the problem of low image stitching calculation efficiency is solved, and efficient image stitching and real-time experience is achieved.
Patent Information
- Application Number
- CN202411320247.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-20
- Publication Date
- 2024-11-12
AI Technical Summary
The prior art has low computing efficiency during image stitching, which is difficult to meet real-time requirements, especially in the fields of security monitoring and vehicle cameras.
By constructing the scale space of the image, detecting feature points and generating descriptors, using pipeline circuits to perform descriptor matching operations, including difference, square and accumulation operations, quickly obtaining the matching distance, thereby realizing image stitching.
It improves the efficiency of image stitching, ensures the real-time experience of image stitching, and can quickly obtain feature points sets and stitch them to meet monitoring needs.
Smart Images

Figure CN118941447A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to an image processing method and device. Background Art
[0002] In the fields of security monitoring, vehicle-mounted cameras, etc., in order to meet the demand for large-field-of-view monitoring of the monitored area, a binocular video acquisition device with two cameras can be used to collect two sets of images with smaller fields of view, and these two sets of images with smaller fields of view can be stitched into one set of images with a larger field of view through image stitching technology.
[0003] To facilitate image stitching, the fields of view of the two cameras of the binocular video acquisition device are intersected, so that the two sets of images captured by the binocular video acquisition device have overlapping areas. The overlapping area contains the same object. By detecting feature points in the image, the boundary of the object can be identified. Then, by matching feature points, the same object in the two sets of images can be identified, and then the two sets of images can be stitched according to the position of the same object. In the above image processing, a large amount of calculation is required. The time required for software calculation through the image processing algorithm is long, the calculation efficiency is low, and it is difficult to meet the real-time requirements of monitoring. Summary of the invention
[0004] The present application provides an image processing method and device to solve the problem of poor real-time performance of image stitching.
[0005] In a first aspect, the present application provides an image processing method, comprising:
[0006] Acquire a first image and a second image to be stitched, wherein the first image and the second image are acquired by binocular acquisition;
[0007] Constructing a scale space corresponding to the first image and a scale space corresponding to the second image;
[0008] Detecting feature points in two scale spaces;
[0009] Generate a descriptor of the feature point, and store the number of layers and coordinate information of the descriptor in the scale space;
[0010] Performing a matching operation on the descriptors of the same layer in the two scale spaces through a pipeline circuit to obtain a matching distance between the two descriptors, wherein the pipeline circuit is configured to perform a difference operation, a square operation, and an accumulation operation on the two descriptors in sequence according to the coordinate information;
[0011] Matching the feature points in the two scale spaces according to the matching distance to obtain a feature point set;
[0012] The first image and the second image are spliced according to the feature point set to obtain a target image.
[0013] In some embodiments, the pipeline circuit includes a first-stage pipeline, a second-stage pipeline, a third-stage pipeline, and a fourth-stage pipeline, and the pipeline circuit performs a matching operation on the descriptors of the same layer in the two scale spaces to obtain a matching distance between the two descriptors, including:
[0014] Comparing the sizes of descriptors of the same layer in the two scale spaces in one dimension through the first-stage pipeline to obtain a comparison result;
[0015] Through the second-stage pipeline, performing a difference operation on the two descriptors in the current dimension according to the comparison result, so that the operation result is a positive number;
[0016] Performing a square operation on the result of the difference operation through the third-stage pipeline;
[0017] The calculation results of the square operation are accumulated through the fourth-stage pipeline, and when the calculation results of the square operation of all dimensions of the descriptor are accumulated, the matching distance between the two descriptors is obtained.
[0018] In some embodiments, performing a matching operation on the descriptors of the same layer in the two scale spaces through a pipeline circuit to obtain a matching distance between the two descriptors includes:
[0019] In one clock cycle, a difference operation is performed on one dimension of the descriptors of the same layer in the two scale spaces, and a square operation is performed on the two descriptors in the last dimension;
[0020] In the next clock cycle, the calculation results of the square operation are accumulated, and when the calculation results of the square operation of all dimensions of the descriptor are accumulated, the matching distance between the two descriptors is obtained.
[0021] In some embodiments, matching the feature points in the two scale spaces according to the matching distance to obtain a feature point set includes:
[0022] Obtaining the minimum matching distance and the second minimum matching distance between the descriptors of the same layer in the two scale spaces;
[0023] Shifting the next smallest matching distance left by one position in the register;
[0024] Compare the next smallest matching distance after left shift with the minimum matching distance;
[0025] If the second smallest matching distance after the left shift is greater than or equal to the minimum matching distance, it is determined that the two feature points corresponding to the minimum matching distance and the second smallest matching distance match each other, and the two matching feature points are added to the feature point set.
[0026] In some embodiments, constructing a scale space corresponding to the first image includes:
[0027] Performing one-dimensional row filtering and one-dimensional column filtering on the first image through a one-dimensional Gaussian filter to obtain a first-order image containing multiple layers of images;
[0028] Downsample one layer of the image;
[0029] The downsampled image is filtered through a one-dimensional Gaussian filter for one-dimensional row filtering and one-dimensional column filtering to obtain a second-order image containing multiple layers of images;
[0030] The images of adjacent layers of the same order are subtracted to obtain a scale space containing a new first-order image and a new second-order image.
[0031] In some embodiments, the step of performing one-dimensional row filtering and one-dimensional column filtering on the first image through a one-dimensional Gaussian filter to obtain a first-order image including multiple layers of images further includes:
[0032] Expands the filter coefficients of a one-dimensional Gaussian filter to integer filter coefficients.
[0033] In some embodiments, generating a descriptor of the feature point includes:
[0034] Calculate the ratio of the horizontal coordinate to the vertical coordinate of the gradient vector of the feature point;
[0035] Obtaining the main gradient direction of the feature point by comparing the ratio with fixed-point approximations of gradient vectors corresponding to a plurality of gradient directions;
[0036] Obtaining the gradient value of the main gradient direction according to the gradient value calculation formula corresponding to the main gradient direction;
[0037] A descriptor of the feature point is generated according to the gradient main direction and the gradient value of the gradient main direction.
[0038] The grayscale values of the overlapping pixels are adjusted according to the weighted average value.
[0039] In a second aspect, the present application provides an image processing device, comprising:
[0040] An acquisition module, used for acquiring a first image and a second image to be stitched, wherein the first image and the second image are acquired by a binocular acquisition method;
[0041] A scale space construction module, used to construct a scale space corresponding to the first image and a scale space corresponding to the second image;
[0042] A feature point detection module, used to detect feature points in a scale space corresponding to the first image and a scale space corresponding to the second image;
[0043] A descriptor generation module, used to generate a descriptor of the feature point, and store the number of layers and coordinate information of the descriptor in the scale space;
[0044] A matching distance calculation module, used for performing a matching operation on two descriptors of the same layer in the scale space through a pipeline circuit to obtain a matching distance between the two descriptors, wherein the pipeline circuit is configured to perform a difference operation, a square operation and an accumulation operation on the two descriptors in sequence according to the coordinate information;
[0045] A feature point set generation module, used for matching the feature points in the two scale spaces according to the matching distance to obtain a feature point set;
[0046] The stitching module is used to stitch the first image and the second image according to the feature point set to obtain a target image.
[0047] In some embodiments, the matching distance calculation module includes:
[0048] A difference calculation unit, used to calculate the difference between two descriptors to be matched in each dimension;
[0049] A square calculation unit, used to calculate the square of the difference output by the difference calculation unit;
[0050] The accumulation operation unit is used to accumulate the squares output by the square operation unit.
[0051] In some embodiments, the matching distance calculation module further includes:
[0052] A comparison unit, used to compare the sizes of two descriptors to be matched in each dimension;
[0053] The difference operation unit is used to exchange the positions of the two descriptors to be matched in the difference operation according to the comparison result of the comparison unit.
[0054] It can be seen from the above technical content that the embodiment of the present application provides an image processing method and device. The method is for the first image and the second image to be spliced. After detecting feature points and generating descriptors through the constructed scale space, the descriptors of the same layer in the two scale spaces are matched by the pipeline circuit to obtain the matching distance. The matching operation includes difference operation, square operation and accumulation operation. The multi-stage circuit of the pipeline circuit can process different data in parallel separately, and the matching distance of the descriptor in different dimensions can be quickly obtained. Then, the feature point set is obtained according to the matching distance, and the first image and the second image are spliced according to the feature point set, which improves the efficiency of image splicing and ensures the real-time experience of image splicing. BRIEF DESCRIPTION OF THE DRAWINGS
[0055] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the embodiments are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.
[0056] Figure 1 A schematic diagram of the structure of an image processing system provided in an embodiment of the present application;
[0057] Figure 2 A schematic diagram of the process of an image processing method provided in an embodiment of the present application;
[0058] Figure 3 A schematic diagram of the decomposition of a two-dimensional Gaussian filter provided in an embodiment of the present application;
[0059] Figure 4 A schematic diagram of a Gaussian difference pyramid provided in an embodiment of the present application;
[0060] Figure 5 A schematic diagram of neighborhood division provided in an embodiment of the present application;
[0061] Figure 6 A schematic diagram of the structure of a pipeline circuit provided in an embodiment of the present application;
[0062] Figure 7 A schematic diagram of a flow chart of a pipeline circuit provided in an embodiment of the present application;
[0063] Figure 8 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0064] The following embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following embodiments do not represent all implementations consistent with the present application. They are only examples of systems and methods consistent with some aspects of the present application as detailed in the claims.
[0065] In order to improve the efficiency of image stitching, the embodiment of the present application provides an image processing solution based on a SOC (System on Chip) chip, which utilizes the programmable hardware circuit of the SOC chip to execute the calculation of the SIFT (Scale-Invariant Feature Transform) part of the algorithm through the programmed hardware circuit to achieve hardware acceleration effect.
[0066] See also Figure 1 , is a schematic diagram of the structure of an image processing circuit based on a SOC chip provided in an embodiment of the present application. In some embodiments, the image processing circuit includes a binocular video acquisition module, an image buffer module, an image display module and an image processing module.
[0067] A binocular video acquisition module, such as the OV5640 top-level module, is used to acquire the image sequence to be spliced. The binocular video acquisition module is electrically connected to a binocular video acquisition device, such as OV5640_land2. The binocular video acquisition device includes two cameras and a binocular adapter board. The two cameras are mounted on the binocular adapter board. The binocular adapter board relatively fixes the positions of the two cameras so that the two cameras have an overlapping area of about 30%-60%. The image sensor in the camera monitors the radiation information to generate an analog signal, which is converted into a digital signal after AD conversion. The digital signal is the image sequence to be spliced. The binocular video acquisition module transmits the image sequence to be spliced to the image cache module.
[0068] The image cache module, such as the DDR3 control module, is used to store the image sequence to be spliced transmitted by the binocular video acquisition module in the off-chip DDR3, and perform read and write operations on the image sequence to be spliced in the off-chip DDR3 according to the read signals of the image processing module and the image display module.
[0069] The image processing module is used to register and stitch the image sequence to be stitched.
[0070] The image display module is used to configure the information of image display, and output the spliced image to the display device through the HDMI port for display, so as to realize the real-time display of the spliced image.
[0071] In the above-mentioned image processing circuit, the image processing module executes the calculation of part of the algorithm of the image sequence to be spliced through the hardware circuit, thereby improving the image processing efficiency.
[0072] To illustrate the process of the image processing module processing the image sequence to be stitched, Figure 2 An image processing method is provided, which may include the following steps:
[0073] Step S100: obtaining a first image and a second image to be stitched, wherein the first image and the second image are acquired by binocular acquisition.
[0074] In some embodiments, two groups of image sequences to be stitched may be captured by two cameras of a binocular video capture device, and two images taken at the same time are obtained from the two groups of image sequences to be stitched, namely, the first image and the second image.
[0075] Step S200: constructing a scale space corresponding to the first image and a scale space corresponding to the second image.
[0076] In some embodiments, the SIFT algorithm extracts feature points of an image in a scale space, and the specific positions of the feature points will not change due to changes in the size of the scale space. Therefore, for the first image and the second image, scale spaces need to be constructed respectively.
[0077] The method for constructing the scale space includes: constructing a Gaussian pyramid corresponding to the source image; and subtracting the adjacent layer images of the same order of the Gaussian pyramid to obtain a Gaussian difference pyramid.
[0078] Since the method of constructing the scale space and the method of extracting feature points are the same for the first image and the second image, this embodiment only takes the first image as an example to illustrate the construction of the scale space and the extraction of feature points. The method of constructing the scale space and extracting feature points for the second image can refer to the processing process of the first image.
[0079] Taking the first image as an example, the construction process of the Gaussian pyramid includes:
[0080] The source image corresponding to the first image (such as the grayscale image of the first image) is Gaussian blurred to different degrees to obtain a first-order image containing multiple layers of images, wherein the first layer of the first-order image is the source image, and the blurring degrees of the second layer and subsequent images are increased in sequence.
[0081] Among them, the method of performing Gaussian blur is to perform low-pass filtering on the source image through a Gaussian function, and the calculation formula includes:
[0082] L(x,y,σ)=G(x,y,σ)*I(x,y)
[0083]
[0084] In the above formula, L(x, y, σ) represents the scale space function, G(x, y, σ) represents the Gaussian function, σ is the scale space factor, the larger the scale space factor, the higher the degree of blur, and different images of the same order have different scale space factors. I(x, y) represents the source image.
[0085] An image in the first-order image (such as the third-to-last layer image) is downsampled to obtain a first-layer image of the second-order image, and the first-layer image is Gaussian blurred to different degrees to obtain a second-order image containing multiple layers of images.
[0086] An image in the second-order image (such as the third-to-last layer image) is downsampled to obtain a first-layer image of the third-order image, and the first-layer image is Gaussian blurred to different degrees to obtain a third-order image containing multiple layers of images.
[0087] By analogy, a Gaussian pyramid containing multi-level images can be obtained, and each level of image includes multiple layers of images.
[0088] Since a large amount of data calculation is involved in the process of generating a Gaussian pyramid, the generation efficiency of the Gaussian pyramid and the calculation accuracy of the image data will directly affect the subsequent extreme point detection. In order to improve the generation efficiency of the Gaussian pyramid, this embodiment optimizes the construction of the scale space in combination with the properties of the Gaussian function, and implements the Gaussian filtering algorithm by adopting a parallel filtering circuit structure, thereby improving the filtering efficiency through the hardware circuit, and the specific implementation is as follows:
[0089] Using the property that the Gaussian function can be separated into rows and columns, the Gaussian function can be converted into the following format:
[0090] I(x,y)*G(x,y,σ)=I(x,y)*g(x,σ)*g(y,σ)
[0091] Based on the above principle, the filtering process of the Gaussian filter function can be further optimized in image processing, and the two-dimensional Gaussian filter can be decomposed into a combination of one-dimensional row filtering and one-dimensional column filtering with the same σ coefficient, that is, one-dimensional row filtering and one-dimensional column filtering are performed through a one-dimensional Gaussian filter, so as to reduce the amount of calculation. Figure 3 As shown, in the hardware circuit, the distributed two-dimensional matrix corresponding to the source image can be subjected to row filtering and column filtering, and the filtering order remains unchanged.
[0092] After obtaining the required Gaussian filter parameters, the row filter coefficients and column filter coefficients of the Gaussian filter function are obtained by the following formula:
[0093]
[0094] Since the Gaussian kernel function has a 3σ criterion, that is, 99.7% of the energy of the Gaussian kernel function is within a radius of 3σ, this embodiment uses 6σ as the sampling basis of the filter template, which can ensure the adequacy of the Gaussian filter to a greater extent. Taking 0=1.6, s=3 as an example, the template size is selected to round up 6σ=9.6 and use an odd number 11 as the template. The final Gaussian filter coefficient template is as follows:
[0095] {0.0018,0.0111,0.0431,0.1141,0.2051,0.2494,0.2051,0.2494,0.2051,0.1141,0.0111,0.0018}
[0096] In some embodiments, after the Gaussian filter coefficient template is convolved with the source image, a Gaussian image is obtained. The pixel values in the Gaussian image are all in decimal form, which is not conducive to storage. To solve this problem, the decimals can be fixed-pointed in advance, and the filter coefficients can be expanded by 1024 times to obtain integer filter coefficients. The integer filter coefficients are convolved with the source image, and the calculated results are enlarged and stored by shifting.
[0097] The Gaussian pyramid structure of this embodiment adopts the form of σ=1.6, s=3, and uses the semigroup property of the Gaussian kernel function to set the σ parameter of each layer according to the structural division of the Gaussian scale space in the SIFT algorithm as follows:
[0098]
[0099] By performing one-dimensional row filtering and one-dimensional column filtering on the first image, a first-order image containing multiple layers of images is obtained, one of the layers of images (such as the third-to-last layer image) is downsampled, and the downsampled image is used as the first layer image of the second-order image. The third-to-last layer image is subjected to one-dimensional row filtering and one-dimensional column filtering through a one-dimensional Gaussian filter to obtain a second-order image containing multiple layers of images.
[0100] After obtaining the Gaussian pyramid corresponding to the first image, the Gaussian difference function σ▽2G (Difference of Gaussian DOG operator) can be used to construct a Gaussian difference pyramid to improve the efficiency of feature point detection.
[0101] The method of using the DOG operator to construct a Gaussian difference pyramid is to subtract the adjacent layer images of the same order, which can significantly improve the algorithm speed. The calculation formula is as follows:
[0102]
[0103] D(x,y,δ)=[G(x,y,kδ)-G(x,y,δ)]*I(x,y)=L(x,y,kδ)-L(x,y,δ)
[0104] Among them, k is the ratio of the scale factors of adjacent layers of the same order, and its value is
[0105] The Gaussian scale space factor for the o-order s-layer is as follows:
[0106]
[0107] According to the principle of Gaussian pyramid, constructing Gaussian scale space requires determining the number of Gaussian pyramid groups, the number of layers in each group, and the standard deviation of the Gaussian kernel function. Different parameter selections will have a residual impact on the number and repeatability of extreme point detection in scale space. In order to further improve the accuracy of monitoring, the standard deviation set above is used to compare the number of feature points in different layers and groups. Finally, two groups of Gaussian pyramids are selected, each with five layers, and a four-layer Gaussian difference pyramid (DoG) is calculated, as shown in Figure 1. Figure 4 As shown, feature points are extracted from the four layers of Gaussian difference pyramid on the right.
[0108] Step S300: Detect feature points in the two scale spaces.
[0109] In some embodiments, in a Gaussian difference pyramid, for each layer of the image, each pixel point may be compared with adjacent pixel points to obtain a local extreme point.
[0110] In some embodiments, after obtaining local extreme value points, invalid extreme value points can be eliminated by using the maximum value suppression method and the ratio purification method, and the remaining local extreme value points are determined as feature points. For the convenience of distinction, the feature points of the first image are recorded as the first feature points, and the feature points of the second image are recorded as the second feature points.
[0111] Step S400: Generate a descriptor of the feature point, and store the number of layers and coordinate information of the descriptor in the scale space.
[0112] In some embodiments, for the sake of distinction, the descriptor of the first feature point is called the first descriptor, and the descriptor of the second feature point is called the second descriptor. The method of generating the first descriptor and the second descriptor is the same, so this embodiment only takes the first image as an example to illustrate the generation of the first descriptor, and the generation of the second descriptor can be obtained by referring to the generation of the first descriptor.
[0113] After obtaining the first feature point, the gradient direction and amplitude of the first feature point can be obtained by an approximate solution, so that the hardware circuit is easy to implement. The specific implementation method includes:
[0114] The gradient direction of the feature point is divided into 11.25° units, and 32 main direction intervals can be obtained. For the 32 direction intervals, a binary tree structure can be used in the hardware circuit later, and the result can be obtained after 5 clock cycles of calculation.
[0115] In some embodiments, for the sake of efficiency in later matching, the dimension of the descriptor may be reduced by merging adjacent gradient directions described by the sub-regions in pairs, thereby improving the efficiency of descriptor generation.
[0116] After dividing the direction interval into units of 11.25°, the gradient direction of the feature point can be obtained by calculating the tangent value at the boundary and the ratio of the horizontal coordinate component to the vertical coordinate component of the gradient vector (dx, dy).
[0117] In some embodiments, in order to better adapt to the design of hardware circuits, the ratio result of dx and dy is approximated by fixed-point approximation to obtain a fixed-point approximation. In this calculation process, only shift and addition operations are required, which can improve the working efficiency of the hardware circuit.
[0118] For a feature point, the ratio of the horizontal coordinate to the vertical coordinate of its gradient vector is compared with the fixed-point approximation of the gradient vectors corresponding to multiple gradient directions to obtain the main gradient direction of the feature point.
[0119] In some embodiments, the gradient value of the main gradient direction is obtained according to the gradient value calculation formula corresponding to the main gradient direction, wherein the gradient value calculation formula is shown in Table 1:
[0120] Table 1
[0121] Gradient interval direction Gradient value calculation formula 0°~11.25° <![CDATA[d x ]]> 11.25°~22.5° <![CDATA[(4-1 / 2)d x ]]> 22.5°~33.75° <![CDATA[(2+1 8)d x ]]> 33.75°~45° <![CDATA[1 / 2d x +d y ]]> 45°~56.25° <h2 style=";text-align:left;direction:ltr"><![CDATA[d <h2 style=";text-align:left;direction:ltr"> x <h2 style=";text-align:left;direction:ltr"> +1 / 2d<h2 style=";text-align:left;direction:ltr"> y <h2 style=";text-align:left;direction:ltr"> ]]><h2 style=";text-align:left;direction:ltr"> <![CDATA[56.25°~67.5 ° ]]> <![CDATA[(2+1 / 8)d x ]]> 67.5°~78.75° <![CDATA[(4-1 / 2)d x ]]> 78.75°~90° <![CDATA[d x ]]>
[0122] Taking the first quadrant as an example, the error between the calculated value and the actual amplitude can be reduced by approximate solution, and the gradient value calculation formula can be derived by hardware circuit implementation. This method can effectively reduce the calculation error. By comparing the settlement results of the traditional SIFT algorithm with Matlab software, the error of the gradient calculation result of each interval is extremely small, indicating that this approximate comparison method is efficient in solving the gradient, and consumes little hardware resources. The clock cycle required for hardware circuit implementation and the delay of the combinational circuit will be greatly reduced.
[0123] According to the main direction of the gradient and the gradient value of the main direction of the gradient, the descriptor of the feature point can be generated. Since the local properties of the feature point and the geometric transformation of the image have an impact on the neighborhood of the feature point, in order to ensure the rotation invariance of the descriptor of the feature point, it is necessary to confirm the main direction of the feature point and then transform the coordinate system on the coordinate system with the main direction of the feature point as the reference direction.
[0124] First, the gradient value m and direction of each pixel are solved in a small spatial area of 3σ around the feature point. The calculation formula is as follows:
[0125]
[0126] Based on the calculated gradient values, a gradient histogram is constructed, and the gradient direction is divided into 36 intervals with a step size of 10 degrees. The gradient direction corresponding to the peak of the histogram represents the main direction of the feature point. In order to improve the stability of the algorithm, the second direction with a gradient peak value of more than 80% is used as the secondary direction of the feature point to improve the quality of image matching.
[0127] After confirming the main direction of the feature point, the robustness of the SIFT algorithm can be enhanced by generating a description vector, i.e., a descriptor. A gradient direction distribution histogram of the descriptor of the feature point is generated from the Gaussian image of the scale where the feature point is located. And it is rotated to the feature direction of the feature point through the coordinate transformation matrix. The calculation formula is as follows:
[0128]
[0129] Each feature point is taken as the center point, a neighborhood containing 16×16 pixels is selected and divided into 4×4 small windows. Each small window is used for histogram statistics in eight directions to obtain the gradient value in each direction, and a total of 16×8 128-dimensional description vectors are obtained.
[0130] In some embodiments, when dividing the neighborhood of a feature point, a neighborhood structure composed of concentric rings may be used, thereby eliminating the need for rotation and linear interpolation operations, significantly reducing the complexity of the hardware circuit. Figure 5 When using circular area calculation, the area describing the vector calculation is the same before and after rotation, avoiding the effect of neighboring interference pixels. At the same time, since the calculation range before and after is the same, the calculation time can be greatly reduced and the efficiency of the algorithm is improved.
[0131] By utilizing the rotation invariance of the circular area, the horizontal coordinate of the gradient histogram before rotation is shifted to obtain the gradient histogram after rotation.
[0132] In some embodiments, to reduce the dimension of the description vector, the following steps may be used:
[0133] See also Figure 5 , taking the feature point as the center, select a circular area with a radius of 11S in the neighborhood of the feature point, and divide the circular area into 8 concentric circles with radii of 1, 2, 3, 4, 5, 7, 9, and 11, respectively, where S represents the scale of the layer where the feature point is located. Rotate the circular area in the main direction to make the image registration rotationally invariant.
[0134] Taking the feature points as the reference, the pixel gradient values in each direction with a step length of 45° are counted on 8 concentric circles to obtain a 64-dimensional description vector, which can be expressed as L = l1, l2, ... l 64 .
[0135] In some embodiments, the length of the description vector may be normalized. Through normalization, the influence of illumination on the accuracy of image stitching may be reduced. During normalization, in order to avoid using a large number of dividers resulting in low computational efficiency, the calculation formula for normalization set in the embodiment of the present application is as follows:
[0136]
[0137] 2 n ≤s≤2 n+1
[0138]
[0139] Among them, w is the 64-dimensional vector before normalization, w i (i=1,2,…,64) represents the i-th element of vector w, l is the normalized 64-dimensional vector, l i (i=1,2,…,64) represents the i-th element of vector L, and s is the binary norm of w.
[0140] Through the above-mentioned dimensionality reduction processing, the description vector is reduced from 128 dimensions to 64 dimensions, which can reduce the matching time of feature points and reduce the amount of data processing.
[0141] This embodiment divides the 16×16 area into 8 concentric sub-areas. This structure can effectively avoid interpolation calculations caused by image rotation, making it easier to implement hardware and reducing the need to calculate histogram statistics again, thus reducing resource consumption. Figure 5 As shown, it can represent the normalized gradient values in 8 gradient directions.
[0142] Step S500: performing a matching operation on two descriptors of the same layer in the scale space through a pipeline circuit to obtain a matching distance between the two descriptors, wherein the pipeline circuit is configured to perform a difference operation, a square operation and an accumulation operation on the two descriptors in sequence according to the coordinate information.
[0143] In some embodiments, the matching of feature points may be determined by matching of descriptors, and the matching of descriptors may be determined by matching distances.
[0144] In some embodiments, the matching distance includes the Euclidean distance. Since the calculation of the Euclidean distance needs to be performed at the same scale, it is necessary to obtain the layer number and coordinate information of the descriptors in the two scale spaces, and then the two descriptors in the same layer in the two scale spaces can be determined as the two descriptors to be matched, and the Euclidean distance of the two descriptors to be matched is calculated.
[0145] In some embodiments, the matching distance may include an approximation of the Euclidean distance. The Euclidean distance involves a square root operation, which will occupy a large amount of computing resources and storage resources. To solve this problem, an approximation of the Euclidean distance is used, such as the square of the Euclidean distance instead of the Euclidean distance, which can reduce resource usage and improve computing efficiency.
[0146] In some embodiments, in order to improve computational efficiency, a matching distance operation can be performed on two descriptors to be matched through a pre-set pipeline circuit. Pipeline operation can improve computational efficiency by decomposing a repetitive process into several sub-processes, each of which can run in parallel with other sub-processes. In an embodiment of the present application, by decomposing the matching distance operation into multiple levels of operations, such as difference operation, square operation and accumulation operation, each level of operation is performed by an instantiated circuit unit, and each circuit unit is executed independently and in parallel under the clock cycle, which can improve computational efficiency and data throughput.
[0147] For two 64-dimensional descriptors, after performing difference operation through the above pipeline circuit, the difference between the two descriptors to be matched in each dimension can be obtained; then through square operation, the square of the difference between the two descriptors to be matched in each dimension can be obtained; finally, through accumulation operation, the sum of the squares of the differences between the two descriptors to be matched in 64 dimensions can be obtained, and the sum of the squares is the matching distance.
[0148] In some embodiments, negative numbers may appear in the difference calculation, resulting in increased hardware circuit complexity. To solve this problem, a comparison operation can be performed before the difference operation to obtain the size comparison result of the two descriptors to be matched in each dimension, and the operation order of the two descriptors in the difference operation is adjusted according to the size comparison result, which can avoid negative numbers in the difference operation.
[0149] To further explain the above pipeline circuit, Figure 6 Schematic diagram of the structure of a pipeline circuit according to some embodiments is shown, Figure 6 As shown, the pipeline circuit includes a comparison unit, a difference operation unit, a square operation unit and an accumulation operation unit.
[0150] In some embodiments, the comparison unit is used to compare the size of two descriptors to be matched in each dimension. The comparison unit may include multiple comparison circuits. The comparison circuit includes two input terminals, the first input terminal inputs the value of the first descriptor in one dimension, and the second input terminal inputs the value of the second descriptor in one dimension. If the value of the first descriptor is greater than the value of the second descriptor, the comparison circuit outputs a high level; otherwise, the comparison circuit outputs a low level.
[0151] In some embodiments, the difference operation unit is used to calculate the difference between the two descriptors to be matched in each dimension. The difference operation unit may include multiple difference operation circuits. The difference operation circuit outputs a high level according to the comparison circuit, takes the value of the first descriptor as the subtrahend, takes the value of the second descriptor as the minuend, then performs a difference operation, and outputs the operation result; according to the comparison circuit outputs a low level, takes the value of the second descriptor as the subtrahend, takes the value of the first descriptor as the minuend, then performs a difference operation, and outputs the operation result.
[0152] In some embodiments, the square operation unit is used to calculate the square of the difference output by the difference operation unit. The square operation unit may include a plurality of multiplication circuits. The square operation circuit multiplies the operation result of the difference operation circuit by itself and then outputs the operation result.
[0153] In some embodiments, the accumulation operation unit is used to calculate the sum of squares of the difference between two descriptors to be matched in each dimension. The accumulation operation unit may include an accumulation circuit, which includes multiple input terminals and can accumulate the operation results of multiple multiplication circuits.
[0154] In some embodiments, the operation logic of each unit of the pipeline circuit can be input through Verilog statements to control the underlying hardware circuit to implement the functions of each unit of the pipeline circuit.
[0155] In some implementations, the operation flow of the pipeline circuit can be found in Figure 7 .like Figure 7 As shown, the pipeline circuit completes the matching distance calculation through a four-stage pipeline, wherein the first stage pipeline includes a comparison unit, the second stage pipeline includes a difference operation unit, the third stage pipeline includes a multiplication unit, and the fourth stage pipeline includes an accumulation operation unit.
[0156] In some embodiments, the first-stage pipeline obtains the number of pyramid levels and descriptor coordinates of the first image, as well as the number of pyramid levels and descriptor coordinates of the second image from the register in the 1st to 4th clock cycles, so that the matching distance of the first descriptor and the second descriptor of the same layer can be calculated in the subsequent clock cycles.
[0157] In some embodiments, in the fifth clock cycle, the comparison unit compares the sizes of the first descriptor and the second descriptor in the first dimension, and the difference operation unit performs a difference operation according to the comparison result. Point_m[4] represents the value of the first descriptor in the first dimension, Point_s[4] represents the value of the second descriptor in the first dimension, Point_m[5] represents the value of the first descriptor in the first dimension, Point_s[5] represents the value of the second descriptor in the first dimension, and so on.
[0158] In some embodiments, in the 6th clock cycle, the squaring operation unit multiplies the result of the difference operation between the first descriptor and the second descriptor in the first dimension by itself, and inputs the result of the self-multiplication operation to the accumulation operation unit for accumulation operation; at the same time, in this clock cycle, the comparison unit compares the sizes of the first descriptor and the second descriptor in the second dimension, and the difference operation unit performs a difference operation based on the comparison result.
[0159] By analogy, in the 68th clock cycle, the square operation unit multiplies the result of the difference operation between the first descriptor and the second descriptor in the 63rd dimension by itself, and inputs the result of the self-multiplication operation to the accumulation operation unit for accumulation operation in the next clock cycle; at the same time, in this clock cycle, the comparison unit compares the sizes of the first descriptor and the second descriptor in the 64th dimension, and the difference operation unit performs a difference operation based on the comparison result.
[0160] In the 69th clock cycle, the square operation unit multiplies the result of the difference operation of the first descriptor and the second descriptor in the 64th dimension by itself, and inputs the result of the self-multiplication operation to the accumulation operation unit for accumulation operation in the 70th clock cycle to obtain the operation result. When the operation results of the square operation of all dimensions of the descriptor are accumulated, the operation result is the matching distance between the first descriptor and the second descriptor. The matching distance is stored in the register for the next step of processing.
[0161] Step S600: matching feature points in the two scale spaces according to the matching distance to obtain a feature point set.
[0162] In some embodiments, the matching distance is used to preliminarily determine whether the first descriptor matches the second descriptor. For example, for a first descriptor, the matching distance between the first descriptor and each second descriptor is compared. If the matching distance between one of the second descriptors and the first descriptor is the smallest, and the matching distance is less than or equal to a preset threshold distance, then the first descriptor is determined to match the second descriptor. If the matching distance is greater than the preset threshold distance, then the first descriptor is determined to not match the second descriptor.
[0163] In some embodiments, in order to save storage resources, all matching distances of the first descriptor may not be stored. Instead, a minimum matching distance and a second minimum matching distance of the first descriptor may be stored respectively in the first register and the second register, and the minimum matching distance and the second minimum matching distance may be used to determine whether the first descriptor has a matching second descriptor.
[0164] For a first descriptor, after calculating the matching distance between the first descriptor and the first second descriptor of the second image, the count of the counter is accumulated, and then the matching distance is stored in the first register.
[0165] After calculating the matching distance between the first descriptor and the second descriptor of the second image, the count of the counter is accumulated, and then the matching distance is compared with the minimum matching distance in the first register. If it is less than the minimum matching distance, the current matching distance is updated to the first register to make it the new minimum matching distance. If it is greater than the minimum matching distance, the current matching distance is updated to the second register to make it the next smallest matching distance.
[0166] After calculating the matching distance between the first descriptor and the third descriptor of the second image, the count of the counter is accumulated, and then the matching distance is compared with the minimum matching distance in the first register. If it is less than the minimum matching distance, the current matching distance is updated to the first register to become the new minimum matching distance. If it is greater than the minimum matching distance, the current matching distance is compared with the second smallest matching distance in the second register. If it is less than the minimum matching distance, the current matching distance is updated to the second register to become the new second smallest matching distance. If it is greater than the second smallest matching distance, the current matching distance is not stored.
[0167] And so on, until the counter overflows, indicating that all matching distances of the first descriptor have been calculated, and the first register is updated based on the last calculated matching distance, or the second register is updated, or the first register or the second register is not updated. At this time, the data in the first register is determined as the minimum matching distance of the first descriptor, and the data in the second register is determined as the second minimum matching distance of the first descriptor.
[0168] In some embodiments, for a first descriptor, the first descriptor and a second descriptor having the smallest matching distance with the first descriptor form a descriptor pair.
[0169] To reduce mismatches, mismatched descriptor pairs can be eliminated based on the minimum matching distance and the second minimum matching distance.
[0170] In some embodiments, after obtaining the minimum matching distance and the second minimum matching distance of the first descriptor, the ratio of the minimum matching distance to the second minimum matching distance can be calculated, and the ratio can be compared with a threshold to determine whether the first descriptor mistakenly matches the second descriptor based on the comparison result.
[0171] Exemplarily, the threshold can be set to 2, that is, if the ratio of the minimum matching distance to the second minimum matching distance is greater than 2, the first descriptor is determined to be mismatched with the second descriptor; if the ratio of the minimum matching distance to the second minimum matching distance is less than or equal to 2, the first descriptor is determined to be matched with the second descriptor.
[0172] In some embodiments, since the division operation occupies a large amount of hardware resources, to solve this problem, the second smallest matching distance stored in the register can be shifted left by one bit, which is equivalent to multiplying the second smallest matching distance by 2. The shifted second smallest matching distance is compared with the maximum matching distance:
[0173] If the second smallest matching distance after the shift is greater than or equal to the maximum matching distance, indicating that the ratio of the minimum matching distance before the shift to the second smallest matching distance is less than or equal to 2, then the first descriptor is determined to match the second descriptor; if the second smallest matching distance after the shift is less than the maximum matching distance, indicating that the ratio of the minimum matching distance before the shift to the second smallest matching distance is greater than 2, then the first descriptor is determined to not match the second descriptor.
[0174] After removing the unmatched descriptor pairs, a feature point set is output according to the remaining descriptors, and the feature point set includes coordinate information and matching information of two feature points corresponding to the descriptor pairs.
[0175] Step S700: splicing the first image and the second image according to the feature point set to obtain a target image.
[0176] In some embodiments, since the first image and the second image each have an independent coordinate system, in order to stitch the first image and the second image, the first image and the second image may be converted to the same coordinate system to obtain a stitched image, ie, a target image.
[0177] In some embodiments, the homography matrix can map one pixel point to another pixel point on a two-dimensional plane, so as to change the position of the pixel point without changing the pixel value. Therefore, in order to achieve image stitching, the transformation parameters of the homography matrix can be determined according to the feature point set, and then the target coordinates of the pixel points in the first image and the pixel points in the second image in the same coordinate system, such as a newly created coordinate system, can be calculated respectively through the homography matrix with the determined transformation parameters.
[0178] In some embodiments, the target coordinates calculated by the homography matrix may be floating point numbers and cannot correspond to the position of a pixel. To solve this problem, the four nearest neighboring pixels can be bilinearly interpolated using a bilinear interpolation algorithm to redetermine the target coordinates.
[0179] In some embodiments, for pixel points in the non-feature point set, after the target coordinates are calculated, the pixel value of the pixel point can be stored in the memory address corresponding to the target coordinates; and for pixel points in the feature point set, since both the first image and the second image contain the pixel point, in order to avoid obvious seams in the target image caused by the brightness difference between the first image and the second image, the pixel values of the pixel points in the feature point set can be weighted, and the weighted sum of the pixel value of the pixel point in the first image and the pixel value of the pixel point in the second image is calculated, and the weighted sum is used as the pixel value of the pixel point in the target image to reduce the brightness difference between the pixel point and adjacent pixels and improve the image quality of the target image.
[0180] The present application also provides an image processing device. Figure 8 The image processing device may include an acquisition module, a scale space construction module, a feature point detection module, a descriptor generation module, a matching distance calculation module, a feature point set generation module and a splicing module.
[0181] The acquisition module is used to acquire a first image and a second image to be stitched, wherein the first image and the second image are acquired by binocular acquisition.
[0182] The scale space construction module is used to construct a scale space corresponding to the first image and a scale space corresponding to the second image.
[0183] The feature point detection module is used to detect feature points in a scale space corresponding to the first image and a scale space corresponding to the second image.
[0184] The descriptor generation module is used to generate the descriptor of the feature point and store the number of layers and coordinate information of the descriptor in the scale space.
[0185] A matching distance calculation module is used to perform a matching operation on two descriptors of the same layer in the scale space through a pipeline circuit to obtain a matching distance between the two descriptors. The pipeline circuit is configured to perform a difference operation, a square operation and an accumulation operation on the two descriptors in sequence according to the coordinate information.
[0186] In some embodiments, the matching distance calculation module may include a difference operation unit, a square operation unit and an accumulation operation unit. The difference operation unit is used to calculate the difference between two descriptors to be matched in each dimension; the square operation unit is used to calculate the square of the difference output by the difference operation unit; and the accumulation operation unit is used to accumulate the squares output by the square operation unit.
[0187] In some embodiments, the matching distance calculation module may further include a comparison unit, which is used to compare the sizes of the two descriptors to be matched in each dimension, and the difference operation unit is also used to swap the positions of the two descriptors to be matched in the difference operation according to the comparison result of the comparison unit.
[0188] A feature point set generation module, used for matching the feature points in the two scale spaces according to the matching distance to obtain a feature point set;
[0189] The stitching module is used to stitch the first image and the second image according to the feature point set to obtain a target image.
[0190] In some embodiments, the working process of each module of the image processing device can refer to the process description in the image processing method, which will not be repeated here.
[0191] It can be known from the above technical content that, for the first image and the second image to be spliced, after detecting feature points and generating descriptors through the constructed scale space, the embodiment of the present application performs a matching operation on the descriptors of the same layer in the two scale spaces through a pipeline circuit to obtain a matching distance. The matching operation includes a difference operation, a square operation, and an accumulation operation. The multi-stage circuit of the pipeline circuit can process different data in parallel independently, so as to quickly obtain the matching distance of the descriptor in different dimensions, and then obtain a feature point set according to the matching distance. The first image and the second image are spliced according to the feature point set, thereby improving the efficiency of image splicing and ensuring the real-time experience of image splicing.
[0192] Similar parts between the embodiments provided in this application can be referenced to each other. The specific implementation methods provided above are only a few examples under the general concept of this application and do not constitute a limitation on the protection scope of this application. For those skilled in the art, any other implementation methods expanded based on the scheme of this application without creative work belong to the protection scope of this application.
Claims
1. An image processing method, characterized in that: include: Acquire a first image and a second image to be stitched, wherein the first image and the second image are acquired by binocular acquisition; Constructing a scale space corresponding to the first image and a scale space corresponding to the second image; Detecting feature points in two scale spaces; Generate a descriptor of the feature point, and store the number of layers and coordinate information of the descriptor in the scale space; Performing a matching operation on the descriptors of the same layer in the two scale spaces through a pipeline circuit to obtain a matching distance between the two descriptors, wherein the pipeline circuit is configured to perform a difference operation, a square operation, and an accumulation operation on the two descriptors in sequence according to the coordinate information; Matching the feature points in the two scale spaces according to the matching distance to obtain a feature point set; The first image and the second image are spliced according to the feature point set to obtain a target image.
2. The image processing method according to claim 1, characterized in that: The pipeline circuit includes a first-stage pipeline, a second-stage pipeline, a third-stage pipeline and a fourth-stage pipeline. The pipeline circuit performs a matching operation on the descriptors of the same layer in the two scale spaces to obtain a matching distance between the two descriptors, including: Comparing the sizes of descriptors of the same layer in the two scale spaces in one dimension through the first-stage pipeline to obtain a comparison result; Through the second-stage pipeline, performing a difference operation on the two descriptors in the current dimension according to the comparison result, so that the operation result is a positive number; Performing a square operation on the result of the difference operation through the third-stage pipeline; The calculation results of the square operation are accumulated through the fourth-stage pipeline, and when the calculation results of the square operation of all dimensions of the descriptor are accumulated, the matching distance between the two descriptors is obtained.
3. The image processing method according to claim 1, characterized in that: The matching operation is performed on the descriptors of the same layer in the two scale spaces by the pipeline circuit to obtain the matching distance of the two descriptors, including: In one clock cycle, a difference operation is performed on one dimension of the descriptors of the same layer in the two scale spaces, and a square operation is performed on the two descriptors in the last dimension; In the next clock cycle, the calculation results of the square operation are accumulated, and when the calculation results of the square operation of all dimensions of the descriptor are accumulated, the matching distance between the two descriptors is obtained.
4. The image processing method according to claim 1, characterized in that: Matching the feature points in the two scale spaces according to the matching distance to obtain a feature point set, including: Obtaining the minimum matching distance and the second minimum matching distance between the descriptors of the same layer in the two scale spaces; Shifting the next smallest matching distance left by one position in the register; Compare the next smallest matching distance after left shift with the minimum matching distance; If the second smallest matching distance after the left shift is greater than or equal to the minimum matching distance, it is determined that the two feature points corresponding to the minimum matching distance and the second smallest matching distance match each other, and the two matching feature points are added to the feature point set.
5. The image processing method according to claim 1, characterized in that: The constructing a scale space corresponding to the first image includes: Performing one-dimensional row filtering and one-dimensional column filtering on the first image through a one-dimensional Gaussian filter to obtain a first-order image containing multiple layers of images; Downsample one layer of the image; The downsampled image is filtered through a one-dimensional Gaussian filter for one-dimensional row filtering and one-dimensional column filtering to obtain a second-order image containing multiple layers of images; The images of adjacent layers of the same order are subtracted to obtain a scale space containing a new first-order image and a new second-order image.
6. The image processing method according to claim 5, characterized in that: The first image is subjected to one-dimensional row filtering and one-dimensional column filtering through a one-dimensional Gaussian filter to obtain a first-order image containing multiple layers of images, and the process also includes: Expands the filter coefficients of a one-dimensional Gaussian filter to integer filter coefficients.
7. The image processing method according to claim 1, characterized in that: The generating of the descriptor of the feature point comprises: Calculate the ratio of the horizontal coordinate to the vertical coordinate of the gradient vector of the feature point; Obtaining the main gradient direction of the feature point by comparing the ratio with fixed-point approximations of gradient vectors corresponding to a plurality of gradient directions; Obtaining the gradient value of the main gradient direction according to the gradient value calculation formula corresponding to the main gradient direction; A descriptor of the feature point is generated according to the gradient main direction and the gradient value of the gradient main direction.
8. An image processing device, characterized in that: include: An acquisition module, used for acquiring a first image and a second image to be stitched, wherein the first image and the second image are acquired by a binocular acquisition method; A scale space construction module, used to construct a scale space corresponding to the first image and a scale space corresponding to the second image; A feature point detection module, used to detect feature points in a scale space corresponding to the first image and a scale space corresponding to the second image; A descriptor generation module, used to generate a descriptor of the feature point, and store the number of layers and coordinate information of the descriptor in the scale space; A matching distance calculation module, used for performing a matching operation on two descriptors of the same layer in the scale space through a pipeline circuit to obtain a matching distance between the two descriptors, wherein the pipeline circuit is configured to perform a difference operation, a square operation and an accumulation operation on the two descriptors in sequence according to the coordinate information; A feature point set generation module, used for matching the feature points in the two scale spaces according to the matching distance to obtain a feature point set; The stitching module is used to stitch the first image and the second image according to the feature point set to obtain a target image.
9. The image processing device according to claim 8, characterized in that: The matching distance calculation module includes: A difference calculation unit, used to calculate the difference between two descriptors to be matched in each dimension; A square calculation unit, used to calculate the square of the difference output by the difference calculation unit; The accumulation operation unit is used to accumulate the squares output by the square operation unit.
10. The image processing device according to claim 8, characterized in that: The matching distance calculation module also includes: A comparison unit, used to compare the sizes of two descriptors to be matched in each dimension; The difference operation unit is used to exchange the positions of the two descriptors to be matched in the difference operation according to the comparison result of the comparison unit.
