Loader front image splicing method based on splicing seam position calculation
By adopting an image stitching method based on the calculation of the slit position in front of the loader, the problems of unstable splicing effect and large calculation amount in the prior art are solved, efficient and stable image stitching is achieved, and the driver's operating vision is significantly improved and work efficiency.
Patent Information
- Application Number
- CN202411560312.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-04
- Publication Date
- 2025-02-14
AI Technical Summary
The prior art has problems in the stitching of images in front of the loader that are unstable, the calculation amount is large, the difficulty in real-time fusion, and the failure to effectively reduce the blind spots in the field of vision.
Using a method based on the slit position calculation, the bucket position and attitude are determined through the angle sensor and the camera, the main and secondary images are selected and the overlapping areas are determined, and the optimal slit position is found by using feature comparison and registration parameters to realize the image Fusion splicing.
The calculation amount is reduced, the splicing speed and efficiency are improved, the stability and optimality of the splicing results are ensured, and the driver's operating vision and operation efficiency are significantly improved.
Smart Images
Figure CN119444562A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of computer image information processing, and more particularly relates to a loader front image splicing method based on splicing position calculation. Background Art
[0002] As a kind of engineering machinery widely used in construction sites, mines, agriculture and other fields, the main function of loaders is to scoop up materials from one place to another through buckets. However, the bucket is usually located in front of the loader. In actual operation, the huge bucket in front of the loader will block the driver's line of sight, affecting the accuracy and safety of the driver's operation.
[0003] With the development of image synthesis technology, the image of objects in the obstructed area in front of the bucket can be determined by synthesizing the images taken in front of the loader, making the bucket transparent. The driver can understand the environment in front of the bucket more clearly, thereby improving the accuracy and safety of the operation. At the same time, it can also help the driver to better perform construction work and improve work efficiency. For the splicing of images in front of the loader, the existing technologies are as follows: Prior art 1: Patent CN116746140A proposes a feature-based method for image stitching, in which random distance selection is used for image stitching, image matching is calculated, and the optimal matching degree is selected to synthesize the front image more accurately. The solution uses an image processor to convert the two captured images into projection images according to multiple preset random distance values, and calculate the random distance with the highest matching degree, and use the distance information to generate a synthetic image, and adjust the weighted value of the image according to the position of the bucket, and output it to the display for display, thereby synthesizing the final image to ensure that the operator can clearly see the objects temporarily blocked by the bucket.
[0004] The disadvantages of prior art one are: 1. In the acquisition of distance information, relying on selecting random distances and calculating the highest matching degree does not necessarily lead to the optimal stitching effect, and the image stitching results obtained are unstable.
[0005] 2. When the position of the loader changes, the random distance information needs to be reset, and the projection image and matching degree need to be recalculated for multiple random distances respectively. This will cause a surge in the amount of calculation in a short time, requiring a large amount of computing resources, and it is difficult to ensure real-time fusion and stitching of the image.
[0006] 3. Whether the bucket is transparent is determined based on whether the bucket blocks the object in front, but it does not explain how to identify the object.
[0007] Prior art 2: Patent CN114998175A proposes an image stitching method based on regional edge strength. The edge strength of the image is obtained through pixel-by-pixel convolution operation, the regional edge strength of each block is calculated after the image is divided into blocks, and the difference in edge strength between the two images is compared to determine their respective weights. It supports parallel processing and then performs weighted fusion to generate the final stitched image.
[0008] The disadvantages of the prior art 2 are: 1. This solution mainly focuses on image stitching technology and does not specifically address the specific environment in which the loader is located after removing the front bucket. Therefore, it is not effective in reducing the blind spot behind the bucket.
[0009] 2. In this method, the weights of the two source images vary greatly in a very small area, which can easily lead to sudden changes in image grayscale, drastic edge transitions, flash points, and flash edges. Especially in video streams, the visual effect is poor, and it is generally used for static image processing.
[0010] 3. This method supports parallel processing, but it requires multiple image up- and down-sampling during the calculation process, which has high computational complexity and requires a large amount of computing resources, making it difficult to achieve complete parallel processing.
[0011] The second existing technology is a general image stitching method, which is not suitable for the scene of the loader. Therefore, a new image stitching processing method is needed to solve the blind spot of the field of vision behind the loader bucket.
[0012] In addition, the traditional solution only focuses on image stitching, and additional processing is required for stitching after removing objects from the scene. In addition, if the traditional image stitching solution is adopted, only two images are stitched into one image, which can only reduce some blind spots, but the blind spots blocked by the bucket still exist, and it cannot optimize the driver's operating efficiency. Summary of the invention
[0013] Therefore, the present invention proposes a method for stitching images in front of a loader based on seam position calculation. The seam position calculation adopts the method of first estimating the stitching area, and then finding the seam based on the features for image stitching. It is a method of finding the seam and realizing image stitching by combining the area and the features. The present invention aims to solve the problems of unstable stitching results and suboptimal stitching effects of traditional images in front of the loader, and to achieve the optimal stitching effect of the images in front of the loader. The core of the present invention is to quickly find the main image and the optimal seam position of the two images in image stitching, so as to stitch the image of the transparent bucket in front of the loader together with the visual information of the upper and lower cameras. Since the bucket picture is not selected during stitching, but the bucket boundary is marked, the spliced image does not contain the bucket, but the bucket boundary can be seen, which significantly enhances the operator's field of vision and work efficiency.
[0014] Therefore, the specific technical solution of the present invention is: A method for stitching images of a loader in front of the vehicle based on stitching position calculation is performed in the following steps: 1) Based on the angle sensor and upper and lower cameras installed in front of the loader, the position and posture of the bucket in the upper and lower camera images are determined, and the boundaries of the bucket in the images are marked; 2) Determine the main image and the secondary image based on the proportion of the bucket occupying the image screen; 3) Determine the overlapping area of the main image and the secondary image; 4) Determine the stitching position of the main image and estimate the initial stitching position of the secondary image; 5) Compare the features of the secondary image with the primary image to obtain the registration parameters; 6) Based on the optimal stitching, the fusion stitching of the two images is completed.
[0015] Specifically, the upper camera is installed on the top of the front of the vehicle, and the lower camera is installed on the bottom of the front of the vehicle, and the two cameras form an oblique cross shooting of the front of the loader.
[0016] Specifically, there are two angle sensors, one is installed at the connection point between the boom and the vehicle body, and the other is installed at the connection point between the boom and the bucket.
[0017] Specifically, the principle for determining the main image and the auxiliary image is: select the image with the smaller image area occupied by the bucket as the main image.
[0018] Specifically, the method for determining the overlapping area of the main image and the auxiliary image is: If the image taken by the upper camera is selected as the main image, the area above the upper boundary of the bucket in the main image is retained as the main image, and the area below the lower boundary of the bucket in the secondary image is retained as the secondary image; If the image taken by the lower camera is selected as the main image, the area below the lower boundary of the bucket in the main image is retained as the main image, and the area above the upper boundary of the bucket in the secondary image is retained as the secondary image; The intersection area between the main image and the auxiliary image is retained as the overlapping area.
[0019] Specifically, the stitching positions of the main and auxiliary images are based on the boundary where the bucket is located in the image; If the main image is selected as the image taken by the upper camera, the stitching position of the main image is determined as the image interface tangent to the upper boundary of the bucket; If the main image is selected as the image taken by the lower camera, the stitching position of the main image is determined as the image interface tangent to the lower boundary of the bucket; If the secondary image is selected as the image taken by the lower camera, the initial stitching position of the secondary image is determined as the image interface tangent to the lower boundary of the bucket; If the secondary image is selected as the image taken by the upper camera, the initial stitching position of the secondary image is determined as the image interface tangent to the upper boundary of the bucket.
[0020] Specifically, the formula for comparing the secondary image features with the primary image features to obtain the registration parameters is: ; in, is the registration parameter value, A and Z represent the images taken by the upper and lower cameras respectively, A i and Z i denote the i-th point image in A and Z respectively, Indicates point A i is the characteristic value of the reference point translated by x pixels horizontally and y pixels vertically. Indicates point Z i is the characteristic value of the reference point translated by x pixels horizontally and y pixels vertically. A positive x indicates horizontal rightward, a positive y indicates vertical upward, and vice versa indicates leftward and downward respectively. m and n are custom parameters, which respectively represent the limited values of the horizontal and vertical index ranges in the field of image processing; The closer it is to 0, the higher the similarity between the two images. The represented seam is taken as the optimal seam.
[0021] More specifically, After normalization, the corrected registration parameters are obtained as follows. In practice, the corrected registration parameters are used as the standard to obtain the optimal stitching: ; in, To correct the registration parameters, Indicates point A i is the absolute value of the gradient when the reference point is translated by x pixels horizontally and y pixels vertically. Indicates point Z i It is the absolute value of the gradient when the reference point is translated by x pixels horizontally and y pixels vertically. A positive x indicates horizontal rightward, a positive y indicates vertical upward, and vice versa indicates leftward and downward respectively.
[0022] Specifically, the process of obtaining the registration parameters includes two processes: obtaining the coarse registration parameters and the fine registration parameters: In the process of obtaining the coarse registration parameters, the range of the extracted feature points is relatively wide, and the step length of the translation of the sub-image stitching reference line is relatively large. In the process of obtaining the fine registration parameters, the range of the extracted feature points is relatively narrow, and the step length of the translation of the sub-image stitching reference line is relatively small. Several groups of coarse registration parameters that are considered to be better are selected from several groups, and the secondary image stitching reference lines corresponding to these groups of better coarse registration parameters are used as references for obtaining fine registration parameters.
[0023] Specifically, when the main image and the auxiliary image are fused and spliced, the two images are spliced and joined and then widened by a number of pixel widths on both sides, and image fusion is performed in the widened area.
[0024] It can be seen that the beneficial effects brought by the technical solution of the present invention are: 1. Reduced computational effort: Compared with traditional methods, this technology changes the determination of the stitching position from comparing the two images to the determination of the stitching position of the main image, while the secondary image is searched based on the features of the stitching position of the main image. This improvement greatly reduces the computational effort, improves the stitching speed and efficiency, and significantly reduces the computational complexity and resource consumption.
[0025] 2. The stitching result is globally optimal: By traversing and optimizing the stitching of the overlapping area of the image in front of the bucket, it can ensure that the stitching result is globally optimal. By comprehensively considering the characteristics of the overlapping area, the optimal stitching position is determined, resulting in a seamless and continuous stitching effect, improving the overall stitching quality and visual expression.
[0026] 3. The present invention uses the bucket position calculated in real time to perform image processing. There is no need to identify the object in front. It only needs to compare the overlapping areas and find the one with the highest similarity, thereby achieving the effect of improving image processing efficiency and reducing image delay. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] Figure 1 is a flow chart of the method of the present invention; Figure 2 It is a schematic diagram of the locations where the angle sensor and camera are installed on the loader; Figure 3 It is a schematic diagram of the overlapping area captured by the upper and lower cameras; Figure 4 It is a schematic diagram of feature comparison of the overlapping area between the main and secondary images. DETAILED DESCRIPTION
[0028] The present invention is described in detail below in conjunction with the embodiments. Those skilled in the art should know that the following embodiments are not the only limitations on the technical solutions of the present invention, and any equivalent changes or modifications made within the spirit of the technical solutions of the present invention should be deemed to fall within the protection scope of the present invention.
[0029] The specific implementation of the method of the present invention is: two upper and lower cameras and at least two angle sensors are arranged in front of the loader, wherein the position of the bucket is located by the angle sensor. According to the three-dimensional spatial position of the bucket relative to the loader, the position of the bucket in the upper and lower camera images is calculated, and the boundary of the bucket is marked in the image; then, based on the proportion of the bucket in the upper and lower camera images, the main image and the secondary image are determined, and then the overlapping area is determined based on the bucket boundary of the upper and lower images; after determining the upper and lower boundaries of the overlapping area; then, based on the main image and the bucket boundary, the stitching position of the main image is estimated, and feature extraction is performed to obtain the specific stitching position of the main image, and at the same time, the initial stitching position of the secondary image is determined based on the secondary image and the boundary of the bucket, and the registration parameters are calculated and recorded; then, the stitching position of the secondary image is moved with a larger step size, rough registration is performed, the registration parameters are calculated, the registration parameters are recorded, and this step is repeated multiple times; then, several groups of areas with the best rough registration parameters are selected, and fine registration is performed, and the stitching position of the main and secondary images determined based on the best registration parameters is moved with a smaller step size, and fine registration is performed, and then the registration parameters are calculated and recorded, and the stitching position of the main and secondary images determined based on the best registration parameters is obtained, and finally, image fusion is performed at the stitching position of the main and secondary images to complete the stitching of the main and secondary images.
[0030] The whole process is detailed as follows Figure 1 As shown, the steps include: 1. Determination of bucket position The present invention can determine the position and posture of the bucket based on multiple angle sensors set on the loader, or other methods can be used. The boundary of the bucket can be projected and marked in the images of the upper and lower cameras through the obtained position and posture. Figure 2 As shown, generally speaking, it is sufficient to set two angle sensors. The first angle sensor 1 is set at the connection point between the boom and the vehicle body to detect the front position of the bucket, and the second angle sensor 2 is set at the connection point between the boom and the bucket to detect the flip angle of the bucket. The angle information sensed by these two sensors and the three-dimensional model information of the bucket itself can accurately depict the three-dimensional spatial position information of the bucket.
[0031] The present invention sets two cameras in front of the loader, the upper camera 3 can be located at the top of the vehicle head, and the lower camera 4 can be located at the bottom of the vehicle head, provided that the two cameras can obliquely shoot the front of the loader, and the generated images can partially overlap. The purpose of the present invention is to form an image of a transparent bucket in front of the loader by splicing the two images, thereby improving the operating efficiency of the driver. For a determined bucket position, there must be a set of optimal image splicing parameters for the upper and lower cameras to achieve the optimal overlap elimination and bucket transparency effect.
[0032] 2. Selection of primary and secondary images In the images taken by the upper and lower cameras, it is obvious that the boundary of the loader bucket can be marked, and then the proportion of the bucket occupying the overall image can be determined based on the bucket boundary to decide which image is used as the main image. The present invention selects the image with the smaller bucket area as the main image. Figure 2 As shown, when the loader is started, the bucket is close to the ground. At this time, the picture taken by the lower camera of the loader is almost completely occupied by the bucket, while only a very small part of the picture taken by the upper camera of the loader is the bucket. At this time, the image taken by the upper camera of the loader is selected as the main image.
[0033] 3. Determination of overlapping areas After selecting the main and auxiliary images, the overlapping area is determined by combining the bucket positions in the main and auxiliary images. The specific operation method is as follows: Figure 3 As shown, the upper boundary of the bucket photographed by the upper camera and the area above it are retained as the main image, and the lower boundary of the bucket photographed by the lower camera and the area below it are retained as the secondary image. Since the upper camera shoots toward the front and lower part of the bucket, and the lower camera shoots toward the front and upper part of the bucket, there is an overlapping area between the two. In this way, the main and secondary images contain all the information of the front image of the loader bucket, but there is a partial overlapping area, which is the area below the upper boundary of the bucket photographed by the upper camera and above the lower boundary of the bucket photographed by the lower camera. If the main image is selected as the image taken by the lower camera, the same processing method can be adopted.
[0034] 4. Determination of the main image stitching position and estimation of the initial stitching position of the secondary image The present invention uses the stitching position of the main image as a fixed position, and uses the stitching position of the secondary image as a dynamic change to gradually approach the stitching position of the main image, and achieve consistency with the stitching position of the main image as much as possible, and finally realize the fusion of the main and secondary images. The stitching positions of the main and secondary images are all based on the boundary where the bucket is located in the image. If the main image is selected as the image taken by the upper camera, the stitching position of the main image is determined as the image interface tangent to the upper boundary of the bucket. If the main image is selected as the image taken by the lower camera, the stitching position of the main image is determined as the image interface tangent to the lower boundary of the bucket.
[0035] In the present invention, the stitching position selected by the main image is fixed, and the stitching position selected by the sub-image is used as a reference. The reference position changes dynamically. If the sub-image is selected as an image taken by the lower camera, the initial stitching position of the sub-image is determined to be the image interface tangent to the lower boundary of the bucket. If the sub-image is selected as an image taken by the upper camera, the initial stitching position of the sub-image is determined to be the image interface tangent to the upper boundary of the bucket.
[0036] 5. Compare the features of the secondary image and the primary image to obtain the registration parameters and obtain the optimal stitching Further, feature extraction can be performed near the seam to obtain the optimal seam. Feature extraction is a relatively mature technology. It should be noted that general feature extraction is performed on the entire picture, while in the present invention, it is only necessary to find the features of the seam that has been determined in the main image in the secondary image, which greatly reduces the amount of calculation for image stitching.
[0037] The feature extraction is performed near the seam of the secondary image, and compared with the seam feature of the primary image to obtain the optimal registration parameter. The present invention intends to use the optimal registration parameter method to obtain the optimal seam, and the seam between the two corresponding to the optimal registration parameter is the optimal seam.
[0038] The method for extracting and comparing features of overlapping areas of images in the present invention is as follows: Figure 4 As shown, let the images taken by the two cameras be represented as A and Z respectively. i and Z i Respectively represent the i-th point image in A and Z, and their pixel feature values are represented as A(i) and Z(i). Indicates point A i is the characteristic value of the reference point translated by x pixels horizontally and y pixels vertically, where x is a positive value, indicating a horizontal translation of x pixels to the right, y is a positive value, indicating a vertical translation of y pixels upward, and x and y are negative values, indicating the opposite direction. Indicates point Z i is the feature value of the reference point when it is horizontally moved by x pixels and vertically moved by y pixels. Comparing the similarity of the pixels of the two images, the main and auxiliary images satisfy the Markov random field property, and its A i The pixel feature value of is only related to the pixel feature value of the neighboring pixels, but not to A i Pixels farther away have less correlation, so the joint probability in the overlapping area will have a peak value. Therefore, the optimal stitching corresponds to the joint probability of pixels in the stitching area is the largest. Based on this idea, in the combination stage of image stitching, the overlapping areas of the images are registered, and the calculation of the registration parameters is defined as shown in formula (1): (1); in, is the registration parameter value, and the parameters m and n are integers, which are defined as the limit values of the horizontal and vertical index ranges in the field of image processing, respectively, and are used to define the size of the local area where the image pixels are operated. Specifically, m represents the horizontal index range, that is, the value range of x, which determines the range of the reference pixel A in the image processing process. i / Z iThe pixel range operated along the horizontal direction (i.e., row direction) of the image is centered at A. This range extends m pixel units to the left and m pixel units to the right from the reference pixel point, thus forming a horizontal pixel band containing 2m+1 pixels. Similarly, n represents the vertical index range, i.e., the value range of y, which determines the vertical index range of the image in the image processing process with reference pixel point A as the center. i / Z i The pixel range operated along the vertical direction (ie, column direction) of the image is centered at n, and this range extends n pixel units upward and n pixel units downward from the reference pixel point, thereby forming a vertical pixel band containing 2n+1 pixels.
[0039] The formula defines a (2m+1)×(2n+1) pixel rectangular window. Within this window, the data of images A and Z are calculated accordingly to obtain the registration parameters , thus achieving similarity comparison of local regions. This parameterized method enables the size of the local calculation region to be flexibly adjusted during image processing to meet different application requirements. According to the properties of Markov random fields, its A i The value of is only related to the pixel feature value of the neighboring pixels. i With Z i Calculated The smaller the value, the more it means that A i With Z i The more similar the pixels are, the more likely they are the same pixel in the two images. On the contrary, when A i With Z i Calculated The larger the value, the more it indicates that A i With Z i The present invention uses the registration parameters of the main and auxiliary images to evaluate the similarity of the overlapping areas of the two images. The smaller it is, the more similar the pixels on both sides of the seam are subjectively. Therefore, when m and n take the same value, the registration parameter The closer it is to 0, the higher the image similarity.
[0040] However, this process retains the original gradient of each image. Even if the pixel difference is large, the registration parameter of this area is small. Therefore, the seam should not be divided here. Therefore, it is necessary to normalize the gradient of formula (1). After normalization, it is shown in formula (2): (2); in, To correct the registration parameters, Indicates point A i is the absolute value of the gradient when the reference point is translated by x pixels horizontally and y pixels vertically. Indicates point Z i It is the absolute value of the gradient when the reference point is translated by x pixels horizontally and y pixels vertically. A positive x indicates horizontal rightward, a positive y indicates vertical upward, and vice versa indicates leftward and downward respectively.
[0041] Corrected registration parameters after normalization Taking into account the influence of the gradients on each image, the registration parameters between all pixels in the overlapping area of the combined image are calculated and corrected, and the corrected registration parameters of the pixels in the overlapping area are obtained. Analysis shows that for obtaining the minimum corrected registration parameters The seam represented by the modified registration parameters is the optimal seam found, with the smallest The best stitching result can be obtained by stitching the mapped seams.
[0042] Based on the theoretical idea of the registration parameter method, the reference line of the secondary image stitching is continuously adjusted to move closer to the stitching position of the main image, gradually matching the optimal stitching. If the main image is taken by the upper camera, the stitching reference line of the lower camera image gradually moves downward; otherwise, it moves upward.
[0043] Therefore, the process of comparing the features of the secondary image with the main image to obtain the registration parameters includes the following two steps: 5.1) Calculate the coarse registration parameters After the initial stitching reference line of the secondary image is determined, the feature points are first extracted. The method of extracting the feature points is: extracting feature points within the range of 0.5Δh above and below the stitching reference line, using these feature points for rough registration calculation, and calculating the coarse registration parameters according to the above-mentioned modified registration formula. The coarse registration parameters describe the preliminary alignment relationship between the two images.
[0044] Then translate the stitching reference line of the secondary image, and set the step length of each translation of the stitching reference line of the secondary image to Δh. Δh is a custom parameter, which can be adjusted according to the needs of the actual application, the size of the image, the processing power or other related factors. If the secondary image is selected as the lower camera image, after each vertical downward translation of one step length, a rough alignment is performed, and feature points are extracted again to calculate the alignment parameters. If the secondary image is selected as the upper camera image, after each vertical upward translation of one step length, feature points are extracted again to calculate the alignment parameters until it is translated to the boundary of the main image, and several sets (optional 3-5 sets) of better alignment parameters are found as the result of the rough alignment.
[0045] The above process may also not extract feature points at the initial stitching reference line, but first perform translation and then extract.
[0046] 5.2) Finding detailed registration parameters According to the selected sets of better rough registration parameters, the area of 0.5Δh above and below the seams of these secondary images is selected for more detailed registration. In the fine registration area, a smaller step size is set, and the step size of each translation is reduced to 0.2Δh, so that the optimal seam position can be found more accurately.
[0047] In the same way, the secondary image stitching reference line corresponding to each set of coarse registration parameters is translated with a step size of 0.2Δh. After each translation, feature points are extracted within the range of 0.1Δh above and below the stitching reference line. After the feature points are extracted, they are used to calculate the fine registration parameters.
[0048] Use these areas for fine registration, translate with a smaller step size each time, and obtain new registration parameters, which describe a more specific alignment relationship between the two images. Fine registration is performed on the basis of coarse registration. Through smaller step size translation and feature point extraction, as well as more accurate feature point matching, the optimal registration parameters are obtained to eliminate the minor errors that may exist in coarse registration, ensuring that the final stitched image is as visually perfect as possible.
[0049] All the registration parameters are compared to obtain a set of optimal registration parameters. The secondary image seam reference line corresponding to the optimal registration parameters is the seam that is ultimately to be matched and fused with the primary image.
[0050] 6. Main and secondary images are merged to complete the stitching of the entire image The optimal seam of the secondary image is obtained according to the optimal registration parameters. After the optimal seam is obtained, the features of the primary and secondary images are extracted on both sides of the seam and added to complete the stitching of the entire image. Since the features of the primary and secondary images are extracted on both sides of the seam without superposition, it is equivalent to no overlapping area of the image, so there will be no ghosting, and the structural information of the original images is retained on both sides of the seam, so all the information behind the bucket can be seen in the stitching result. Since the boundary of the bucket is retained in the previous image, the boundary of the bucket can also be seen, so a "transparent bucket" is formed.
[0051] Furthermore, the stitching method can reduce structural distortions such as ghosting to a certain extent, but the stitching may still cause structural distortions in other situations, such as misalignment on both sides of the stitching, so the stitching still needs to be optimized. In this process, the image fusion method can be used to widen the stitching by several pixel widths (usually 16 pixels) on both sides, and then the image fusion method is used on the widened area of the stitching to further reduce the structural distortion. The image fusion method also belongs to the prior art and will not be described in detail here.
[0052] Based on the above embodiments, it can be seen that the main innovations of the present invention are: (1) The object to be removed (bucket) is marked on the boundary of the upper and lower camera images to determine the degree of obstruction of the front line of sight by the object, and the main and auxiliary images are selected accordingly; (2) Using a method combining regions and features to find the seam, the seam position of the main image is fixed, the seam position of the secondary image is used as a reference, and the similarity of pixel features is used as an evaluation parameter. The features in a very small area are searched and optimized, which can quickly find the optimal seam position. Compared with searching for the best fit between the two images, a lot of computational effort is saved. (3) By also considering the gradients of each image, a registration parameter correction algorithm is proposed to evaluate the matching degree of different image stitching.
Claims
1. A method for stitching images in front of a loader based on stitching position calculation, characterized in that: Follow these steps: 1) Based on the angle sensor and upper and lower cameras installed in front of the loader, the position and posture of the bucket in the upper and lower camera images are determined, and the boundaries of the bucket in the images are marked; 2) Determine the main image and the secondary image based on the proportion of the bucket occupying the image screen; 3) Determine the overlapping area of the main image and the secondary image; 4) Determine the stitching position of the main image and estimate the initial stitching position of the secondary image; 5) Compare the features of the secondary image with the primary image to obtain the registration parameters; 6) Based on the optimal stitching, the fusion stitching of the two images is completed.
2. The method for stitching images in front of a loader based on stitching position calculation according to claim 1, characterized in that: The upper camera is installed on the top of the front of the vehicle, and the lower camera is installed on the bottom of the front of the vehicle. The two cameras form an oblique cross shooting of the front of the loader.
3. The method for stitching images in front of a loader based on stitching position calculation according to claim 1, characterized in that: There are two angle sensors, one is installed at the connection point between the boom and the vehicle body, and the other is installed at the connection point between the boom and the bucket.
4. The method for stitching images in front of a loader based on stitching position calculation according to claim 1, characterized in that: The principle for determining the main and auxiliary images is: select the image with the smaller image area occupied by the bucket as the main image.
5. The method for stitching images in front of a loader based on stitching position calculation according to claim 1, characterized in that: The method for determining the overlapping area of the main and auxiliary images is: If the image taken by the upper camera is selected as the main image, the area above the upper boundary of the bucket in the main image is retained as the main image, and the area below the lower boundary of the bucket in the secondary image is retained as the secondary image; If the image taken by the lower camera is selected as the main image, the area below the lower boundary of the bucket in the main image is retained as the main image, and the area above the upper boundary of the bucket in the secondary image is retained as the secondary image; The intersection area between the main image and the auxiliary image is retained as the overlapping area.
6. The method for stitching images in front of a loader based on stitching position calculation according to claim 1, characterized in that: The joint positions of the main and auxiliary images are all based on the boundary of the bucket in the image as a reference; If the main image is selected as the image taken by the upper camera, the stitching position of the main image is determined as the image interface tangent to the upper boundary of the bucket; If the main image is selected as the image taken by the lower camera, the stitching position of the main image is determined as the image interface tangent to the lower boundary of the bucket; If the secondary image is selected as the image taken by the lower camera, the initial stitching position of the secondary image is determined as the image interface tangent to the lower boundary of the bucket; If the secondary image is selected as the image taken by the upper camera, the initial stitching position of the secondary image is determined as the image interface tangent to the upper boundary of the bucket.
7. The method for stitching images in front of a loader based on stitching position calculation according to claim 1, characterized in that: The formula for comparing the secondary image features with the primary image features to obtain the registration parameters is: ； in, is the registration parameter value, A and Z represent the images taken by two cameras respectively. Let A i and Z i denote the i-th point image in A and Z respectively, Indicates point A i is the characteristic value of the reference point translated by x pixels horizontally and y pixels vertically. Indicates point Z i is the characteristic value of the reference point translated by x pixels horizontally and y pixels vertically. A positive x indicates horizontal rightward, a positive y indicates vertical upward, and vice versa indicates leftward and downward respectively. m and n are custom parameters, which respectively represent the limited values of the horizontal and vertical index ranges in the field of image processing; The closer it is to 0, the higher the similarity between the two images. The represented seam is taken as the optimal seam.
8. The method for stitching images in front of a loader based on stitching position calculation according to claim 7, characterized in that: right After normalization, the corrected registration parameters are obtained as follows. In practice, the corrected registration parameters are used as the standard to obtain the optimal stitching: ； in, To correct the registration parameters, Indicates point A i is the absolute value of the gradient when the reference point is translated by x pixels horizontally and y pixels vertically. Indicates point Z i It is the absolute value of the gradient when the reference point is translated by x pixels horizontally and y pixels vertically. A positive x indicates horizontal rightward, a positive y indicates vertical upward, and vice versa indicates leftward and downward respectively.
9. The method for stitching images in front of a loader based on stitching position calculation according to claim 7 or 8, characterized in that: The process of obtaining the registration parameters includes two processes: obtaining the coarse registration parameters and the fine registration parameters; In the process of obtaining the coarse registration parameters, the range of the extracted feature points is relatively wide, and the step length of the translation of the sub-image stitching reference line is relatively large. In the process of obtaining the fine registration parameters, the range of the extracted feature points is relatively narrow, and the step length of the translation of the sub-image stitching reference line is relatively small. Several groups of better coarse registration parameters are selected from several groups, and the secondary image stitching reference lines corresponding to these groups of better coarse registration parameters are used as references for obtaining fine registration parameters.
10. The method for stitching images in front of a loader based on stitching position calculation according to claim 1, characterized in that: When the main image and the auxiliary image are fused and spliced, the two images are spliced and joined and then widened by a number of pixel widths on both sides, and image fusion is performed in the widened area.