Image splicing method and device of deviation rectification linear array camera, electronic equipment and medium

Abstract

The embodiment of the invention discloses an image splicing method and device of a deviation rectification linear array camera, electronic equipment and a medium. The method comprises the following steps: enabling at least two line-scan digital cameras to directly face a checkerboard calibration plate for image acquisition to obtain target images acquired by the line-scan digital cameras respectively; for target images acquired by two adjacent linear array cameras, determining adjacent checkerboard demarcation points as feature points according to a pixel point with the maximum pixel gradient in the target images and pixel points adjacent to the pixel point with the maximum pixel gradient; and according to the image coordinates of the feature points in the target image and the plane coordinates of the feature points in a plane coordinate system established by taking the checkerboard calibration plate as a reference, determining a perspective transformation matrix for transformation between the image coordinates and the plane coordinates. According to the scheme, under the condition that the line-scan digital camera keeps a large view field range, the images of the line-scan digital camera are precisely spliced, and the fusion degree of the spliced images is improved.

Description

Technical Field

[0001] The present application relates to the field of machine vision technology, and in particular, to an image stitching method, device, electronic device, and medium for a rectifying linear array camera. Background Art

[0002] In the modern industrial manufacturing system, accuracy is the core indicator for measuring production efficiency and product quality. With the booming development of automation and intelligent manufacturing technologies, the requirements for equipment accuracy, system stability, and response speed in the industrial production field are becoming increasingly stringent. In this context, the rectifying sensor, as a core component in the automatic control system, plays a crucial role, and its performance directly affects the stability of the production process and the quality control of the final product. A rectifying sensor can monitor the motion state of the target object in real time and achieve automatic rectifying operations through a precise feedback mechanism.

[0003] Currently, when the current rectifying sensor and measurement technology are faced with the requirements of large field of view and high-precision measurement, they are facing major challenges in terms of process and performance such as limited resolution improvement, significant accuracy loss, and a sharp increase in the complexity of the calibration process. These limitations severely restrict the applicable range and actual effect of these technologies in specific application scenarios. Therefore, exploring and developing new technologies and new methods that can break through the above bottlenecks is of crucial significance for promoting the further development of industrial automation production. Summary of the Invention

[0004] Embodiments of the present application provide an image stitching method, device, electronic device, and medium for a rectifying linear array camera, so as to improve the measurement accuracy of the linear array camera and the fusion degree of the stitched images of the linear array camera when the linear array camera forms a larger field of view range.

[0005] According to one aspect of the present application, there is provided an image stitching method for a rectifying linear array camera, the method including:

[0006] Causing at least two linear array cameras to collect images facing a checkerboard calibration board to obtain target images respectively collected by each linear array camera;

[0007] For the target images collected by two adjacent linear array cameras, determining the adjacent checkerboard demarcation point as a feature point according to the pixel gradient maximum pixel point in the target image and the adjacent pixel points of the pixel gradient maximum pixel point;

[0008] Determining a perspective transformation matrix for converting between the image coordinates and the plane coordinates according to the image coordinates of the feature point in the target image and the plane coordinates of the feature point in the plane coordinate system established based on the checkerboard calibration board, so as to perform coordinate transformation on each target data point in the target image based on the perspective transformation matrix to obtain a plane stitched image.

[0009] According to one aspect of the present application, there is provided an image stitching device for a deviation-correcting line array camera, the device comprising:

[0010] An image acquisition module, configured to cause at least two line array cameras to face a checkerboard calibration board for image acquisition, so as to obtain target images respectively acquired by each line array camera;

[0011] A feature point determination module, configured to, for the target images acquired by two adjacent line array cameras, determine adjacent checkerboard demarcation points as feature points according to the pixel points with the maximum pixel gradient in the target images and the adjacent pixel points of the pixel points with the maximum pixel gradient;

[0012] A perspective transformation matrix determination module, configured to determine a perspective transformation matrix for the conversion between the image coordinates and the plane coordinates according to the image coordinates of the feature points in the target images and the plane coordinates of the feature points in a plane coordinate system established based on the checkerboard calibration board, so as to perform coordinate transformation on each target data point in the target images based on the perspective transformation matrix to obtain a plane stitched image.

[0013] According to another aspect of the present application, there is provided an electronic device, the electronic device comprising:

[0014] At least one processor; and

[0015] A memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the image stitching method of the deviation-correcting line array camera according to any embodiment of the present application.

[0017] According to another aspect of the present application, there is provided a computer-readable storage medium, the computer-readable storage medium stores computer instructions, and the computer instructions are used to implement the image stitching method of the deviation-correcting line array camera according to any embodiment of the present application when executed by a processor.

[0018] In the technical solution of the embodiment of the present application, at least two linear array cameras are directed at a checkerboard calibration board for image acquisition to obtain target images respectively acquired by each linear array camera. For the target images acquired by two adjacent linear array cameras, according to the pixel points with the maximum pixel gradient in the target images and the adjacent pixel points of the pixel points with the maximum pixel gradient, the adjacent checkerboard demarcation points are determined as feature points, thereby making full use of the pixel value distribution characteristics of the checkerboard and the gradient characteristics of the pixel points to accurately determine the checkerboard demarcation points as feature points. According to the image coordinates of the feature points in the target images and the plane coordinates of the feature points in the plane coordinate system established with the checkerboard calibration board as the reference, the perspective transformation matrix for the conversion between the image coordinates and the plane coordinates is determined, thereby more accurately determining the transformation relationship between the image coordinates, being able to achieve linear and non-linear transformations, effectively eliminating the accuracy loss of non-linearity, and improving the accuracy of the linear array camera and the fusion degree of the stitched images.

[0019] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present application, nor is it used to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0021] Figure 1 It is a flowchart of an image stitching method for a deviation correction linear array camera provided by an embodiment of the present application;

[0022] Figure 2 It is a schematic diagram of the relative positions of a linear array camera and a checkerboard calibration board provided by an embodiment of the present application;

[0023] Figure 3 It is a schematic diagram of a checkerboard calibration board provided by an embodiment of the present application;

[0024] Figure 4 It is a flowchart of an image stitching method for a deviation correction linear array camera provided by another embodiment of the present application;

[0025] Figure 5 It is a flowchart of an image stitching method for a deviation correction linear array camera provided by yet another embodiment of the present application;

[0026] Figure 6 It is a gradient change line graph provided by an embodiment of the present application;

[0027] Figure 7 The structural schematic diagram of an image stitching device for a deviation-corrected linear array camera provided by an embodiment of the present application;

[0028] Figure 8 The structural schematic diagram of an electronic device provided by an embodiment of the present application. Specific embodiments

[0029] In order to enable those skilled in the art to better understand the solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0030] It should be noted that the terms "first", "second", "third", "fourth", "actual", "preset", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present application described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0031] Figure 1 The flowchart of an image stitching method for a deviation-corrected linear array camera provided by an embodiment of the present application. The embodiments of the present application are applicable to the situation of stitching images of a linear array camera. This method can be executed by an image stitching device of a deviation-corrected linear array camera. The image stitching device of the deviation-corrected linear array camera can be implemented in the form of hardware and / or software, and the image stitching device of the deviation-corrected linear array camera can be configured in an electronic device. As Figure 1 shown, the method includes:

[0032] S110. Cause at least two linear array cameras to face a checkerboard calibration board for image acquisition, and obtain target images respectively acquired by each linear array camera.

[0033] Among them, a line array camera is a camera that uses a line array image sensor. The line array camera performs continuous scanning along the moving direction of the object to be detected through a linear sensor. It continuously acquires images to achieve high-speed capture and high resolution. During the scanning process, each pixel unit sequentially captures a row of the image, and the entire image is obtained through mechanical movement (such as a conveyor belt or a scanning head). The line array camera in the embodiment of the present application can be a deviation correction line array camera. The deviation correction line array camera is mainly used to perform precise position or direction correction on continuously moving objects in a machine vision system. The checkerboard calibration board is a pre-set calibration board with a checkerboard pattern drawn on it. The checkerboard is a pattern composed of grids of black and white (or other color combinations). The shape of the grids is not limited and can be square, such as Figure 2 shown, triangular, diamond-shaped, hexagonal, etc. The material of the checkerboard calibration board can be selected according to the actual situation. For example, it can be plastic, wood, or glass. The material characteristics of the glass material include a small coefficient of thermal expansion and high hardness, ensuring high-precision manufacturing and long-term dimensional stability. In addition, the high optical transparency of the glass helps to reduce light loss and distortion during the calibration process. At the same time, after its surface is treated, it can effectively control reflection and shadow, improving the accuracy of the calibration result. The glass material also has good long-term stability, is not easy to deform and age, and ensures the shape and size stability of the calibration board during long-term use. In terms of production, the glass checkerboard calibration board can be manufactured by high-precision processes, and has strong durability and is easy to maintain. The area of the checkerboard calibration board needs to be large enough to cover the field of view of at least two line array cameras. At least two line array cameras need to face the checkerboard calibration board to collect images. The meaning of facing is that the lens surfaces of at least two line array cameras are parallel to the checkerboard calibration board. The relative relationship between the line array camera and the checkerboard calibration board is as Figure 3 shown.

[0034] In the embodiment of the present application, in order to calibrate at least two line array cameras, it is necessary to convert the coordinate points in the images collected by at least two line array cameras into actual coordinates, and it is also necessary to perform stitching processing on the images collected by at least two line array cameras to form a complete image. In the above process, to achieve the conversion of coordinates, it is necessary to select feature points to determine the conversion matrix. The target image is obtained by collecting images of the checkerboard calibration board through at least two line array cameras, so as to determine the feature points based on the checkerboard image in the target image.

[0035] In the embodiments of the present application, multiple frames of images can be collected by a linear array camera. However, the target images participating in the subsequent process need to be the same frame of images collected by at least two linear array cameras at the same moment, so as to be more referenceable and avoid position deviations between different frames of images. At least two linear array cameras can be controlled by the same controller, and the controller simultaneously sends control signals to at least two linear array cameras to control the at least two linear array cameras to perform image acquisition. Before this, the controller can start sending control signals to detect and correct the time when the control signals reach at least two linear array cameras, so that the time when the control signals reach at least two linear array cameras is the same.

[0036] S120. For the target images collected by two adjacent linear array cameras, determine the adjacent checkerboard demarcation points as feature points according to the pixel points with the largest pixel gradient in the target images and the adjacent pixel points of the pixel points with the largest pixel gradient.

[0037] In the embodiments of the present application, the coordinate transformation and stitching process are described by taking the target images collected by two adjacent linear array cameras as an example. The target images collected by other adjacent linear array cameras are the same in principle, and the coordinate transformation and image stitching are performed in the same way.

[0038] Exemplarily, for the target images collected by two adjacent linear array cameras, both of which contain checkerboard images, the checkerboard images contain checkerboards of different colors arranged alternately, and there are obvious boundaries between the checkerboards of different colors. The feature points can be directly and accurately determined according to the obvious boundaries. Specifically, the checkerboard boundary points separate the checkerboards of different colors, so they have the characteristic of large pixel gradient. Based on this characteristic, the demarcation points of adjacent checkerboards can be determined as feature points.

[0039] In the embodiments of the present application, in order to more accurately determine the demarcation points, determine the pixel points with the largest pixel gradient in the target images and the adjacent pixel points of the pixel points with the largest pixel gradient. Determine the adjacent checkerboard demarcation points according to the pixel points with the largest pixel gradient and the adjacent pixel points of the pixel points with the largest pixel gradient, which can realize sub-pixel level analysis according to the change characteristics of the pixel gradient, more accurately determine the adjacent checkerboard demarcation points with the largest pixel gradient, and avoid errors in the determined demarcation points caused by the fact that the pixel points with the largest pixel gradient are not actually the positions with the largest pixel gradient. Specifically, the pixel values of the pixel points with the largest pixel gradient can be fitted with the pixel values of the adjacent pixel points, and the position point with the actual largest pixel gradient, that is, the adjacent checkerboard demarcation point, can be determined from the fitting result as the feature point.

[0040] S130. Determine a perspective transformation matrix for the conversion between the image coordinates and the planar coordinates based on the image coordinates of the feature points in the target image and the planar coordinates of the feature points in the planar coordinate system established with the checkerboard calibration board as the reference, so as to perform coordinate transformation on each target data point in the target image based on the perspective transformation matrix to obtain a planar mosaic image.

[0041] Exemplarily, after determining the feature points, the image coordinates of the feature points in the target image can be determined, and based on the positions of the feature points in the checkerboard calibration board, the planar coordinates of the feature points in the planar coordinate system established with the checkerboard calibration board as the reference can be determined. The image coordinates of the feature points in the target image can be determined based on the method in S120 above. For example, the pixel point with the maximum pixel gradient is (30, 0) in the target image, and the horizontally adjacent pixel points are (29, 0) and (31, 0). If it is determined that the actual maximum pixel gradient is located in the middle of (30, 0) and (29, 0) according to the pixel gradient of the pixel point with the maximum pixel gradient and the pixel gradients of the adjacent pixel points, then the coordinates of the feature point can be determined as (29.5, 0). The planar coordinates of the feature points in the planar coordinate system can be determined according to the actual positions of the feature points in the checkerboard calibration board. For example, special marks can be set in advance at the checkerboard boundary points of the checkerboard calibration board. After determining the feature points in the target image, compare and match them with the marked checkerboard boundary points in the checkerboard calibration board to determine which boundary point the feature point is. After determining the corresponding boundary point in the checkerboard calibration board, determine the planar coordinates of the boundary point in the planar coordinate system.

[0042] Exemplarily, determine the perspective transformation matrix according to the image coordinates and the planar coordinates of the feature points, that is, the perspective transformation matrix for converting the image coordinates into the planar coordinates. Currently, when performing stitching calibration on the images of two adjacent linear array cameras, often only the image data is offset in the horizontal direction to achieve image stitching. However, simple rigid transformation ignores the installation errors of the mechanism itself, and it is very difficult to eliminate such non-linear accuracy losses during use. In the embodiments of the present application, by performing perspective transformation to convert the image coordinates of the feature points into planar coordinates, both linear transformation and non-linear transformation are achieved, improving the accuracy of coordinate transformation and the fusion degree of image stitching.

[0043] In the technical solution of the embodiment of the present application, at least two linear array cameras are made to face the checkerboard calibration board for image acquisition, and target images respectively acquired by each linear array camera are obtained. For the target images acquired by two adjacent linear array cameras, according to the pixel point with the maximum pixel gradient in the target image and the adjacent pixel points of the pixel point with the maximum pixel gradient, the boundary point between adjacent checkerboards is determined as a feature point, so as to make full use of the pixel value distribution characteristics of the checkerboard and the gradient characteristics of the pixel points to accurately determine the boundary point of the checkerboard as a feature point. According to the image coordinates of the feature point in the target image and the plane coordinates of the feature point in the plane coordinate system established based on the checkerboard calibration board, the perspective transformation matrix for the conversion between the image coordinates and the plane coordinates is determined, so as to more accurately determine the transformation relationship between the image coordinates, which can realize linear and non-linear transformations, effectively eliminate the accuracy loss of non-linearity, and improve the accuracy of the linear array camera and the fusion degree of the stitched images.

[0044] Figure 4 The flowchart of an image stitching method for a rectifying linear array camera provided by another embodiment of the present application is based on the above embodiment for optimization. For the solutions not described in detail in the embodiment of the present application, refer to the above embodiment. As Figure 4 shown, the method of the embodiment of the present application specifically includes the following steps:

[0045] S210. Make at least two linear array cameras face the checkerboard calibration board for image acquisition, and obtain target images respectively acquired by each linear array camera.

[0046] S220. For the target images acquired by two adjacent linear array cameras, according to the pixel point with the maximum pixel gradient in the target image and the adjacent pixel points of the pixel point with the maximum pixel gradient, determine the boundary point between adjacent checkerboards as a feature point.

[0047] S230. For the same feature point, construct a relationship equation between the image coordinates of the feature point, a preset perspective transformation matrix, and the plane coordinates; wherein, the elements in the preset perspective transformation matrix are parameters to be solved.

[0048] Exemplarily, a preset perspective transformation matrix can be set, and the elements in the preset perspective transformation matrix are set as unknown parameters to be solved. Constructing the relationship equation between the image coordinates of the feature point, the preset perspective transformation matrix, and the plane coordinates means making the product of the image coordinates and the preset perspective transformation matrix equal to the plane coordinates. Further splitting the equation can obtain the relationship formula between the image coordinates, the parameters to be solved of the preset perspective transformation matrix, and the plane coordinates.

[0049] Exemplarily, the preset perspective transformation matrix can be set as:

[0050]

[0051] where R 2×2 and T 2×1 are rotation and translation transformations in a rigid transformation, and V T represents a variable that changes the shape of an object in a perspective transformation. s is a scaling factor, which is generally normalized to 1.

[0052] Define the image coordinates of the feature points of the left linear array camera as (x l , y l ), define the image coordinates of the feature points of the right linear array camera as (x r , y r ), and define the plane coordinates of the feature points as (x o , y o ). Calculate the perspective transformation matrix H l of the left linear array camera and the perspective transformation matrix H r of the right linear array camera respectively, so as to achieve:

[0053]

[0054] S240. Substitute the image coordinates and plane coordinates corresponding to at least four feature points into the relationship equation, solve the parameters to be solved in the preset perspective transformation matrix, determine the perspective transformation matrix, and perform coordinate transformation on each target data point in the target image based on the perspective transformation matrix to obtain a plane mosaic image.

[0055] Exemplarily, the feature point image coordinates (x i , y i ) correspond to the plane coordinates (x j , y j ), H is the preset perspective transformation matrix. According to the perspective transformation rule, we can get:

[0056]

[0057] Substitute the expression and expand it, we can get:

[0058]

[0059] Normalize the scaling factor s to 1, and we can further simplify it to get:

[0060]

[0061] The perspective transformation matrix H to be solved has 8 degrees of freedom. Expand the matrix to get:

[0062]

[0063] where h11 , h 12 , h 13 …… is the original matrix coefficient divided by s. According to the correspondence between plane coordinates and homogeneous coordinates It can be further simplified to obtain:

[0064] h 11 x i +h 12 y i +h 13 -h 31 x i x j -h 32 y i x j -x j = 0;

[0065] h 21 x i +h 22 y i +h 23 -h 31 x i y j -h 32 y i y j -y j = 0;

[0066] Therefore, at least four sets of corresponding feature points are required to solve the perspective transformation matrix H with eight degrees of freedom. Input the image coordinates and plane coordinates of at least four feature points into the above equation to solve the parameters to be solved, that is, assign values to the unknown elements in the preset perspective transformation matrix, and obtain the perspective transformation matrix.

[0067] In the embodiment of the present application, bringing the image coordinates and plane coordinates corresponding to at least four feature points into the relational equation to solve the parameters to be solved in the preset perspective transformation matrix and determine the perspective transformation matrix includes:

[0068] Solving the relational equation based on at least two preset solving algorithms to obtain at least two candidate perspective transformation matrices;

[0069] Converting the image coordinates of the feature points based on the at least two candidate perspective transformation matrices to obtain the converted coordinates of the feature points;

[0070] Determining the final perspective transformation matrix from the candidate perspective transformation matrices according to the similarity between the converted coordinates of the feature points and the plane coordinates of the feature points.

[0071] Exemplarily, a preset solution algorithm can be used to solve the relational equation. For example, methods such as LMS, LMedS, RANSAC, and RHO can be used to solve the relational equation. In the embodiments of the present application, the relational equation can be solved based on at least two preset algorithms to obtain at least two candidate perspective transformation matrices. The image coordinates of the feature points are converted based on the at least two candidate perspective transformation matrices to obtain the converted coordinates of the feature points, that is, the plane coordinates obtained by conversion based on the candidate perspective transformation matrices. The converted coordinates are compared with the actual plane coordinates of the feature points to determine the similarity between the converted coordinates and the plane coordinates. The final perspective transformation matrix is determined from the candidate perspective transformation matrices according to the similarity. Specifically, the candidate perspective transformation matrix corresponding to the maximum similarity is used as the final perspective transformation matrix. The coordinates obtained after converting the image coordinates of the feature points based on this perspective transformation matrix are closest to the actual plane coordinates.

[0072] The embodiments of the present application provide an image stitching method for a rectifying linear array camera. For the same feature point, a relational equation between the image coordinates of the feature point, a preset perspective transformation matrix, and the plane coordinates is constructed; wherein, the elements in the preset perspective transformation matrix are parameters to be solved; the image coordinates and plane coordinates corresponding to at least four feature points are brought into the relational equation to solve the parameters to be solved in the preset perspective transformation matrix, and the perspective transformation matrix is determined. By constructing the perspective transformation matrix, various non-linear factors such as installation errors and lens distortions in the actual scene can be considered. It can not only accurately capture and correct the non-linear deformation of the image caused by complex factors such as equipment installation deviation and inherent lens distortion, but also dynamically adapt to the spatial transformation relationship with different viewing angle differences, thus realizing the in-depth reconstruction and precise alignment of image information.

[0073] Figure 5 The figure is a flowchart of an image stitching method for a rectifying linear array camera provided by another embodiment of the present application. The embodiments of the present application are optimized based on the above embodiments. For the solutions not described in detail in the embodiments of the present application, refer to the above embodiments. As Figure 5 shown, the method of the embodiments of the present application specifically includes the following steps:

[0074] S310. Let at least two linear array cameras face the checkerboard calibration board for image acquisition to obtain the target images respectively acquired by each linear array camera.

[0075] S320. Search for the pixel point with the maximum pixel gradient in the target image as the pixel point with the maximum pixel gradient, and obtain the adjacent pixel points on one side of the pixel point with the maximum pixel gradient and the adjacent pixel points on the other side of the pixel point with the maximum pixel gradient.

[0076] Exemplarily, when the checkerboard calibration board does not move, the target image obtained by the line array camera image acquisition is actually only an image of one pixel row. In this pixel row, the pixel point with the maximum pixel gradient can be searched as the pixel point with the maximum pixel gradient, and the adjacent pixel points on one side of the pixel point with the maximum pixel gradient, as well as the adjacent pixel points on the other side of the pixel point with the maximum pixel gradient, can be obtained. For example, assume that the pixel point with the maximum pixel gradient is (30, 0) in the target image, then the adjacent pixel point on one side is (29, 0), and the adjacent pixel point on the other side is (31, 0).

[0077] S330. Perform quadratic function difference interpolation on the gradient modulus values of the pixel point with the maximum pixel gradient and the adjacent pixel points to obtain the adjacent checkerboard demarcation points of sub-pixel positioning as feature points.

[0078] Exemplarily, perform quadratic function difference interpolation calculation on the gradient modulus values of the pixel point with the maximum pixel gradient and the adjacent pixel points to obtain the adjacent checkerboard demarcation points of sub-pixel positioning as feature points. Specifically, assume that the edge point set is A, B, C in sequence, where B is the pixel point with the maximum pixel gradient found by the pixel-level algorithm, and A and C are adjacent pixel points. Through the sub-pixel edge extraction algorithm, a new edge point η can be obtained, and the calculation formula is as follows:

[0079]

[0080] where ‖g(A)‖, ‖g(B)‖, and ‖g(C)‖ respectively represent the gradient modulus values at points A, B, and C. η is the adjacent checkerboard demarcation point and serves as a feature point. Exemplarily, as Figure 6 shown, the horizontal axis X identifies different position points, and the vertical axis Y represents the pixel gradient value. B is the pixel point with the maximum pixel gradient, but actually, there is a sub-pixel-level η between pixel point B and pixel point C where the pixel gradient is the largest, and this position point is the actual checkerboard demarcation point and serves as a feature point.

[0081] S340. Determine the perspective transformation matrix for the conversion between the image coordinates and the plane coordinates based on the image coordinates of the feature points in the target image and the plane coordinates of the feature points in the plane coordinate system established with the checkerboard calibration board as the reference, so as to perform coordinate transformation on each target data point in the target image based on the perspective transformation matrix to obtain a plane mosaic image.

[0082] In a large field of view measurement scenario, even a tiny pixel-level error can lead to significant measurement deviation. During the process of extracting feature points, the edge precision at the pixel level is restricted by the physical resolution of the sensor, and the extracted edge points often deviate from the actual feature points to some extent. In the embodiments of the present application, a sub-pixel low edge extraction algorithm is adopted. According to the maximum value of the pixel gradient and the gradients of adjacent pixel points, a quadratic function fitting is performed to obtain the sub-pixel level position points, thereby more accurately determining the checkerboard boundary points.

[0083] As a non-limiting implementation manner, the determination process of the image coordinates of the feature points and the plane coordinates of the feature points includes:

[0084] In the target image, determine each adjacent checkerboard demarcation point as a feature point, and determine the image coordinates of the feature point according to the position of the feature point in the target image;

[0085] For each feature point, determine the position of the same feature point belonging to the same in the checkerboard calibration board in the target image, and determine the plane coordinates of the same feature point in the plane coordinate system established based on the checkerboard calibration board according to the position of the same feature point in the checkerboard calibration board.

[0086] Exemplarily, in the target image, determine adjacent checkerboard demarcation points as feature points according to the above scheme. In a target image, there may be more than one feature point, that is, there may be multiple groups of adjacent checkerboard demarcation points. The number of feature points can be adaptively determined according to actual requirements. For example, if the coordinates of four feature points are required based on equation solving needs, two feature points can be respectively selected in the target images of two adjacent line array cameras. Determine the image coordinates of the feature points in the target image according to the positions of the feature points in the target image, see the scheme of the above embodiments. For each feature point, the position of the same feature point belonging to the feature point in the target image in the checkerboard calibration board can be determined, and the plane coordinates of the feature point can be determined according to the position of the same feature point in the checkerboard calibration board. The same feature point is manifested as the point where the checkerboard demarcation point in the checkerboard calibration board is presented in the target image after being image-captured by the line array camera.

[0087] In the embodiments of the present application, among the checkerboards within the field of view of the line array camera in the checkerboard calibration board, there are at least two checkerboards with different sizes;

[0088] Correspondingly, the determination process of the same feature point includes:

[0089] In the target image, determine the pixel range belonging to the same checkerboard according to the pixel value distribution of the pixel points;

[0090] Determine the correspondence between the checkerboards reflected by the pixel ranges in the target image and the checkerboards in the checkerboard calibration board according to the pixel ranges of different checkerboards and the size characteristics of the checkerboards in the checkerboard calibration board;

[0091] Determine the checkerboard boundary where the feature point is located in the checkerboard calibration board according to the checkerboard boundary where the feature point is located in the target image and the correspondence;

[0092] Determine the same feature point in the checkerboard calibration board according to the checkerboard boundary where the feature point is located in the checkerboard calibration board.

[0093] Exemplarily, at least two sizes of checkerboards can be preset in the checkerboard calibration board to facilitate the distinction of each checkerboard. In the target image, identify the checkerboards and determine the pixel ranges of the same checkerboard. For checkerboards with different sizes, their pixel ranges are different and there is a relative size relationship. In the checkerboard calibration board, the size characteristics of the checkerboards are different and there is a relative size relationship. The correspondence between the same checkerboards in the target image and the checkerboard can be determined according to the relative size relationship of the pixel ranges of each checkerboard in the target image and the relative size relationship of each checkerboard in the checkerboard calibration board. For example, if the size rule of the pixel ranges of four checkerboards in the target image is large, small, large, small, then these four checkerboards correspond to the four checkerboards in the checkerboard calibration board with the size rule of large, small, large, small, so as to determine the correspondence. In the target image, determine the checkerboard boundary where the feature point is located. According to the correspondence between the same checkerboard in the target image and the checkerboard calibration board, the checkerboard boundary to which the feature point belongs in the checkerboard calibration board can be determined, so as to lock the same feature point.

[0094] In the embodiment of the present application, among the checkerboards in the checkerboard calibration board within the field of view of the line array camera, there are at least two groups of adjacent checkerboards with different colors;

[0095] Correspondingly, the determination process of the same feature point includes:

[0096] In the target image, determine the pixel values belonging to the same checkerboard by determining the pixel value distribution of the pixel points;

[0097] Determine the correspondence between the checkerboards reflected by the pixel values in the target image and the checkerboards in the checkerboard calibration board according to the pixel values of different checkerboards and the color distribution of the checkerboards in the checkerboard calibration board;

[0098] Determine the checkerboard boundary where the feature point is located in the checkerboard calibration board according to the checkerboard boundary where the feature point is located in the target image and the correspondence;

[0099] Determine the same feature points in the checkerboard calibration board according to the checkerboard boundaries where the feature points are located in the checkerboard calibration board.

[0100] Exemplarily, each checkerboard in the checkerboard calibration board can also be set to a different color, where there are at least two groups of adjacent checkerboards with different colors. For example, the first checkerboard and the second checkerboard in the checkerboard calibration board are two adjacent checkerboards, with one black and one white color, and the third checkerboard and the fourth checkerboard are two adjacent checkerboards, with one red and one green color. Identify the target image, determine the pixel values of each checkerboard, and according to the pixel values of the checkerboards and the known color distribution of the checkerboards in the checkerboard calibration board, determine the correspondence between each checkerboard in the target image and each checkerboard in the checkerboard calibration board. Combine the checkerboard boundaries where the feature points are located in the target image to determine the checkerboard boundaries where the feature points are located in the checkerboard calibration board, and then determine the same feature points in the checkerboard calibration board as those in the target image.

[0101] The above solution can clearly distinguish the checkerboards at different positions by making a discriminative setting for the checkerboards, and further clearly distinguish the feature points at the boundaries of different checkerboards, which is convenient for quickly locking the same feature points and obtaining the image coordinates and plane coordinates corresponding to the feature points.

[0102] Figure 7 The following is a schematic structural diagram of an image stitching device for a rectifying line array camera provided by an embodiment of the present application. This device can execute the image stitching method for a rectifying line array camera provided in any embodiment of the present application, and has corresponding functional modules and beneficial effects for executing the method. As Figure 7 shown, the device includes:

[0103] An image acquisition module 410, configured to make at least two line array cameras face the checkerboard calibration board for image acquisition, and obtain target images respectively acquired by each line array camera;

[0104] A feature point determination module 420, configured to, for the target images acquired by two adjacent line array cameras, determine the adjacent checkerboard boundary points as feature points according to the pixel points with the maximum pixel gradient in the target image and the adjacent pixel points of the pixel points with the maximum pixel gradient;

[0105] A perspective transformation matrix determination module 430, configured to determine a perspective transformation matrix for converting between the image coordinates and the plane coordinates according to the image coordinates of the feature points in the target image and the plane coordinates of the feature points in a plane coordinate system established based on the checkerboard calibration board, so as to perform coordinate transformation on each target data point in the target image based on the perspective transformation matrix to obtain a plane stitched image.

[0106] In an embodiment of the present application, the perspective transformation matrix determination module 430 determines a perspective transformation matrix for the conversion between the image coordinates and the plane coordinates according to the image coordinates of the feature points in the target image and the plane coordinates of the feature points in a plane coordinate system established based on the checkerboard calibration board, including:

[0107] For the same feature point, establish a relationship equation among the image coordinates of the feature point, a preset perspective transformation matrix, and the plane coordinates; wherein, the elements in the preset perspective transformation matrix are parameters to be solved.

[0108] Substitute the image coordinates and plane coordinates corresponding to at least four feature points into the relationship equation, solve the parameters to be solved in the preset perspective transformation matrix, and determine the perspective transformation matrix.

[0109] In an embodiment of the present application, the perspective transformation matrix determination module 430 substitutes the image coordinates and plane coordinates corresponding to at least four feature points into the relationship equation, solves the parameters to be solved in the preset perspective transformation matrix, and determines the perspective transformation matrix, including:

[0110] Solve the relationship equation based on at least two preset solution algorithms to obtain at least two candidate perspective transformation matrices;

[0111] Convert the image coordinates of the feature points based on the at least two candidate perspective transformation matrices to obtain the converted coordinates of the feature points;

[0112] Determine the final perspective transformation matrix from the candidate perspective transformation matrices according to the similarity between the converted coordinates of the feature points and the plane coordinates of the feature points.

[0113] In an embodiment of the present application, the feature point determination module 420 determines the boundary points between two grids of the checkerboard as feature points according to the pixel points with the maximum pixel gradient in the target image and the adjacent pixel points of the pixel points with the maximum pixel gradient, including:

[0114] Search for the pixel point with the maximum pixel gradient in the target image as the pixel point with the maximum pixel gradient, and obtain the adjacent pixel points on one side of the pixel point with the maximum pixel gradient and the adjacent pixel points on the other side of the pixel point with the maximum pixel gradient;

[0115] Perform quadratic function difference compensation on the gradient modulus values of the pixel point with the maximum pixel gradient and the adjacent pixel points to obtain the adjacent checkerboard boundary points with sub-pixel positioning as feature points.

[0116] In an embodiment of the present application, the device further includes: a coordinate determination module, configured to:

[0117] In the target image, determine each adjacent checkerboard demarcation point as a feature point, and determine the image coordinates of the feature point according to the position of the feature point in the target image;

[0118] For each feature point, determine the position of the same feature point belonging to the same in the checkerboard calibration board in the target image, and determine the plane coordinates of the same feature point in the plane coordinate system established based on the checkerboard calibration board according to the position of the same feature point in the checkerboard calibration board.

[0119] In the embodiment of the present application, among the checkerboards within the field of view of the line array camera in the checkerboard calibration board, there are at least two checkerboards with different sizes;

[0120] Correspondingly, the coordinate determination module is further configured to:

[0121] In the target image, determine the pixel range belonging to the same checkerboard according to the pixel value distribution of the pixel points;

[0122] According to the pixel ranges of different checkerboards and the size characteristics of the checkerboards in the checkerboard calibration board, determine the correspondence between the checkerboards reflected by each pixel range in the target image and the checkerboards in the checkerboard calibration board;

[0123] According to the checkerboard boundary where the feature point is located in the target image and the correspondence, determine the checkerboard boundary where the feature point is located in the checkerboard calibration board;

[0124] According to the checkerboard boundary where the feature point is located in the checkerboard calibration board, determine the same feature point in the checkerboard calibration board.

[0125] In the embodiment of the present application, among the checkerboards within the field of view of the line array camera in the checkerboard calibration board, there are at least two groups of adjacent checkerboards with different colors;

[0126] Correspondingly, the coordinate determination module is further configured to:

[0127] In the target image, determine the pixel values belonging to the same checkerboard by determining the pixel value distribution of the pixel points;

[0128] According to the pixel values of different checkerboards and the color distribution of the checkerboards in the checkerboard calibration board, determine the correspondence between the checkerboards reflected by each pixel value in the target image and the checkerboards in the checkerboard calibration board;

[0129] According to the checkerboard boundary where the feature point is located in the target image and the correspondence, determine the checkerboard boundary where the feature point is located in the checkerboard calibration board;

[0130] Determine the same feature point in the checkerboard calibration board according to the checkerboard boundary where the feature point is located in the checkerboard calibration board.

[0131] The image stitching device of a rectifying line array camera provided by an embodiment of the present application can execute the image stitching method of a rectifying line array camera provided by any embodiment of the present application, and has functional modules and beneficial effects corresponding to the execution of the method.

[0132] Figure 8 FIG. shows a schematic structural diagram of an electronic device 10 that can be used to implement embodiments of the present application. The electronic device is intended to represent various forms of digital computers, such as, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, personal digital processors, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present application described and / or claimed herein.

[0133] As Figure 8 shown, the electronic device 10 includes at least one processor 11, and a memory communicatively connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc. Among them, the memory stores a computer program executable by the at least one processor, and the processor 11 can execute various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. The input / output (I / O) interface 15 is also connected to the bus 14.

[0134] A plurality of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disc, etc.; and a communication unit 19, such as a network card, a modem, a wireless transceiver for image stitching of a rectifying line array camera, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0135] The processor 11 may be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the image stitching method for the rectifying line array camera.

[0136] In some embodiments, the image stitching method for the rectifying line array camera may be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed onto the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the image stitching method for the rectifying line array camera described above may be executed. Alternatively, in other embodiments, the processor 11 may be configured to execute the image stitching method for the rectifying line array camera in any other suitable manner (e.g., by means of firmware).

[0137] Various embodiments of the systems and techniques described above herein may be implemented in digital electronic circuitry, integrated circuit systems, field programmable gate arrays (FPGA), application specific integrated circuits (ASIC), application specific standard products (ASSP), systems on a chip (SOC), complex programmable logic devices (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: implemented in one or more computer programs that may be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a special-purpose or general-purpose programmable processor that may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit the data and instructions to the storage system, the at least one input device, and the at least one output device.

[0138] The computer program for implementing the method of the present application may be written in any combination of one or more programming languages. These computer programs may be provided to the processor of a general-purpose computer, a special-purpose computer, or other programmable image stitching device for the rectifying line array camera, such that when the computer programs are executed by the processor, the functions / operations specified in the flowchart and / or block diagram are implemented. The computer programs may be executed entirely on the machine, partially on the machine, as a stand-alone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0139] In the context of this application, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer-readable storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, the computer-readable storage medium can be a machine-readable signal medium. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0140] To provide for interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can also be used to provide for interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0141] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0142] A computing system may include a client and a server. The client and the server are generally far from each other and usually interact via a communication network. The client-server relationship is created by computer programs running on respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system, solving the defects of difficult management and weak business scalability existing in traditional physical hosts and VPS services.

[0143] It should be understood that various forms of the processes shown above can be used, steps can be reordered, added, or deleted. For example, the steps described in this application can be executed in parallel, sequentially, or in a different order, as long as the information expected by the technical solution of this application can be achieved, and no limitation is imposed herein.

[0144] The above specific embodiments do not constitute a limitation on the protection scope of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application shall be included within the protection scope of this application.

Claims

1. An image stitching method for a deflection correction linear array camera, characterized in that: The method comprises: At least two linear array cameras are directed to capture images on the checkerboard calibration plate to obtain target images captured by each linear array camera; For target images captured by two adjacent line array cameras, adjacent chessboard boundary points are determined as feature points according to the pixel point with the maximum pixel gradient in the target image and the adjacent pixel points of the pixel point with the maximum pixel gradient; According to the image coordinates of the feature points in the target image and the plane coordinates of the feature points in a plane coordinate system established with the checkerboard calibration plate as a reference, a perspective transformation matrix for converting between the image coordinates and the plane coordinates is determined, so as to perform coordinate transformation on each target data point in the target image based on the perspective transformation matrix to obtain a plane stitched image.

2. The method according to claim 1, characterized in that: According to the image coordinates of the feature point in the target image and the plane coordinates of the feature point in the plane coordinate system established with the checkerboard calibration plate as a reference, determining a perspective transformation matrix for converting between the image coordinates and the plane coordinates, including: For the same feature point, construct a relationship equation between the image coordinates of the feature point, the preset perspective transformation matrix and the plane coordinates; wherein the elements in the preset perspective transformation matrix are the parameters to be solved; The image coordinates and the plane coordinates corresponding to at least four feature points are substituted into the relationship equation, the parameters to be solved in the preset perspective transformation matrix are solved, and the perspective transformation matrix is ​​determined.

3. The method according to claim 2, characterized in that Substituting the image coordinates and the plane coordinates corresponding to at least four feature points into the relationship equation, solving the parameters to be solved in the preset perspective transformation matrix, and determining the perspective transformation matrix, including: Solving the relational equation based on at least two preset solving algorithms to obtain at least two candidate perspective transformation matrices; transforming the image coordinates of the feature points based on the at least two candidate perspective transformation matrices to obtain transformed coordinates of the feature points; According to the similarity between the transformed coordinates of the feature points and the plane coordinates of the feature points, a final perspective transformation matrix is ​​determined from the candidate perspective transformation matrices.

4. The method according to claim 1, characterized in that Determining two boundary points of a chessboard as feature points according to a pixel point with a maximum pixel gradient in the target image and adjacent pixels of the pixel point with a maximum pixel gradient, including: Searching for a pixel point with the maximum pixel gradient in the target image as the pixel point with the maximum pixel gradient, and obtaining adjacent pixel points on one side of the pixel point with the maximum pixel gradient and adjacent pixel points on the other side of the pixel point with the maximum pixel gradient; A quadratic function difference is performed on the pixel point with the maximum pixel gradient and the gradient modulus values ​​of the adjacent pixel points to obtain adjacent chessboard boundary points with sub-pixel positioning as feature points.

5. The method according to claim 1, characterized in that The process of determining the image coordinates of the feature points and the plane coordinates of the feature points includes: In the target image, each adjacent chessboard boundary point is determined as a feature point, and the image coordinates of the feature point are determined according to the position of the feature point in the target image; For each feature point, the position of the same feature point as that in the target image in the checkerboard calibration plate is determined, and the plane coordinates of the same feature point in the plane coordinate system established with the checkerboard calibration plate as a reference are determined according to the position of the same feature point in the checkerboard calibration plate.

6. The method according to claim 5, characterized in that Among the checkerboard grids in the checkerboard calibration plate within the field of view of the linear array camera, at least two checkerboard grids have different sizes; Accordingly, the process of determining the same feature point includes: In the target image, determining a pixel range belonging to the same chessboard according to pixel value distribution of pixel points; Determining the correspondence between the checkerboard reflected by each pixel range in the target image and the checkerboard in the checkerboard calibration plate according to the pixel range of different checkerboards and the size characteristics of the checkerboard in the checkerboard calibration plate; Determining the checkerboard boundary where the feature point is located in the checkerboard calibration plate according to the checkerboard boundary where the feature point is located in the target image and the corresponding relationship; The same feature point in the checkerboard calibration plate is determined according to the checkerboard boundary where the feature point is located in the checkerboard calibration plate.

7. The method according to claim 1, characterized in that Among the checkerboards in the checkerboard calibration plate within the field of view of the linear array camera, there are at least two groups of adjacent checkerboards with different colors; Accordingly, the process of determining the same feature point includes: In the target image, determining the pixel value distribution of the pixel points and determining the pixel values ​​belonging to the same checkerboard; Determining the correspondence between the checkerboard reflected by each pixel value in the target image and the checkerboard in the checkerboard calibration plate according to the pixel values ​​of different checkerboards and the color distribution of the checkerboards in the checkerboard calibration plate; Determining the checkerboard boundary where the feature point is located in the checkerboard calibration plate according to the checkerboard boundary where the feature point is located in the target image and the corresponding relationship; The same feature point in the checkerboard calibration plate is determined according to the checkerboard boundary where the feature point is located in the checkerboard calibration plate.

8. An image stitching device for a deflection correction linear array camera, characterized in that: The device comprises: An image acquisition module, used to enable at least two linear array cameras to acquire images facing the chessboard calibration plate, and obtain target images acquired by each linear array camera; A feature point determination module is used to determine adjacent chessboard boundary points as feature points for a target image captured by two adjacent linear array cameras according to a pixel point with a maximum pixel gradient in the target image and adjacent pixels of the pixel point with a maximum pixel gradient; A perspective transformation matrix determination module is used to determine the perspective transformation matrix for converting between the image coordinates and the plane coordinates according to the image coordinates of the feature points in the target image and the plane coordinates of the feature points in the plane coordinate system established with the chessboard calibration plate as a reference, so as to perform coordinate transformation on each target data point in the target image based on the perspective transformation matrix to obtain a plane stitching image.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the image stitching method for the deflection correction line array camera according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the image stitching method for a deflection correction line array camera according to any one of claims 1 to 7 when executed.

Images