A cabinet image stitching system and method based on robot inspection

Abstract

The present invention discloses a cabinet image splicing system and method based on robot inspection, which belongs to the field of image splicing technology. The system of the present invention includes: an image automatic correction module: using computer vision technology to detect the boundaries or corners of the equipment in the image, and correcting the tilted equipment image into a square equipment image; an image calculation offset module: detecting the common feature points or boundaries between adjacent images, and automatically adjusting the positions of adjacent images; an optimal splicing point selection module: by aligning and cropping adjacent images, calculating the splicing points, and generating a spliced ​​image; an adaptive machine learning module: regularly monitoring the performance indicators of image splicing, using a supervised learning machine learning algorithm to train an adaptive model, the model will input features and predict the optimal image processing and splicing parameters; an image quality evaluation and optimization module: automatically evaluating the quality of the spliced ​​image, and implementing image optimization technology based on the evaluation results.

Description

Technical Field

[0001] The present invention relates to the technical field of image stitching, and in particular to a cabinet image stitching system and method based on robot inspection. Background Art

[0002] When the robot inspects the cabinet, there is a deviation between the direction in which it stops each time and the relative angle of the cabinet, so there is a deviation in the angle of the cabinet photographed by the industrial camera; the originally rectangular surface of the cabinet equipment appears as a trapezoid in the image, and the degree of the trapezoid is related to the size of the angle deviation. When the image is cropped according to a fixed rule, the left side of the equipment is not cropped into the green retention range. If there is a key inspection information indicator light here, the indicator light will disappear when it is combined into a complete image; the excess part on the right side of the equipment will be cropped into the image range. If there is a key inspection information indicator light here, it will cause the indicator light to appear double when it is combined into a complete image, that is, there was originally one indicator light, but there are two in the spliced ​​image.

[0003] When the robot inspects the cabinet, the distance from the cabinet to the position where it stops each time may have a large deviation, such as a deviation of more than 20 mm. According to the pinhole imaging principle of the industrial camera, the cabinet distance from the camera is different, and the cabinet range covered by the camera imaging image is also different. Assuming that the field of view of the industrial camera lens is 60 degrees, when the robot inspects, the industrial camera is 600 mm away from the cabinet, the error can reach 11.55 mm. The key information indicator light of the cabinet inspection is generally about 5 mm in diameter. The actual cabinet range obtained by cropping the image according to the fixed rule has a deviation of 11.55 mm. If the error is +20 mm, the indicator light at the cropping edge is not cropped into the reserved range, resulting in the disappearance of the indicator light when it is combined into a complete image. If the error is -20 mm, the indicator light at the cropping edge is cropped into the reserved range in the images of the upper and lower adjacent elevator heights, resulting in the existence of ghosting of the indicator light when it is combined into a complete image. Adjacent images are cropped according to fixed rules and then spliced. The brightness of the image cropping may vary greatly, and the image splicing is not smooth, resulting in obvious image splicing marks. Summary of the invention

[0004] The purpose of the present invention is to provide a cabinet image stitching system and method based on robot inspection to solve the problems raised in the above background technology.

[0005] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0006] A cabinet image stitching system based on robot inspection, comprising:

[0007] Image automatic correction module: Use computer vision technology to detect the boundaries or corners of the device in the image to determine the position and orientation of the device; based on the detected device position and orientation, automatically apply affine transformation to correct the tilted device image to a square device image;

[0008] Image calculation offset module: uses computer vision technology to detect common feature points or boundaries between adjacent images, and automatically calculates horizontal and vertical offsets based on the detected feature points or boundaries to determine the relative positions of adjacent images on the device installation surface; automatically adjusts the positions of adjacent images based on the calculated offsets;

[0009] Optimal stitching point selection module: by aligning and cropping adjacent images, converting the cropped upper and lower images into grayscale images respectively, generating a difference image, and calculating the stitching points, the final generated image will be the stitching of the two adjacent images;

[0010] Adaptive machine learning module: regularly monitors the performance indicators of image stitching and uses supervised learning machine learning algorithms to train an adaptive model that inputs features and predicts optimal image processing and stitching parameters;

[0011] Image quality assessment and optimization module: automatically assesses the quality of stitched images and implements image optimization technology based on the assessment results.

[0012] The operation steps of the automatic image correction module are as follows:

[0013] S2-1. Use the hough line detection algorithm to detect horizontal and vertical lines in the image and select the angle of the longest line in the upper part of the image Angle of the longest straight line at the bottom The angle of the longest straight line on the left The angle of the longest straight line on the right

[0014] S2-2. Assuming that the width and height of the image are w and h respectively, translate the longest straight line in the upper part to pass through the image origin (0, 0), then the equation of the upper straight line is:

[0015]

[0016] Translate the longest straight line at the bottom to pass through the image origin (0, h), and the equation of the lower line is:

[0017]

[0018] Translate the longest straight line on the left to pass through the image origin (0, 0), and the equation of the straight line on the left is:

[0019]

[0020] Translate the longest straight line on the right to pass through the image origin (w, 0), and the equation of the straight line on the right is:

[0021]

[0022] Among them, x is the horizontal coordinate of the point on the straight line, and y is the vertical coordinate of the point on the straight line; the following parameters are obtained: the upper left vertex left_up_point where the upper straight line equation intersects with the left straight line equation, the left vertex left_down_point where the lower straight line equation and the left straight line equation intersect, the upper left vertex right_up_point where the upper straight line equation and the right straight line equation intersect, and the left vertex right_down_point where the lower straight line equation and the right straight line equation intersect;

[0023] S2-3. Calculate the affine transformation matrix M that transforms the four vertices of left_up_point, left_down_point, right_up_point, and right_down_poin to the four vertices (0, 0), (0, h), (w, 0), and (w, h) through affine transformation. Use M to affine transform the image and turn it into a rectangular image.

[0024] According to step S2-3, the method for calculating the matrix M is as follows:

[0025] set up Where w is the width of the image, h is the height of the image, and the coordinates of left_up_point are (x lu ,y lu ), the coordinates of left_down_point are (x ld ,y ld ), the coordinates of right_up_point are (x ru ,yr u ), the coordinates of right_down_point are (x rd ,y rd );

[0026] M=P r T *P d , where P r T is the matrix P r The transpose of , we get the following formula:

[0027] I d =M*I r

[0028] Among them, I d is the image of the transformed matrix M, I ris the image of the matrix M before transformation.

[0029] The image calculation offset module adopts a conventional feature point detection and matching algorithm to obtain matching pairs of upper and lower adjacent image feature points. The scheme steps are as follows:

[0030] S4-1. The distance from the robot-mounted laser radar measurement camera to the cabinet surface is set to d. The cabinet surface to the equipment installation plane is a cabinet fixed parameter set to d1. The lifting height difference when shooting adjacent images is h. The focal length of the camera is f. The formula for the imaging deviation of the same position on the equipment installation plane in adjacent images is:

[0031]

[0032] S4-2, suppose the feature points of adjacent image matching: the coordinates of the upper image are (x1, y1), and the coordinates of the lower image are (x2, y2). Design a feature point screening algorithm to screen the feature point matching pairs whose x-coordinate deviations are within the thresh range. The screened feature points will be evenly distributed on the equipment installation plane. The formula is as follows:

[0033]

[0034] S4-3. Calculate the x and y offsets of each matching feature point and the total offset respectively. The formula is as follows:

[0035]

[0036] Among them, lsx err ,lsy err The initial value is 0, sx err and sy err According to the formula, the calculation is repeated, and the number of cycles is the number of matching feature points; lsx err is the last calculated sx err Value, lsy err is the last sy calculated err value; calculated sx err is the total displacement in the height h direction, sy err is the total offset in the width w direction;

[0037] S4-4, assuming that the total feature point data after screening is n, calculate the offset in the height h direction and the width w direction respectively:

[0038]

[0039] Among them, offset x is the offset in the high h direction, offset y is the offset in the width w direction.

[0040] The operation steps of the optimal splicing point selection module are as follows:

[0041] S5-1, according to the offset x and offset y Offset adjacent images to align the overlapping parts of the upper and lower adjacent images, and then crop the overlapping parts respectively;

[0042] S5-2, converting the cropped overlapping images into grayscale images respectively, taking the absolute value after performing the difference, and obtaining the overlapped difference image;

[0043] S5-3, the difference image is stored in a computer in a format represented as a determinant, the minimum value of each column in the difference image is calculated as the splicing point of the upper and lower images, and each column of splicing points is connected into a line;

[0044] S5-4, stitching adjacent images using the line as the overlapping portion of adjacent images.

[0045] According to step S5-2, the cropped upper image is represented as I_upper, the lower image is represented as I_lower, and the difference image D is calculated using the following formula:

[0046] D(x, y)=|I_upper(x3, y3)-I_lower(x3, y3)_

[0047] Among them, D(x, y) represents the pixel value at position (x3, y3) in the difference image D, I_upper(x3, y3) represents the grayscale pixel value at position (x3, y3) of the upper image, and I_lower(x3, y3) represents the grayscale pixel value at position (x3, y3) of the lower image.

[0048] The operation steps of the adaptive machine learning module are as follows:

[0049] S7-1. Regularly monitor the performance indicators of image stitching, including image brightness, image quality, stitching accuracy, stitching speed, memory and computing resource usage;

[0050] S7-2, based on the collected performance indicators, perform feature engineering to extract useful features;

[0051] S7-3. Using a supervised learning machine learning algorithm, train an adaptive model; the model takes input features and predicts optimal image processing and stitching parameters;

[0052] S7-4, when the adaptive model training is completed, the machine learning module will use the model to generate recommended image processing and stitching parameters based on the current performance indicators;

[0053] The S7-5,module interacts with the components of the robot inspection system in real,time, receives the actual execution parameters and image processing results, and,continuously updates the adaptive model according to the actual execution results.

[0054] According to step S7-2, the image brightness is used as an estimate of the ambient brightness to calculate the average brightness of each image, which is expressed as:

[0055]

[0056] Among them, EL represents the ambient brightness, n represents the total number of pixels in the image, and Light i Represents the brightness value of the i-th pixel in the image;

[0057] Other monitored performance indicators will be directly used as input features.

[0058] According to step S7-3, in supervised learning, the algorithm learns from a data set containing known inputs and corresponding outputs, which are manually labeled and provide reference answers for the model's correct predictions. The algorithm learns the relationship between the input data and the output results. The training process includes:

[0059] Data preprocessing: including data cleaning, feature selection, data standardization and other steps to optimize training results.

[0060] Model selection: Choose an appropriate algorithm based on the nature of the problem.

[0061] Learning: The algorithm learns on the training set and tries to find the mapping between input and output.

[0062] Evaluation: Use the test set to evaluate the model performance.

[0063] By analyzing known performance indicators, such as image quality, stitching speed, etc., and image processing results, a model that can predict the optimal image processing and stitching parameters is trained. This process involves feature engineering, model training, and real-time updates to adapt to changing data and environmental conditions.

[0064] The image quality assessment and optimization module receives the spliced ​​images as input, uses the structural similarity index to evaluate each spliced ​​image, and generates a quality score for each image; then analyzes the quality assessment results of each spliced ​​image, determines which images need further optimization based on the score, and sets a threshold to determine which images are considered to be of low quality or need improvement; once the image that needs to be optimized is identified, the image optimization technology will be applied to improve the quality, and the applied image optimization technology will use image correction, noise reduction, contrast enhancement, and color correction; the module outputs the optimized image to replace the original spliced ​​image.

[0065] A cabinet image stitching method based on robot inspection includes the following steps:

[0066] S100, using computer vision technology to detect the boundaries or corners of each device image, and automatically applying an affine transformation to correct the tilted device image into a square device image;

[0067] S200, automatically calculating horizontal and vertical offsets based on the corrected image, determining relative positions of adjacent images on the device installation surface, and automatically adjusting positions of adjacent images according to the calculated offsets;

[0068] S300, aligning and cropping adjacent images, converting the cropped upper and lower images into grayscale images respectively, and generating a difference image;

[0069] S400, based on the difference image, calculating the minimum value of each column as the splicing point of the upper and lower images;

[0070] S500, connecting the stitching points into lines, and using the lines as stitching points of overlapping parts of adjacent images;

[0071] S600, regularly monitor the performance indicators of image stitching, use supervised learning machine learning algorithms to train an adaptive model, and the system will use the model to generate recommended image processing and stitching parameters;

[0072] S700, automatically evaluates the quality of the stitched image, uses the image quality evaluation algorithm to analyze the image, and adopts image optimization technology based on the evaluation results.

[0073] Compared with the prior art, the beneficial effects achieved by the present invention are:

[0074] The automatic image correction method of the present invention corrects the tilted image of the equipment into a square image, reduces the influence of the camera shooting angle and the non-vertical cabinet during the actual inspection process, and improves the image stitching effect;

[0075] The present invention automatically calculates adjacent images, and can adaptively calculate the offset of adjacent image stitching based on the horizontal and vertical offsets on the equipment installation surface, thereby reducing the imaging field deviation caused by the insufficient distance between the camera and the cabinet during the actual inspection process, and improving the image stitching effect;

[0076] The present invention adopts the optimal splicing point method of adjacent images to adaptively calculate the optimal splicing position of the overlapping parts of adjacent images, reduce the sudden change of adjacent images during transition during splicing, and improve the image splicing effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0078] Figure 1 It is a system structure diagram of a cabinet image stitching system based on robot inspection of the present invention;

[0079] Figure 2 It is a schematic diagram of the steps of a cabinet image stitching method based on robot inspection of the present invention. DETAILED DESCRIPTION

[0080] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0081] See also Figure 1-Figure 2 , the present invention provides a technical solution:

[0082] According to one embodiment of the present invention, Figure 1 As shown in the system structure diagram of a cabinet image stitching system based on robot inspection, a cabinet image stitching system based on robot inspection includes:

[0083] Image automatic correction module: Use computer vision technology to detect the boundaries or corners of the device in the image to determine the position and orientation of the device; based on the detected device position and orientation, automatically apply affine transformation to correct the tilted device image to a square device image;

[0084] Image calculation offset module: uses computer vision technology to detect common feature points or boundaries between adjacent images, and automatically calculates horizontal and vertical offsets based on the detected feature points or boundaries to determine the relative positions of adjacent images on the device installation surface; automatically adjusts the positions of adjacent images based on the calculated offsets;

[0085] Optimal stitching point selection module: by aligning and cropping adjacent images, converting the cropped upper and lower images into grayscale images respectively, generating a difference image, and calculating the stitching points, the final generated image will be the stitching of the two adjacent images;

[0086] Adaptive machine learning module: regularly monitors the performance indicators of image stitching and uses supervised learning machine learning algorithms to train an adaptive model that inputs features and predicts optimal image processing and stitching parameters;

[0087] Image quality assessment and optimization module: automatically assesses the quality of stitched images and implements image optimization technology based on the assessment results.

[0088] The operation steps of the automatic image correction module are as follows:

[0089] S2-1. Use the hough line detection algorithm to detect horizontal and vertical lines in the image and select the angle of the longest line in the upper part of the image Angle of the longest straight line at the bottom The angle of the longest straight line on the left The angle of the longest straight line on the right

[0090] S2-2. Assuming that the width and height of the image are w and h respectively, translate the longest straight line in the upper part to pass through the image origin (0, 0), then the equation of the upper straight line is:

[0091]

[0092] Translate the longest straight line at the bottom to pass through the image origin (0, h), and the equation of the lower line is:

[0093]

[0094] Translate the longest straight line on the left to pass through the image origin (0, 0), and the equation of the straight line on the left is:

[0095]

[0096] Translate the longest straight line on the right to pass through the image origin (w, 0), and the equation of the straight line on the right is:

[0097]

[0098] Among them, x is the horizontal coordinate of the point on the straight line, and y is the vertical coordinate of the point on the straight line; the following parameters are obtained: the upper left vertex left_up_point where the upper straight line equation intersects with the left straight line equation, the left vertex 1eft_down_point where the lower straight line equation and the left straight line equation intersect, the upper left vertex right_up_point where the upper straight line equation and the right straight line equation intersect, and the left vertex right_down_point where the lower straight line equation and the right straight line equation intersect;

[0099] S2-3. Calculate the affine transformation matrix M that transforms the four vertices of left_up_point, left_down_point, right_up_point, and right_down_poin to the four vertices (0, 0), (0, h), (w, 0), and (w, h) through affine transformation. Use M to affine transform the image and turn it into a rectangular image.

[0100] According to step S2-3, the method for calculating the matrix M is as follows:

[0101] set up Where w is the width of the image, h is the height of the image, and the coordinates of left_up_point are (x lu ,y lu ), the coordinates of left_down_point are (x ld ,y ld ), the coordinates of right_up_point are (x ru ,y ru ), the coordinates of right_down_point are (x rd ,y rd );

[0102] M=P r T *P d , where P r T is the transpose of the matrix Px, then we get the following formula:

[0103] I d =M*I r

[0104] Among them, I d is the image of the transformed matrix M, I r is the image of the matrix M before transformation.

[0105] The image calculation offset module adopts a conventional feature point detection and matching algorithm to obtain matching pairs of upper and lower adjacent image feature points. The scheme steps are as follows:

[0106] S4-1. The distance from the robot-mounted laser radar measurement camera to the cabinet surface is set to d. The cabinet surface to the equipment installation plane is a cabinet fixed parameter set to d1. The lifting height difference when shooting adjacent images is h. The focal length of the camera is f. The formula for the imaging deviation of the same position on the equipment installation plane in adjacent images is:

[0107]

[0108] S4-2, suppose the feature points of adjacent image matching: the coordinates of the upper image are (x1, y1), and the coordinates of the lower image are (x2, y2). Design a feature point screening algorithm to screen the feature point matching pairs whose x-coordinate deviations are within the thresh range. The screened feature points will be evenly distributed on the equipment installation plane. The formula is as follows:

[0109]

[0110] S4-3. Calculate the x and y offsets of each matching feature point and the total offset respectively. The formula is as follows:

[0111]

[0112] Among them, lsx err ,lsy err The initial value is 0, sx err and sy err According to the formula, the calculation is repeated, and the number of cycles is the number of matching feature points; lsx err is the last calculated sx err Value, lsy err is the last sy calculated err value; calculated sx err is the total displacement in the height h direction, sy err is the total offset in the width w direction;

[0113] S4-4, assuming that the total feature point data after screening is n, calculate the offset in the height h direction and the width w direction respectively:

[0114]

[0115] Among them, offset x is the offset in the high h direction, offset y is the offset in the width w direction.

[0116] The operation steps of the optimal splicing point selection module are as follows:

[0117] S5-1, according to the offset x and offset y Offset adjacent images to align the overlapping parts of the upper and lower adjacent images, and then crop the overlapping parts respectively;

[0118] S5-2, converting the cropped overlapping images into grayscale images respectively, taking the absolute value after performing the difference, and obtaining the overlapped difference image;

[0119] S5-3, the difference image is stored in a computer in a format represented as a determinant, the minimum value of each column in the difference image is calculated as the splicing point of the upper and lower images, and each column of splicing points is connected into a line to obtain the following result:

[0120]

[0121] S5-4, stitching adjacent images using the line as the overlapping portion of adjacent images.

[0122] According to step S5-2, the cropped upper image is represented as I_upper, the lower image is represented as I_lower, and the difference image D is calculated using the following formula:

[0123] D(x, y)=|I_upper(x3, y3)-I_lower(x3, y3)|

[0124] Among them, D(x, y) represents the pixel value at position (x3, y3) in the difference image D, I_upper(x3, y3) represents the grayscale pixel value at position (x3, y3) of the upper image, and I_lower(x3, y3) represents the grayscale pixel value at position (x3, y3) of the lower image.

[0125] The operation steps of the adaptive machine learning module are as follows:

[0126] S7-1. Regularly monitor the performance indicators of image stitching, including image brightness, image quality, stitching accuracy, stitching speed, memory and computing resource usage;

[0127] S7-2, based on the collected performance indicators, perform feature engineering to extract useful features;

[0128] S7-3. Using a supervised learning machine learning algorithm, train an adaptive model; the model takes input features and predicts optimal image processing and stitching parameters;

[0129] S7-4, when the adaptive model training is completed, the machine learning module will use the model to generate recommended image processing and stitching parameters based on the current performance indicators;

[0130] The S7-5,module interacts with the components of the robot inspection system in real,time, receives the actual execution parameters and image processing results, and,continuously updates the adaptive model according to the actual execution results.

[0131] According to step S7-2, the image brightness is used as an estimate of the ambient brightness to calculate the average brightness of each image, which is expressed as:

[0132]

[0133] Among them, EL represents the ambient brightness, n represents the total number of pixels in the image, and Light i Represents the brightness value of the i-th pixel in the image;

[0134] Other monitored performance indicators will be directly used as input features.

[0135] The image quality assessment and optimization module receives the spliced ​​images as input, uses the structural similarity index to evaluate each spliced ​​image, and generates a quality score for each image; then analyzes the quality assessment results of each spliced ​​image, determines which images need further optimization based on the score, and sets a threshold to determine which images are considered to be of low quality or need improvement; once the image that needs to be optimized is identified, the image optimization technology will be applied to improve the quality, and the applied image optimization technology will use image correction, noise reduction, contrast enhancement, and color correction; the module outputs the optimized image to replace the original spliced ​​image.

[0136] According to another embodiment of the present invention, Figure 2 As shown in the schematic diagram of the steps of a cabinet image stitching method based on robot inspection, a cabinet image stitching method based on robot inspection includes the following steps:

[0137] S100, using computer vision technology to detect the boundaries or corners of each device image, and automatically applying an affine transformation to correct the tilted device image into a square device image;

[0138] S200, automatically calculating horizontal and vertical offsets based on the corrected image, determining relative positions of adjacent images on the device installation surface, and automatically adjusting positions of adjacent images according to the calculated offsets;

[0139] S300, aligning and cropping adjacent images, converting the cropped upper and lower images into grayscale images respectively, and generating a difference image;

[0140] S400, based on the difference image, calculating the minimum value of each column as the splicing point of the upper and lower images;

[0141] S500, connecting the stitching points into lines, and using the lines as stitching points of overlapping parts of adjacent images;

[0142] S600, regularly monitor the performance indicators of image stitching, use supervised learning machine learning algorithms to train an adaptive model, and the system will use the model to generate recommended image processing and stitching parameters;

[0143] S700, automatically evaluates the quality of the stitched image, uses the image quality evaluation algorithm to analyze the image, and adopts image optimization technology based on the evaluation results.

[0144] According to step S100, we prepare a tilted device image that needs to be corrected, and the size of the image is 800 pixels wide * 600 pixels high. We use a boundary detection algorithm to detect the boundaries of the device in the image. The boundary detection algorithm successfully detects the four corner points of the device and obtains the following corner point coordinates: upper left corner: (100, 150), upper right corner: (600, 120), lower left corner: (80, 500), lower right corner: (620, 490). Based on the detected corner points, we can calculate the affine transformation matrix to correct the image to a square. The affine transformation matrix is ​​calculated, which describes how to map each pixel of the original image to the corrected position. By applying this transformation, we correct the tilted device image to a square device image. The size of the corrected image remains unchanged 800 pixels wide * 600 pixels high, but now the device is square in the image and is not tilted.

[0145] According to step S200, we have two rectified device images, each with a size of 800 pixels wide * 600 pixels high. We use computer vision technology to detect common feature points between the two images to determine their relative positions. We successfully detected 4 feature points between the two images and obtained the following feature point coordinates: Feature points of image 1: Feature point 1: (200, 400), feature point 2: (400, 300); Feature points of image 2: Feature point 1: (600, 400), feature point 2: (800, 300); Based on the detected feature points, we can calculate the horizontal and vertical offsets to determine the relative positions of adjacent images on the device mounting surface. We calculated the following offsets: horizontal offset: 200 pixels, vertical offset: 100 pixels; Based on the calculated offsets, we can automatically adjust the position of the second image to align with the first image. Specifically, move the second image 200 pixels in the horizontal direction and 100 pixels in the vertical direction.

[0146] According to step S400, we have two difference images, each with a size of 800 pixels wide and 300 pixels high. We compare the pixel values ​​of each column in each difference image and find the minimum value of each column. For the first difference image, we calculated the minimum values ​​of 800 columns and obtained a list of 800 minimum values. For the second difference image, we also calculated the minimum values ​​of 800 columns and obtained another list of 800 minimum values. Since there are 800 minimum values, we can determine 800 stitching points, each of which corresponds to a column in the difference image. The following are some results: the column minimum value list of the first difference image: [1, 2, 3, ..., 10, 5, 1, 0, 2, ...], the column minimum value list of the second difference image: [0, 2, 1, ..., 5, 3, 1, 0, 1, ...].

[0147] According to step S500, we have calculated 800 stitching points, which correspond to each column in the difference image. Each stitching point has an abscissa (x) and a ordinate (y). For example, the coordinates of stitching point 1 are (x1, y1), the coordinates of stitching point 2 are (x2, y2), and so on, until stitching point 800. We connect these stitching points in order into a line. This line will represent the stitching points of the overlapping parts of adjacent images.

[0148] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0149] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A cabinet image stitching system based on robot inspection, characterized in that: include: Image automatic correction module: Use computer vision technology to detect the boundaries or corners of the device in the image to determine the position and orientation of the device; based on the detected device position and orientation, automatically apply affine transformation to correct the tilted device image to a square device image; Image calculation offset module: uses computer vision technology to detect common feature points or boundaries between adjacent images, and automatically calculates horizontal and vertical offsets based on the detected feature points or boundaries to determine the relative positions of adjacent images on the device installation surface; automatically adjusts the positions of adjacent images based on the calculated offsets; Optimal stitching point selection module: by aligning and cropping adjacent images, converting the cropped upper and lower images into grayscale images respectively, generating a difference image, and calculating the stitching points, the final generated image will be the stitching of the two adjacent images; Adaptive machine learning module: regularly monitors the performance indicators of image stitching and uses supervised learning machine learning algorithms to train an adaptive model that inputs features and predicts optimal image processing and stitching parameters; Image quality assessment and optimization module: automatically assesses the quality of stitched images and implements image optimization technology based on the assessment results.

2. According to the cabinet image stitching system based on robot inspection according to claim 1, it is characterized by: The operation steps of the automatic image correction module are as follows: S2-1. Use the hough line detection algorithm to detect horizontal and vertical lines in the image and select the angle of the longest line in the upper part of the image. Angle of the longest straight line at the bottom The angle of the longest straight line on the left The angle of the longest straight line on the right S2-2, assuming that the width and height of the image are w and h respectively, translate the longest straight line in the upper part to pass through the image origin (0, 0), and the equation of the upper straight line is: Translate the longest straight line at the bottom to pass through the image origin (0, h), and the equation of the lower straight line is: Translate the longest straight line on the left to pass through the image origin (0, 0), and the equation of the straight line on the left is: Translate the longest straight line on the right to pass through the image origin (w, 0), and the equation of the straight line on the right is: Among them, x is the horizontal coordinate of the point on the straight line, and y is the vertical coordinate of the point on the straight line; the following parameters are obtained: the upper left vertex left_up_point where the upper straight line equation intersects with the left straight line equation, the left vertex left_down_point where the lower straight line equation and the left straight line equation intersect, the upper left vertex right_up_point where the upper straight line equation and the right straight line equation intersect, and the left vertex right_down_point where the lower straight line equation and the right straight line equation intersect; S2-3. Calculate the affine transformation matrix M that transforms the four vertices of left_up_point, left_down_point, right_up_point, and right_down_poin to the four vertices (0, 0), (0, h), (w, 0), and (w, h) through affine transformation. Use M to affine transform the image and turn it into a rectangular image.

3. According to the cabinet image stitching system based on robot inspection according to claim 2, it is characterized by: According to step S2-3, the method for calculating the matrix M is as follows: set up Where w is the width of the image, h is the height of the image, and the coordinates of left_up_point are (x lu ,y lu ), the coordinates of left_down_point are (x ld ,y ld ), the coordinates of right_up_point are (x ru ,y ru ), the coordinates of right_down_point are (x rd ,y rd ); M=P r T *P d , where P r T is the matrix P r The transpose of , we get the following formula: I d =M*I r Among them, I d is the image of the transformed matrix M, I r is the image of the matrix M before transformation.

4. The cabinet image stitching system based on robot inspection according to claim 1 is characterized in that: The image calculation offset module adopts a conventional feature point detection and matching algorithm to obtain matching pairs of upper and lower adjacent image feature points. The scheme steps are as follows: S4-1. The distance from the robot-mounted laser radar measurement camera to the cabinet surface is set to d. The cabinet surface to the equipment installation plane is a cabinet fixed parameter set to d1. The lifting height difference when shooting adjacent images is h. The focal length of the camera is f. The formula for the imaging deviation of the same position on the equipment installation plane in adjacent images is: S4-2, suppose the feature points of adjacent image matching: the coordinates of the upper image are (x1, y1), and the coordinates of the lower image are (x2, y2). Design a feature point screening algorithm to screen the feature point matching pairs whose x-coordinate deviations are within the thresh range. The screened feature points will be evenly distributed on the equipment installation plane. The formula is as follows: S4-3. Calculate the x and y offsets of each matching feature point and the total offset respectively. The formula is as follows: Among them, lsx err ,lsy err The initial value is 0, sx err and sy err According to the formula, the calculation is repeated, and the number of cycles is the number of matching feature points; lsx err is the last calculated sx err Value, lsy err is the last sy calculated err Value; calculated sx err is the total displacement in the height h direction, sy err is the total offset in the width w direction; S4-4, assuming that the total feature point data after screening is n, calculate the offset in the height h direction and the width w direction respectively: Among them, offset x is the offset in the high h direction, offset y is the offset in the width w direction.

5. The cabinet image stitching system based on robot inspection according to claim 4 is characterized in that: The operation steps of the optimal splicing point selection module are as follows: S5-1, according to the offset x and offset y Offset adjacent images to align the overlapping parts of the upper and lower adjacent images, and then crop the overlapping parts respectively; S5-2, converting the cropped overlapping images into grayscale images respectively, taking the absolute value after performing the difference, and obtaining the overlapped difference image; S5-3, the difference image is stored in a computer in a format represented as a determinant, the minimum value of each column in the difference image is calculated as the splicing point of the upper and lower images, and each column of splicing points is connected into a line; S5-4, stitching adjacent images using the line as the overlapping portion of adjacent images.

6. The cabinet image stitching system based on robot inspection according to claim 5 is characterized in that: According to step S5-2, the cropped upper image is represented as I_upper, the lower image is represented as I_lower, and the difference image D is calculated according to the following formula: D(x, y)=|I_upper(x3, y3)-I_lower(x3, y3)| Among them, D(x, y) represents the pixel value at position (x3, y3) in the difference image D, I_upper(x3, y3) represents the grayscale pixel value at position (x3, y3) of the upper image, and I_lower(x3, y3) represents the grayscale pixel value at position (x3, y3) of the lower image.

7. The cabinet image stitching system based on robot inspection according to claim 1 is characterized in that: The operation steps of the adaptive machine learning module are as follows: S7-1. Regularly monitor the performance indicators of image stitching, including image brightness, image quality, stitching accuracy, stitching speed, memory and computing resource usage; S7-2, based on the collected performance indicators, perform feature engineering to extract useful features; S7-3. Using a supervised learning machine learning algorithm, train an adaptive model; the model takes input features and predicts optimal image processing and stitching parameters; S7-4, when the adaptive model training is completed, the machine learning module will use the model to generate recommended image processing and stitching parameters based on the current performance indicators; The S7-5,module interacts with the components of the robot inspection system in real,time, receives the actual execution parameters and image processing results, and,continuously updates the adaptive model according to the actual execution results.

8. The cabinet image stitching system based on robot inspection according to claim 7 is characterized in that: According to step S7-2, the image brightness is used as an estimate of the ambient brightness to calculate the average brightness of each image, which is expressed as: Among them, EL represents the ambient brightness, n represents the total number of pixels in the image, and Light i Represents the brightness value of the i-th pixel in the image; Other monitored performance indicators will be directly used as input features.

9. The cabinet image stitching system based on robot inspection according to claim 1 is characterized in that: The image quality assessment and optimization module receives the spliced ​​images as input, uses the structural similarity index to evaluate each spliced ​​image, and generates a quality score for each image; then analyzes the quality assessment results of each spliced ​​image, determines which images need further optimization based on the score, and sets a threshold to determine which images are considered to be of low quality or need improvement; once the image that needs to be optimized is identified, the image optimization technology will be applied to improve the quality, and the applied image optimization technology will use image correction, noise reduction, contrast enhancement, and color correction; the module outputs the optimized image to replace the original spliced ​​image.

10. A cabinet image stitching method based on robot inspection, applied to a cabinet image stitching system based on robot inspection as claimed in any one of claims 1 to 9, characterized in that: The following steps are involved: S100, using computer vision technology to detect the boundaries or corners of each device image, and automatically applying an affine transformation to correct the tilted device image into a square device image; S200, automatically calculating horizontal and vertical offsets based on the corrected image, determining relative positions of adjacent images on the device installation surface, and automatically adjusting positions of adjacent images according to the calculated offsets; S300, aligning and cropping adjacent images, converting the cropped upper and lower images into grayscale images respectively, and generating a difference image; S400, based on the difference image, calculating the minimum value of each column as the splicing point of the upper and lower images; S500, connecting the stitching points into lines, and using the lines as stitching points of overlapping parts of adjacent images; S600, regularly monitor the performance indicators of image stitching, use supervised learning machine learning algorithms to train an adaptive model, and the system will use the model to generate recommended image processing and stitching parameters; S700, automatically evaluates the quality of the stitched image, uses the image quality evaluation algorithm to analyze the image, and adopts image optimization technology based on the evaluation results.

Images