Image registration method and related device

Abstract

The present application discloses an image registration method and related devices, the image registration method comprising: obtaining a first image obtained by a first camera device shooting a test object and a second image obtained by a second camera device shooting a test object; using the first image to perform offset correction on the second image to obtain a correction image; performing interpolation operation on the correction image to obtain a registration image; fusing the first image and the registration image to obtain a fused image. Through such a design, the use of flow mode registration can reduce delay, and after the second image is captured, the offset correction is performed to obtain a correction image without black edges, and the correction image is directly registered, which not only does not require the use of additional RAM resources to cache multiple lines for edge filling, but also can solve the problems of black edges and poor edge filling effects, thereby improving the efficiency of image registration.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image registration method and related devices. Background Art

[0002] Image registration is the process of spatially transforming two or more images acquired by different cameras or under different conditions (illuminance, camera position, angle, etc.) so that each image can be geometrically matched. The purpose of image registration is to remove the geometric inconsistencies between the image to be registered and the reference image, including translation, rotation and deformation, in preparation for further image processing. It is a key step in image processing and analysis, and a necessary prerequisite for image comparison, data fusion, change analysis, and target recognition. Image registration technology and devices based on field programmable gate arrays (FPGAs) can achieve image registration under various conditions, are easy to expand, improve the reuse rate of registration devices, and can intuitively view the effects of image registration, achieve online registration, are highly practical, and are easy to promote and use.

[0003] The image registration methods currently used usually include the following: (1) Block processing and double rate synchronous dynamic random access memory (DDR) for image caching; (2) Stream processing and random access memory (RMA) resources for storing multiple lines of edge padding; (3) Stream processing and level 1 DDR cache for copying edge padding. However, the above methods have the following defects: (1) large delay; (2) excessive RAM resource consumption or black edge problem; (3) unsatisfactory edge padding effect. Therefore, a new image registration method is urgently needed to solve the above problems. Summary of the invention

[0004] The main technical problem solved by the present application is to provide an image registration method and related devices, which can not only reduce the delay, but also solve the problems of black edges and poor edge filling effect.

[0005] In order to solve the above technical problems, a technical solution adopted in the present application is: to provide an image registration method, including: obtaining a first image obtained by a first camera device taking pictures of an object to be measured and a second image obtained by a second camera device taking pictures of the object to be measured; using the first image to perform offset correction on the second image to obtain a corrected image; performing interpolation operation on the corrected image to obtain a registered image; and fusing the first image and the registered image to obtain a fused image.

[0006] The step of using the first image to perform offset correction on the second image to obtain a corrected image includes: obtaining a first central image from the first image; and using the first central image to perform offset correction on the second image to obtain a corrected image.

[0007] Wherein, the step of obtaining a first central image from the first image includes: setting a first starting coordinate of the first central image according to a difference between a resolution of the first image and a target resolution; wherein the target resolution is smaller than the resolution of the first image; and obtaining the first central image having the target resolution from the first image using the first starting coordinate.

[0008] Among them, the step of using the first central image to perform offset correction on the second image to obtain a corrected image includes: initializing the coordinates of the second image using the first starting coordinates, and obtaining a second central image with the same resolution as the first central image from the second image; obtaining offset parameters of the second central image based on the first central image and the second central image; using the offset parameters to perform offset processing on the first starting coordinates to obtain second starting coordinates; and using the second starting coordinates to obtain the corrected image with the target resolution from the second image.

[0009] Among them, the resolution of the first image includes a first width value and a first height value, and the target resolution includes a second width value and a second height value; the step of setting the first starting coordinates of the first central image according to the difference between the resolution of the first image and the target resolution includes: obtaining the width difference between the first width value and the second width value, and obtaining the height difference between the first height value and the second height value; using half of the width difference as the horizontal coordinate value of the first starting coordinate, and using half of the height difference as the vertical coordinate value of the first starting coordinate, so as to obtain the first starting coordinates of the first central image.

[0010] Among them, the offset parameters include a horizontal coordinate offset parameter value and a vertical coordinate offset parameter value, and the horizontal coordinate offset parameter value is smaller than the horizontal coordinate value of the first starting coordinate, and the vertical coordinate offset parameter value is smaller than the vertical coordinate value of the first starting coordinate; the resolution of the second image includes a third width value and a third height value; the third width value is greater than the sum of the second width value and twice the maximum value of the absolute value of the horizontal coordinate offset parameter value, and the third height value is greater than the sum of the second height value and twice the maximum value of the absolute value of the vertical coordinate offset parameter value.

[0011] Among them, the step of performing an interpolation operation on the corrected image to obtain a registered image includes: obtaining a transformation matrix using the first central image and the corrected image; obtaining the registration coordinates of the corrected image according to the transformation matrix; performing an interpolation operation on the corrected image according to the registration coordinates to obtain a registered image; wherein the resolution of the registered image is the target resolution.

[0012] Among them, before the step of performing interpolation operation on the corrected image to obtain the registered image, it includes: performing image signal processing on the corrected image; the step of fusing the first image and the registered image to obtain the fused image includes: fusing the first central image and the registered image to obtain the fused image.

[0013] Wherein, before the step of fusing the first central image and the registered image to obtain the fused image, it also includes: performing image signal processing on the first central image; after the step of fusing the first central image and the registered image to obtain the fused image, it includes: in response to the existence of ghosting in the fused image, returning to the step of performing offset correction on the second image using the first image to obtain a corrected image.

[0014] To solve the above technical problems, another technical solution adopted in the present application is: to provide an electronic device, comprising a memory and a processor coupled to each other, wherein the memory stores program instructions, and the processor is used to execute the program instructions to implement the image registration method mentioned in any of the above embodiments.

[0015] In order to solve the above technical problems, another technical solution adopted in the present application is: providing a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is used to implement the image registration method mentioned in any of the above embodiments.

[0016] Different from the prior art, the beneficial effects of the present application are as follows: the present application provides a method of obtaining a first image of an object to be measured obtained by a first camera device and a second image of an object to be measured obtained by a second camera device, using the first image to perform offset correction on the second image to obtain a corrected image, performing interpolation operation on the corrected image to obtain a registered image, and fusing the first image and the registered image to obtain a fused image. Through such a design, the use of flow mode registration can reduce latency, and after acquiring the second image, offset correction is performed to obtain a corrected image without black edges, and the corrected image is directly registered, which not only does not require the use of additional RAM resources to cache multiple lines for edge filling, but also can solve problems such as black edges and poor edge filling effects, thereby improving the efficiency of image registration. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0018] Figure 1 It is a flowchart of an implementation method of an image registration method in the prior art;

[0019] Figure 2 yes Figure 1 A flowchart of an implementation method of the registration module 1;

[0020] Figure 3 yes Figure 2 Operational schematic diagram of an implementation method of block processing;

[0021] Figure 4 yes Figure 1 Schematic diagram of DDR read and write cache area;

[0022] Figure 5 It is a flowchart of another implementation method of the image registration method in the prior art;

[0023] Figure 6 yes Figure 5 A flowchart of an implementation method of the registration module 1;

[0024] Figure 7 yes Figure 6 A schematic diagram of an implementation method of offset correction;

[0025] Figure 8 is a schematic diagram of an implementation method corresponding to an edge filling operation;

[0026] Fig. 9 It is a flowchart of another implementation method of the image registration method in the prior art;

[0027] Fig.10 is a schematic diagram of another implementation method of the edge filling operation;

[0028] Fig.11 It is a flowchart of an implementation method of the image registration method of the present application;

[0029] Fig.12 This is a schematic diagram of the sensor manual pixel array;

[0030] Fig.13 yes Fig.11 A schematic flow chart of an implementation method of step S2 in FIG.

[0031] Fig.14 yes Fig.13A schematic flow chart of an implementation method of step S10;

[0032] Fig.15 yes Fig.14 Steps S100-S101 in the figure correspond to a schematic diagram of an implementation method;

[0033] Fig.16 yes Fig.14 A schematic flow chart of an implementation method of step S100;

[0034] Fig.17 yes Fig.13 A schematic flow chart of an implementation method of step S11;

[0035] Fig.18 yes Fig.17 Steps S110-S113 in the figure correspond to a schematic diagram of an implementation method;

[0036] Fig.19 yes Fig.17 Schematic diagram corresponding to the four cases of offset parameters;

[0037] Fig. 20 yes Fig.11 A schematic flow chart of an implementation method of step S3 in FIG.

[0038] Fig.21 yes Fig.11 Step S4 corresponds to a flow chart of an implementation method;

[0039] Fig. 22 It is a structural schematic diagram of an embodiment of the image registration device of the present application;

[0040] Fig.23 It is a schematic diagram of a framework of an implementation method of an electronic device of the present application;

[0041] Fig.24 It is a schematic diagram of a framework of an implementation scheme of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0043] In the present application, the first image captured by the first camera device on the object to be measured may be a visible light image or other types of images, which is not limited in the present application. The second image captured by the second camera device on the object to be measured may be an infrared image or other types of images such as a grayscale image, which is not limited in the present application. The schemes mentioned in the present application are all described by taking visible light images and infrared images as examples.

[0044] The following is an introduction to several image registration methods used in the prior art. Figure 1-4 , Figure 1 is a flowchart of an implementation method of an image registration method in the prior art. Figure 2 yes Figure 1 A flowchart of an implementation method of the registration module 1 is shown in FIG. Figure 3 yes Figure 2 The operation diagram of the block processing implementation method is as follows, Figure 4 yes Figure 1 Schematic diagram of DDR read and write cache area. Figure 1 As shown, the steps of the above-mentioned image registration method are as follows: (1) the image taken by camera 1 is sent to FPGA for collection via sensor 1 as a visible light image, and after a series of image signal processing, the image is stored in DDR cache area 1; (2) the image taken by camera 2 is sent to FPGA for collection via sensor 2 as an infrared image, and after a series of image signal processing (ISP image processing), the image is stored in DDR cache area 2; (3) the registration module takes out the corresponding block image from DDR cache area 2 for processing based on the coordinate information and block information, and writes it into DDR cache area 3 after the processing is completed; (4) the fusion module takes out the visible light image from DDR cache area 1, takes out the registered infrared image from DDR cache area 3, fuses the registered infrared image and the visible light image, and outputs them. Figure 2 As shown, the corresponding block image in the above step (3) is processed in blocks, and the registration coordinates are calculated using the transformation matrix (tform matrix). Then, the pixel values ​​of the infrared image are interpolated using the registration coordinates, and the registered infrared image is obtained by registering the infrared image. Figure 3 As shown, Figure 2 The block processing in is to divide the infrared image with a width of W and a height of H into small blocks with a width of w1 and a height of h1. However, the above method has certain defects. Figure 4As shown in the figure, due to the inconsistent read and write speeds of the DDR cache area, the ISP image processing clock, DDR read and write clock, and fusion processing output clock may all be different. In order to avoid image faults across frames, all areas involving the DDR cache are required to cache at least one frame of data before starting the next step of processing. When the ISP image processing starts to write the third frame (frame3) into the DDR cache area, the registration module can process the second frame image (frame2). At this time, the fusion module reads the first frame (frame1) image. It can be seen that this process will introduce a delay of 2 frames. Taking the 30fps frame rate commonly used in actual visible light and infrared image registration and fusion application scenarios as an example, a delay of 66.67ms will be introduced. In some scenarios with high requirements for image delay, a delay of 66.67ms will directly determine the competitiveness of the product.

[0045] See also Figure 5-Figure 8 , Figure 5 is a flowchart of another embodiment of the image registration method in the prior art. Figure 6 yes Figure 5 A flowchart of an implementation method of the registration module 1 is shown in FIG. Figure 7 yes Figure 6 A schematic diagram of an implementation method of offset correction in FIG. Figure 8 is a schematic diagram of an implementation method of the edge filling operation. Figure 5 As shown in FIG. 1 , the steps of the above-mentioned image registration method are as follows: (1) the image captured by camera 1 is sent to FPGA for acquisition via sensor 1 as a visible light image, and after a series of ISP image processing, the image is sent to the fusion module; (2) the image captured by camera 2 is sent to FPGA for acquisition via sensor 2 as an infrared image, and after a series of ISP image processing, the image is sent to the registration module; (3) the registration module first performs offset correction, and then uses the tform matrix to calculate the registration coordinates for interpolation to obtain the registered infrared image, as shown in FIG. Figure 6 As shown, the registered infrared image is then given to the fusion module; (4) The fusion module fuses the registered infrared image and the visible light image and outputs them. The original image needs to be offset by Δx and Δy to obtain an offset image before subsequent registration interpolation operations can be performed. According to the positive and negative conditions of Δx and Δy, there are Figure 7 There are four cases shown in the figure. In the solution of stream mode registration implemented by FPGA (without DDR cache), in order to achieve the above offset, according to the maximum value of Δy, which may be plus or minus 30, FPGA needs to open up a RAM storage space to store 30 lines of data to implement the edge filling operation. However, for the system, the RAM resource consumption occupied by the storage of 30 lines of data is too large. Another solution from the perspective of resources is to directly save this consumption. For the upper and lower parts beyond the original image, directly do not output data or output a completely black area, such as Figure 8 (a) and Figure 8As shown in (b), only left and right edge padding is supported, but top and bottom edge padding is not supported. After using this solution, although RAM resources can be saved, when Δy exceeds the support range of the original image, the image will have upper and lower black edges, such as Figure 8 (c) and Figure 8 (d) If the extreme case of Δy is to be supported, 30 rows of RAM resources need to be consumed, which is too much for the system; if no RAM resources are consumed, there is no DDR storage, and upper and lower padding is not supported, there will be upper and lower black edges.

[0046] See also Fig. 9 and Fig.10 , Fig. 9 is a flowchart of another implementation method of the image registration method in the prior art. Fig.10 FIG. 1 is a schematic diagram of another implementation of the edge filling operation. Fig. 9 As shown, the steps of the above-mentioned image registration method are as follows: (1) the image taken by camera 1 is sent to FPGA for collection via sensor 1 as a visible light image, and after a series of ISP image processing, the image is stored in DDR cache area 1; (2) the image taken by camera 2 is sent to FPGA for collection via sensor 2 as an infrared image, and after a series of ISP image processing, the image is given to the registration module; (3) the registration module first performs offset correction, then calculates the registration coordinates and interpolates to obtain the registered infrared image, and writes it to DDR cache area 2 after the processing is completed; (4) the registered infrared image is taken out from DDR cache area 2, the edges are filled, the black edges are removed, and the image is sent to the fusion module; (5) the fusion module takes out the visible light image from DDR cache area 1, fuses it with the edge-filled infrared image, and then outputs it. Among them, the registration module and Figure 5 The registration module in is the same as that in Figure 6 Similarly, the offset correction in this method is Figure 6 The offset correction is the same as in Figure 6 The difference is that this method does not use 30 lines of RAM cache, and the image written to DDR by the registration output has upper and lower black edges, such as Figure 8 (c) and Figure 8 As shown in (d). After the DDR cache, the logic controlling the reading of DDR can implement the upper and lower edge padding processing to solve the black edge problem. In the current product line common design, repeated edge padding is adopted. For the black edge above the correction image, the first row of the correction image is used to pad |Δy| rows. For the black edge below the correction image, the last row of the correction image is used to pad |Δy| rows. Fig.10As shown. This method only has 1 level of DDR processing, and knows the maximum value range of Δy. It can control the time of entering and exiting DDR without consuming more than 1 frame, and only needs to set a delay of about 10 lines. Although this method can reduce the delay and solve the problem of excessive RAM storage resource consumption and black edges, the edge filling operation is performed after the correction is completed, and the edge filling is copied, which deviates from the result obtained by the original image first filling and then correcting the required algorithm principle, resulting in unsatisfactory edge filling effect.

[0047] The following is an introduction to the image registration method provided by this application.

[0048] See also Fig.11 , Fig.11 : is a flow chart of an embodiment of the image registration method of the present application. The image registration method comprises:

[0049] S1: Acquire a first image of the object to be measured obtained by a first camera device and a second image of the object to be measured obtained by a second camera device.

[0050] Specifically, the first image captured by the first camera device is sent to the FPGA for collection through the first sensor. The collection method is consistent with the prior art and will not be described in detail in this application. The image provided by the first sensor is a picture that exceeds the image size we need. Assuming that the effective data resolution of the normal output of the first sensor is 2560x1440, the maximum resolution that the first sensor can provide can reach 2704x1568. Fig.12 As shown, Fig.12 is a schematic diagram of the sensor manual pixel array. During the FPGA acquisition process, the first center image is extracted according to the starting coordinates (x0, y0), the effective width W (such as 2560) and the effective height H (such as 1440). The second camera captures the second image through the second sensor and sends it to the FPGA for acquisition. The second image is a large image, and its resolution is the same as that of the first image, that is, Fig.12 Medium resolution image 2704x1568.

[0051] S2: Using the first image to perform offset correction on the second image to obtain a corrected image.

[0052] Specifically, in this embodiment, please refer to Fig.13 , Fig.13 yes Fig.11 Schematic diagram of a flow chart of an implementation method of step S2 in FIG. Specifically, step S2 includes:

[0053] S10: Obtain a first central image from the first image.

[0054] Specifically, in this embodiment, please refer to Fig.14 and Fig.15 , Fig.14 yes Fig.13 A schematic flow chart of an implementation method of step S10 in FIG. Fig.15 yes Fig.14 Steps S100-S101 in the figure correspond to a schematic diagram of an implementation method. Specifically, step S10 includes:

[0055] S100: Setting first starting coordinates of a first central image according to a difference between a resolution of the first image and a target resolution.

[0056] Specifically, the target resolution is smaller than the resolution of the first image. For example, see Fig.15 The resolution of the visible light image sent by the first sensor to the FPGA is 1960x1110, and the target resolution is set to 1920x1080.

[0057] In this embodiment, the resolution of the first image includes a first width value and a first height value, and the target resolution includes a second width value and a second height value. Specifically, the first width value is 1960, the first height value is 1110, the second width value is 1920, and the second height value is 1080.

[0058] Preferably, in this embodiment, please refer to Fig.15 and Fig.16 , Fig.16 yes Fig.14 Schematic diagram of a flow chart of an implementation method of step S100 in FIG. Specifically, step S100 includes:

[0059] S200: Obtain a width difference between a first width value and a second width value, and obtain a height difference between a first height value and a second height value.

[0060] Specifically, the width difference is 1960-1920=40, and the height difference is 1110-1080=30.

[0061] S201: taking half of the width difference as the abscissa value of the first starting coordinate, and taking half of the height difference as the ordinate value of the first starting coordinate, to obtain the first starting coordinate of the first central image.

[0062] Specifically, the first starting coordinates include a horizontal coordinate value x0 and a vertical coordinate value y0. Optionally, the horizontal coordinate value x0=40 / 2=20, and the vertical coordinate value y0=30 / 2=15, that is, the first starting coordinates are (20, 15). In this way, the first center image can be located in the middle of the first image, so that the image obtained in the subsequent process is more accurate. Of course, in other embodiments, the first starting coordinates can also be determined by other methods, which are not limited in this application.

[0063] S101: Obtain a first central image with a target resolution from a first image using first starting coordinates.

[0064] For more details, please refer to Fig.15 , in the first image, a first center image with a resolution of 1920x1080 is obtained with the first starting coordinate (20,15) as the starting point. In this way, the first center image can be located at the center of the first image, and a correction image with a more appropriate position can be obtained in the subsequent process.

[0065] S11: Using the first central image to perform offset correction on the second image to obtain a corrected image.

[0066] Specifically, in this embodiment, please refer to Fig.17 and Fig.18 , Fig.17 yes Fig.13 A schematic flow chart of an implementation method of step S11 in FIG. Fig.18 yes Fig.17 Steps S110-S113 in the figure correspond to a schematic diagram of an implementation method. Specifically, step S11 includes:

[0067] S110: Initializing the coordinates of the second image using the first starting coordinates, and obtaining a second central image having the same resolution as the first central image from the second image.

[0068] Specifically, the resolution of the second central image is the same as that of the first central image, which is also 1920x1080. During initialization, the starting coordinates of the second central image are the same as those of the first starting coordinates, which is (20, 15). The second central image is obtained through the offset cropping module according to the above parameters.

[0069] S111: Obtaining an offset parameter of the second central image according to the first central image and the second central image.

[0070] Specifically, the offset parameter includes a horizontal coordinate offset parameter value and a vertical coordinate offset parameter value. When initialized, the horizontal coordinate offset parameter value Δx is 0, and the vertical coordinate offset parameter value Δy is 0. At this time, the position of the first center image is consistent with the position of the second center image, which is also the center image. The first center image acquired by visible light image capture and the second center image obtained by offset cropping are sent to the online registration device, and the offset parameter Δx / Δy is obtained by calculation. The offset parameter here is constantly changing. The optimal offset parameter is obtained by continuous calculation, and the offset parameter is passed to the offset cropping module. In this embodiment, please continue to refer to Fig.15 , the horizontal coordinate offset parameter value Δx is less than the horizontal coordinate value x0 of the first starting coordinate, and the vertical coordinate offset parameter value Δy is less than the vertical coordinate value y0 of the first starting coordinate.

[0071] In addition, in this embodiment, please continue to refer to Fig.18 , the resolution of the second image includes a third width value and a third height value, the third width value is greater than the sum of the second width value and twice the maximum value max|Δx| of the absolute value of the horizontal coordinate offset parameter value, and the third height value is greater than the sum of the second height value and twice the maximum value max|Δy| of the absolute value of the vertical coordinate offset parameter value.

[0072] S112: Performing an offset process on the first starting coordinate using the offset parameter to obtain a second starting coordinate.

[0073] Specifically, the offset cropping module receives the newly configured offset parameters Δx / Δy. For example, Fig.18 As shown, assuming that the obtained Δx value is -10 and the Δy value is 5, the abscissa value x0 of the second starting coordinate is 20-(-10)=30, and the ordinate value y0 of the second starting coordinate is 15-5=10. Therefore, the second starting coordinate is (30,10).

[0074] S113: Acquire a corrected image with a target resolution from the second image using the second starting coordinates.

[0075] For more details, please refer to Fig.18 , in the second image, the second starting coordinate (30,10) is used as the starting point, and the second image is offset and cropped according to the target resolution 1920x1080 to obtain a corrected image. The corrected image obtained in this way has a more appropriate position in the second image and will not be too biased in a certain direction, thereby improving the efficiency of image registration.

[0076] In this embodiment, the second central image is obtained according to the target resolution, the second central image is offset using the offset parameter to obtain an offset map, and the first image is cropped according to the offset map to obtain a corrected image. Fig.19 , Fig.19 yes Fig.17 Schematic diagram corresponding to the four cases of offset parameters. According to the positive and negative conditions of the offset parameters Δx and Δy, the obtained offset map and corrected image have the following four cases. Fig.19 As shown, when Δx>0 and Δy>0, as Fig.19 As shown in a, the offset map is offset to the upper left of the first central image; when Δx<0 and Δy>0, as Fig.19 As shown in b, the offset map is offset to the upper right of the first central image; when Δx>0 and Δy<0, as shown in Fig.19 As shown in c, the offset map is offset to the lower left of the first central image; when Δx<0 and Δy<0, as Fig.19As shown in FIG. 3 , the offset image is offset to the lower right of the first center image. The corrected image obtained in this way does not have a black edge. The corrected image is then directly registered. It not only does not require the use of additional RAM resources to cache multiple lines for edge filling, but also can solve the problems of black edges and poor edge filling effect, thereby improving the efficiency of image registration.

[0077] S3: Perform interpolation operation on the corrected image to obtain the registered image.

[0078] Specifically, in this embodiment, please refer to Fig. 20 , Fig. 20 yes Fig.11 Schematic diagram of a flow chart of an implementation method of step S3 in FIG. Specifically, step S3 includes:

[0079] S20: Obtain a transformation matrix using the first central image and the correction image.

[0080] Specifically, a transformation matrix (tform matrix) is obtained according to the first central image and the correction image through an online registration device.

[0081] S21: Obtaining the registration coordinates of the corrected image according to the transformation matrix.

[0082] Specifically, the registration coordinates of the corrected image are obtained using the tform matrix obtained in step S20.

[0083] S22: Performing an interpolation operation on the corrected image according to the registration coordinates to obtain a registered image.

[0084] Specifically, the resolution of the registered image is the target resolution. In this embodiment, the above-mentioned step of interpolating the corrected image according to the registration coordinates is consistent with the solution in the prior art, and this application will not repeat it here. By adjusting the position of the corrected image by the registration coordinates, a better quality registered image can be obtained.

[0085] Specifically, in this embodiment, before step S3, the process includes: performing image signal processing on the corrected image, thereby improving the image quality of the corrected image, and further improving the quality of the fused image outputted at last. In this embodiment, the above steps are consistent with the solutions in the prior art, and the present application will not repeat them here.

[0086] S4: Fusing the first image and the registered image to obtain a fused image.

[0087] Specifically, in this embodiment, step S4 specifically includes: fusing the first central image and the registration image to obtain a fused image. In this embodiment, the above steps are consistent with the solutions in the prior art, and the present application will not repeat them here.

[0088] Through such a design, the use of streaming mode registration can reduce latency. After acquiring the second image, offset correction is performed to obtain a corrected image without black edges. The corrected image is directly registered. Not only does it not require the use of additional RAM resources to cache multiple lines for edge filling, but it can also solve problems such as black edges and poor edge filling effects, thereby improving the efficiency of image registration.

[0089] In addition, in this embodiment, before the step of fusing the first central image and the registered image to obtain a fused image, the method further includes: performing image signal processing on the first central image, thereby improving the image quality of the first central image, and further improving the quality of the fused image outputted at last. In this embodiment, the above steps are consistent with the solutions in the prior art, and the present application will not repeat them here.

[0090] In addition, in this embodiment, see Fig.21 , Fig.21 yes Fig.11 Step S4 in the figure corresponds to a flow chart of an implementation method. After the step of fusing the first central image and the registered image to obtain a fused image, the method includes:

[0091] S30: Determine whether there is ghosting in the fused image.

[0092] Specifically, the effect of the fused image finally outputted can be viewed through a display or the like to determine whether there is ghosting in the fused image.

[0093] S31: If yes, return to the step of using the first image to perform offset correction on the second image to obtain a corrected image.

[0094] Specifically, if there is ghosting in the fused image, the offset parameters Δx / Δy and the tform matrix and other parameters are adjusted through the online registration device, and the corrected image and the registered image are re-acquired according to the adjusted parameters, so as to optimize the quality of the final output fused image.

[0095] S32: Otherwise, output the fused image.

[0096] Specifically, if there is no ghosting in the fused image, it means that the fused image meets the output standard, and the obtained fused image is directly output. In this way, it can be ensured that the output fused image is the optimal solution, thereby improving the efficiency of image registration.

[0097] See also Fig. 22 , Fig. 22 1 is a schematic diagram of the structure of an embodiment of the image registration device of the present application. The image registration device specifically includes:

[0098] The acquisition module 10 is used to acquire a first image of the object to be measured obtained by a first camera device and a second image of the object to be measured obtained by a second camera device.

[0099] The correction module 12 is coupled to the acquisition module 10 and is used to perform offset correction on the second image using the first image to obtain a corrected image.

[0100] The registration module 14 is coupled to the correction module 12 and is used to perform an interpolation operation on the correction image to obtain a registered image.

[0101] The fusion module 16 is coupled to the registration module 14 and is used to fuse the first image and the registered image to obtain a fused image.

[0102] See also Fig.23 , Fig.23 1 is a schematic diagram of a framework of an electronic device of the present application. The electronic device includes a memory 20 and a processor 22 coupled to each other. Specifically, in this embodiment, the memory 20 stores program instructions, and the processor 22 is used to execute the program instructions to implement the image registration method mentioned in any of the above embodiments.

[0103] Specifically, the processor 22 may also be referred to as a CPU (Central Processing Unit). The processor 22 may be an integrated circuit chip having the ability to process signals. The processor 22 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. In addition, the processor 22 may be implemented by a plurality of integrated circuit chips.

[0104] See also Fig.24 , Fig.24It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. The computer-readable storage medium 30 stores a computer program 300, which can be read by a computer, and the computer program 300 can be executed by a processor to implement the image registration method mentioned in any of the above embodiments. Among them, the computer program 300 can be stored in the above-mentioned computer-readable storage medium 30 in the form of a software product, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The computer-readable storage medium 30 with a storage function can be a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, RandomAccess Memory), a disk or an optical disk, etc., which can store program codes, or a terminal device such as a computer, a server, a mobile phone, and a tablet.

[0105] In summary, different from the prior art, the present application provides a method of obtaining a first image and a second image of the same object to be measured by a first camera device and a second camera device, using the first image to perform offset correction on the second image to obtain a corrected image, interpolating the corrected image to obtain a registered image, and using the first image and the registered image to obtain a fused image. With such a design, the use of flow mode registration can reduce latency, and after acquiring the second image, offset correction is performed to obtain a corrected image without black edges, and the corrected image is directly registered, which not only does not require the use of additional RAM resources to cache multiple lines for edge filling, but also can solve problems such as black edges and poor edge filling effects, thereby improving the efficiency of image registration.

[0106] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An image registration method, characterized in that: include: Acquire a first image of the object to be measured taken by a first camera and obtained by a second camera and obtained by a second camera and obtained by a second camera; Using the first image, performing offset correction on the second image to obtain a corrected image; Performing an interpolation operation on the corrected image to obtain a registered image; fusing the first image and the registered image to obtain a fused image; The step of using the first image to perform offset correction on the second image to obtain a corrected image comprises: obtaining a first central image from the first image; The second image is offset corrected using the first central image to obtain a corrected image.

2. The image registration method according to claim 1, characterized in that: The step of obtaining a first central image from the first image comprises: Setting a first starting coordinate of the first center image according to a difference between a resolution of the first image and a target resolution; wherein the target resolution is smaller than a resolution of the first image; The first central image having the target resolution is obtained from the first image using the first starting coordinates.

3. The image registration method according to claim 2, characterized in that: The step of using the first central image to perform offset correction on the second image to obtain a corrected image comprises: Initializing the coordinates of the second image using the first starting coordinates, and obtaining a second central image having the same resolution as the first central image from the second image; Obtaining an offset parameter of the second central image according to the first central image and the second central image; Performing an offset process on the first starting coordinate using the offset parameter to obtain a second starting coordinate; The corrected image having the target resolution is acquired from the second image using the second starting coordinates.

4. The image registration method according to claim 3, characterized in that: The resolution of the first image includes a first width value and a first height value, and the target resolution includes a second width value and a second height value; the step of setting the first starting coordinates of the first center image according to the difference between the resolution of the first image and the target resolution includes: Obtaining a width difference between the first width value and the second width value, and obtaining a height difference between the first height value and the second height value; A half of the width difference is used as the abscissa value of the first starting coordinate, and a half of the height difference is used as the ordinate value of the first starting coordinate, so as to obtain the first starting coordinate of the first central image.

5. The image registration method according to claim 4, characterized in that: The offset parameter includes a horizontal coordinate offset parameter value and a vertical coordinate offset parameter value, and the horizontal coordinate offset parameter value is smaller than the horizontal coordinate value of the first starting coordinate, and the vertical coordinate offset parameter value is smaller than the vertical coordinate value of the first starting coordinate; The resolution of the second image includes a third width value and a third height value; the third width value is greater than the sum of the second width value and twice the maximum value of the absolute value of the horizontal coordinate offset parameter value, and the third height value is greater than the sum of the second height value and twice the maximum value of the absolute value of the vertical coordinate offset parameter value.

6. The image registration method according to claim 1, characterized in that: The step of performing an interpolation operation on the corrected image to obtain a registered image comprises: Obtaining a transformation matrix using the first central image and the corrected image; Obtaining registration coordinates of the corrected image according to the transformation matrix; An interpolation operation is performed on the corrected image according to the registration coordinates to obtain a registered image; wherein the resolution of the registered image is a target resolution.

7. The image registration method according to claim 1, characterized in that: Before the step of performing interpolation operation on the corrected image to obtain the registered image, the method comprises: performing image signal processing on the corrected image; The step of fusing the first image and the registered image to obtain a fused image comprises: The first central image and the registered image are fused to obtain the fused image.

8. The image registration method according to claim 7, characterized in that: Before the step of fusing the first central image and the registered image to obtain the fused image, the method further includes: performing image signal processing on the first central image; After the step of fusing the first central image and the registered image to obtain the fused image, the method further comprises: In response to the presence of ghosting in the fused image, the process returns to the step of using the first image to perform offset correction on the second image to obtain a corrected image.

9. An electronic device, characterized in that: The invention comprises a memory and a processor coupled to each other, wherein the memory stores program instructions, and the processor is used to execute the program instructions to implement the image registration method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is used to implement the image registration method according to any one of claims 1 to 8.

Images