Image stitching method, device, equipment and storage medium

Abstract

The application provides an image splicing method and device, equipment and a storage medium, and relates to the technical field of image processing. The method comprises the following steps: acquiring a first image, a second image and a calibration plate image of a target object; determining a splicing relationship between the first image and the second image according to the calibration plate image; the splicing relationship is used to indicate a splicing operation of splicing the second image to the first image; optimizing the splicing relationship to obtain an optimized splicing relationship; the fitting degree of the optimized splicing relationship and the first image is greater than the fitting degree of the splicing relationship and the first image, and / or the fitting degree of the optimized splicing relationship and the second image is greater than the fitting degree of the splicing relationship and the second image; and splicing the first image and the second image based on the optimized splicing relationship to obtain a spliced image.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image stitching method, apparatus, device and storage medium. Background Technology

[0002] Image stitching is the process of combining multiple images of the same scene into a larger image, allowing more information to be presented in a single frame. When stitching images of a particular object, a calibration board is typically used to determine the stitching relationships between the images, and the images are then stitched together according to these relationships.

[0003] However, due to the thickness of the calibration plate, the calibration plate and the actual object are usually not on the same plane, which results in the calibration plate image being able to be stitched well, while the image of the actual object is stitched poorly. Summary of the Invention

[0004] To address the aforementioned technical issues, this application provides an image stitching method, apparatus, device, and storage medium that can improve image stitching results.

[0005] In a first aspect, this application provides an image stitching method, the method comprising: acquiring a first image, a second image, and a calibration board image of a target object; the first image corresponding to a first part of the target object, the second image corresponding to a second part of the target object, and the first image and the second image having an overlapping area; the calibration board image being an image corresponding to a calibration board used to calibrate the target object; determining a stitching relationship between the first image and the second image based on the calibration board image; the stitching relationship being used to indicate a stitching operation of stitching the second image to the first image; optimizing the stitching relationship to obtain an optimized stitching relationship; the degree of fit between the optimized stitching relationship and the first image being greater than the degree of fit between the stitching relationship and the first image, and / or, the degree of fit between the optimized stitching relationship and the second image being greater than the degree of fit between the stitching relationship and the second image; and stitching the first image and the second image based on the optimized stitching relationship to obtain a stitched image.

[0006] In this application, an electronic device stitches together different images with overlapping areas. During the stitching process, a calibration board image containing the same target object in the overlapping area is used as a reference. Since the calibration board image corresponds to the calibration board used to calibrate the target object, after the electronic device optimizes the stitching relationship between different images based on the calibration board image, the optimized stitching relationship can better fit the image area associated with the target object in different images. Furthermore, when images containing different parts of the same target object have certain deviations, the electronic device uses a calibration board image with a complete target object as a reference to determine the optimized stitching relationship, and performs stitching between images based on the optimized stitching relationship. This improves the display effect of the target object in the stitched image and enhances the precision and accuracy of image stitching.

[0007] In one possible implementation, the stitching relationship is optimized to obtain an optimized stitching relationship, including: determining the overlapping area between the first image and the second image; optimizing the stitching relationship based on the overlapping area to obtain an optimized stitching relationship; the pose adjustment of the target object in the overlapping area by the optimized stitching relationship is different from the pose adjustment of the target object in the overlapping area by the stitching relationship.

[0008] In one possible implementation, the stitching relationship includes multiple image transformation parameters; based on the overlapping region, the stitching relationship is optimized to obtain an optimized stitching relationship, including: determining the stitching error between the first image and the second image in the overlapping region; and adjusting each image transformation parameter according to the stitching error to obtain the optimized stitching relationship.

[0009] In one possible implementation, the image transformation parameters are adjusted according to the stitching error to obtain an optimized stitching relationship. This includes: for any image transformation parameter, the image transformation parameter is adjusted by searching the neighborhood data of the image transformation parameter until the stitching error is less than or equal to a preset threshold.

[0010] In one possible implementation, the stitching relationship includes multiple image transformation parameters; based on the overlapping region, the stitching relationship is optimized to obtain an optimized stitching relationship, including: determining the image transformation parameters to be adjusted; the image transformation parameters to be adjusted are the image transformation parameters that undergo geometric transformations among the multiple image transformation parameters; determining the stitching error between the first image and the second image in the overlapping region; and adjusting the image transformation parameters to be adjusted according to the stitching error to obtain the optimized stitching relationship.

[0011] In one possible implementation, the optimized stitching relationship includes a first optimized stitching relationship and a second optimized stitching relationship; the first optimized stitching relationship is determined during the calibration stage of the target object; the second optimized stitching relationship is determined during the stitching stage of the image of the target object.

[0012] Secondly, this application provides an image stitching apparatus, which includes an acquisition module, a determination module, a processing module, and a stitching module; the acquisition module is used to acquire a first image, a second image, and a calibration plate image of a target object; the first image corresponds to a first part of the target object, the second image corresponds to a second part of the target object, and the first image and the second image have an overlapping area; the calibration plate image is an image corresponding to a calibration plate used to calibrate the target object; the determination module is used to determine the stitching relationship between the first image and the second image based on the calibration plate image; the stitching relationship is used to indicate the stitching operation of stitching the second image to the first image; the processing module is used to optimize the stitching relationship to obtain an optimized stitching relationship; the degree of adaptation between the optimized stitching relationship and the first image is greater than the degree of adaptation between the stitching relationship and the first image, and / or, the degree of adaptation between the optimized stitching relationship and the second image is greater than the degree of adaptation between the stitching relationship and the second image; the stitching module is used to stitch the first image and the second image based on the optimized stitching relationship to obtain a stitched image.

[0013] In one possible implementation, the processing module is specifically used to: determine the overlapping area between the first image and the second image; optimize the stitching relationship based on the overlapping area to obtain an optimized stitching relationship; the pose adjustment of the target object in the overlapping area by the optimized stitching relationship is different from the pose adjustment of the target object in the overlapping area by the stitching relationship.

[0014] In one possible implementation, the stitching relationship includes multiple image transformation parameters; the processing module is specifically used to: determine the stitching error between the first image and the second image in the overlapping area; and adjust each image transformation parameter according to the stitching error to obtain an optimized stitching relationship.

[0015] In one possible implementation, the processing module is specifically used to: for any image transformation parameter, adjust the image transformation parameter by searching the neighborhood data of the image transformation parameter until the stitching error is less than or equal to a preset threshold.

[0016] In one possible implementation, the stitching relationship includes multiple image transformation parameters; the processing module is specifically used to: determine the image transformation parameters to be adjusted; the image transformation parameters to be adjusted are the image transformation parameters that undergo geometric transformation among the multiple image transformation parameters; determine the stitching error between the first image and the second image in the overlapping area; and adjust the image transformation parameters to be adjusted according to the stitching error to obtain an optimized stitching relationship.

[0017] In one possible implementation, the optimized stitching relationship includes a first optimized stitching relationship and a second optimized stitching relationship; the first optimized stitching relationship is determined during the calibration stage of the target object; the second optimized stitching relationship is determined during the stitching stage of the image of the target object.

[0018] Thirdly, this application provides an electronic device, including: a processor and a memory; the memory stores processor-executable instructions; when the processor is configured to execute the instructions, the electronic device implements the method described in the first aspect above.

[0019] Fourthly, this application provides a computer program product that, when run in an electronic device, causes the electronic device to execute the methods related to the first aspect described above, thereby implementing the methods of the first aspect.

[0020] Fifthly, this application provides a computer-readable storage medium comprising: software instructions; which, when executed in an electronic device, cause the electronic device to implement the method described in the first aspect.

[0021] The beneficial effects of the second to fifth aspects mentioned above can be referred to the first aspect, and will not be repeated here. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 The image calibration process of the related technology provided in the embodiments of this application;

[0024] Figure 2 This is a schematic diagram illustrating the effect of splicing using related technologies, provided as an embodiment of this application.

[0025] Figure 3 This is a schematic diagram of the composition of the image stitching system provided in the embodiments of this application;

[0026] Figure 4 A schematic diagram illustrating the composition of the electronic device provided in the embodiments of this application;

[0027] Figure 5 A schematic flowchart illustrating the image stitching method provided in an embodiment of this application;

[0028] Figure 6 This is a schematic diagram illustrating the effect of splicing using the original splicing gate, provided in an embodiment of this application.

[0029] Figure 7 This is a schematic diagram illustrating the effect of splicing using optimized splicing relationships, provided in an embodiment of this application.

[0030] Figure 8This is a schematic diagram illustrating the composition of the image stitching device provided in an embodiment of this application. Detailed Implementation

[0031] To enable those skilled in the art to better understand the technical solutions of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0032] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0033] Furthermore, in the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The term "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, in the description of the embodiments of this application, "multiple" refers to two or more.

[0034] Before providing a detailed explanation of the embodiments of this application, some related terms and technologies involved in the embodiments of this application will be introduced first.

[0035] Calibration Target: In applications such as machine vision, image measurement, photogrammetry, and 3D reconstruction, a geometric model of camera imaging is needed to correct lens distortion, determine the conversion relationship between physical dimensions and pixels, and determine the relationship between the 3D geometric position of a point on the surface of a spatial object and its corresponding point in the image. By photographing a plate with a fixed-spacing pattern array and performing calibration algorithms, the geometric model of the camera can be obtained, thus yielding high-precision measurement and reconstruction results. The plate with the fixed-spacing pattern array is the calibration target.

[0036] Image stitching: Image stitching is the process of combining multiple images from the same scene into a larger image, so that more information can be presented in the same picture.

[0037] In some close-range image acquisition scenarios, it is necessary to capture images over a wide field of view. This is typically achieved using multiple cameras to acquire images, which are then stitched together. When stitching images, a calibration board is usually used to determine the stitching relationships between the images, and the images are then stitched together according to these relationships.

[0038] like Figure 1 The diagram illustrates the image calibration process of a related technology, which can be divided into a calibration stage and a stitching stage. In the calibration stage, a calibration board is typically placed on the target object to be stitched, and then an array of cameras is used to acquire images containing the calibration board, with each camera capturing a portion of the calibration board. Further, the acquired images are processed to extract image features, and then the physical coordinates and image coordinates of the features on the calibration board are obtained. The transformation relationship between the physical plane and the image plane is then calculated to obtain calibration information. This calculated transformation relationship is actually the transformation relationship between the calibration board plane and the pixel plane, i.e., the stitching relationship. In the stitching stage, the camera acquires images of the target object to be stitched, and the previously calculated transformation relationship is used to stitch the images together.

[0039] The above method can be used when the target object is large, the camera is far from the target object, and the stitching accuracy requirement is not high. However, when the target object is small, the camera is close to the target, and the stitching accuracy requirement is high, the above method cannot meet the requirements. Specifically, due to the thickness of the calibration plate, or for other reasons, the calibration plate and the actual target object are usually not on the same plane. This results in the calibration plate image being stitched correctly, while the image of the actual target object cannot be stitched correctly. For example... Figure 2 As shown in the diagram, the image of the calibration board can be stitched together successfully, but the stitching area of ​​the target object image below the calibration board shows ghosting, resulting in a poor stitching effect.

[0040] In view of the above problems, embodiments of this application provide an image stitching method, apparatus, device and storage medium, which improves the stitching scheme of related technologies, so that the images of the target objects under the calibration board can also be aligned, achieving a good stitching effect.

[0041] The image stitching method provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0042] The image stitching method provided in this application can be applied to image stitching systems. Figure 3 A schematic diagram of one structure of this image stitching system is shown. For example... Figure 3As shown, the image stitching system includes an image acquisition device 100 and an image stitching device 200. The image acquisition device 100 and the image stitching device 200 can be connected via a wired or wireless network.

[0043] The image acquisition device 100 can be used to acquire images of a target object.

[0044] The image acquisition device 100 can be a camera or network camera (IPC) set in a specific location.

[0045] In some embodiments, the image acquisition device 100 can also be used to send the acquired images to the image stitching device 200.

[0046] As described above, the image acquisition device 100 and the image stitching device 200 can be connected via a wired network or a wireless network. This wired or wireless network may include one or more media or devices capable of transmitting image data from the image acquisition device 100 to the image stitching device 200.

[0047] In some embodiments, the wired or wireless network may include one or more communication media that enable the image acquisition device 100 to transmit image data directly to the image stitching device 200 in real time. In this embodiment, the image acquisition device 100 may modulate the image data according to a communication standard (e.g., a wireless communication protocol) and transmit the modulated image data to the image stitching device 200. The one or more communication media may include wireless and / or wired communication media, such as radio frequency (RF) spectrum or one or more physical transmission lines. Optionally, the one or more communication media may form part of a packet-based network, such as a local area network, a wide area network, or a global network (e.g., the Internet). Optionally, the one or more communication media may also include routers, switches, base stations, or other devices that facilitate communication between the image acquisition device 100 and the image stitching device 200.

[0048] The image stitching device 200 can be used to stitch together the images acquired by the image acquisition device 100 to obtain a stitched image. The specific stitching process can be referred to the image stitching method described in the following method embodiments, which will not be repeated here.

[0049] The image stitching device 200 can be an electronic device with computing and processing capabilities, such as a computer or server.

[0050] The server can be a single server or a server cluster consisting of multiple servers. In some embodiments, the server cluster can also be a distributed cluster. Optionally, the server can also be implemented on a cloud platform, such as a private cloud, public cloud, hybrid cloud, community cloud, distributed cloud, inter-cloud, and multi-cloud, or any combination thereof. This application does not limit this aspect.

[0051] It should be noted that the above Figure 3 The example described uses the image acquisition device 100 and the image stitching device 200 as independent devices. Optionally, the image acquisition device 100 and the image stitching device 200 can also be combined into one device. For example, the image acquisition device 100 or its corresponding functions, and the image stitching device 200 or its corresponding functions can be integrated into one device. This application does not limit this.

[0052] The image stitching method provided in this application embodiment can be executed by the image stitching device 200 described above. As mentioned above, the image stitching device 200 can be an electronic device with computing processing capabilities, such as a computer or server. Optionally, the image stitching device 200 can also be a processor (e.g., a central processing unit (CPU)) in the aforementioned electronic device; or, the image stitching device 200 can also be an application (APP) with passenger behavior detection function installed in the aforementioned electronic device; or, the image stitching device 200 can also be a functional module with passenger behavior detection function in the aforementioned electronic device, etc. This application embodiment does not impose any limitations on this.

[0053] For simplicity, the image stitching device 200 will be used as an example for the following description.

[0054] Figure 4 This is a schematic diagram illustrating the composition of an electronic device provided in an embodiment of this application. For example... Figure 4 As shown, the electronic device may include: a processor 10, a memory 20, a communication line 30, a communication interface 40, and an input / output interface 50.

[0055] The processor 10, memory 20, communication interface 40, and input / output interface 50 can be connected via communication line 30.

[0056] The processor 10 is used to execute instructions stored in the memory 20 to implement the image stitching method provided in the following embodiments of this application. The processor 10 can be a CPU, a network processor (NP), a digital signal processor (DSP), a microprocessor, a microcontroller (MCU) / single-chip microcomputer / microcontroller, a programmable logic device (PLD), or any combination thereof. The processor 10 can also be any other device with processing capabilities, such as a circuit, device, or software module; this embodiment of the application does not limit this. In one example, the processor 10 may include one or more CPUs, for example... Figure 4 CPU0 and CPU1 are mentioned. As an optional implementation, the electronic device may include multiple processors; for example, in addition to processor 10, it may also include processor 60. Figure 4 (The example shown is a dashed line).

[0057] The memory 20 is used to store instructions. For example, the instructions may be computer programs. Optionally, the memory 20 may be a read-only memory (ROM) or other types of static storage devices that can store static information and / or instructions; it may also be a random access memory (RAM) or other types of dynamic storage devices that can store information and / or instructions; it may also be an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media, or other magnetic storage devices, etc. The embodiments of this application do not limit this.

[0058] It should be noted that the memory 20 can exist independently of the processor 10 or it can be integrated with the processor 10. The memory 20 can be located inside or outside the electronic device, and this application embodiment does not impose any restrictions on this.

[0059] Communication line 30 is used to transmit information between the components included in the electronic device.

[0060] The communication interface 40 is used to communicate with other devices (such as the image acquisition device 100 described above) or other communication networks. These other communication networks can be Ethernet, radio access network (RAN), wireless local area network (WLAN), etc. The communication interface 40 can be a module, circuit, transceiver, or any device capable of enabling communication.

[0061] Input / output interface 50 is used to enable human-computer interaction between users and electronic devices. For example, it enables action interaction or information exchange between users and electronic devices.

[0062] For example, the input / output interface 50 can be a mouse, keyboard, display screen, or touch screen. Action or information interaction between the user and the electronic device can be achieved through a mouse, keyboard, display screen, or touch screen.

[0063] It should be noted that, Figure 4 The structures shown do not constitute a limitation on electronic devices, except... Figure 4 In addition to the components shown, electronic devices may include more or fewer components than illustrated, or combinations of certain components, or different component arrangements.

[0064] The image stitching method provided in the embodiments of this application will be described below.

[0065] Figure 5 This is a schematic flowchart illustrating the image stitching method provided in an embodiment of this application. Optionally, this method can be implemented by someone with the above-described... Figure 4 The electronic device with the hardware structure shown performs, such as Figure 5 As shown, the method includes S101 to S104.

[0066] S101, The electronic device acquires a first image, a second image, and a calibration board image of the target object.

[0067] The first image corresponds to the first part of the target object, the second image corresponds to the second part of the target object, and the first and second images have overlapping areas. The calibration board image is the image corresponding to the calibration board used to calibrate the target object.

[0068] As one possible implementation, electronic devices can acquire a first image, a second image, and a calibration plate image of the target object through an image acquisition device.

[0069] For example, the image acquisition device captures a first image of a first part of the target object and a second image of a second part of the target object. During the calibration process of the target object, a calibration plate is attached to the target object, and the image acquisition device captures an image of the calibration plate. Further, the image acquisition device sends the captured first image, second image, and calibration plate image to an electronic device.

[0070] As another possible implementation, the electronic device can obtain a first image, a second image, and a calibration board image of the target object from its own database.

[0071] For example, an electronic device can acquire a first image, a second image, and a calibration board image of the target object from a photo album.

[0072] S102. The electronic device determines the splicing relationship between the first image and the second image based on the calibration board image.

[0073] The stitching relationship is used to indicate the stitching operation of stitching the second image onto the first image. For example, electronic devices can read specific operations such as rotation, scaling, shearing, and translation during image stitching by using the stitching relationship.

[0074] As one possible implementation, during the calibration phase, the calibration board is typically placed on the target object to be stitched together. Then, a camera array is used to acquire images containing the calibration board, with one camera capturing a portion of the board, resulting in multiple images of the calibration board to be stitched together. Electronic equipment can process the acquired calibration board images to extract their features. Further, based on these features, the electronic equipment determines the physical and image coordinates of the calibration board. It can then calculate the transformation relationship (i.e., the stitching relationship) between the physical and image planes, obtaining calibration information containing this transformation relationship.

[0075] It should be noted that the transformation relationship calculated here is actually the transformation relationship between the calibration board plane and the pixel plane. Electronic devices can use this transformation relationship to stitch together multiple calibration board images to obtain a complete calibration board image.

[0076] In practical applications, the transformation relation contains parameters that transfer the image to be stitched from one pose to another. The pose includes information such as the scale, angle, and position of the target object. For example, the transformation relation can be described using homography matrices, quaternions, polynomial parameters, etc.

[0077] In some embodiments, one method for calculating the stitching relationship is to extract the first set of corner points and the second set of corner points from the first image and the second image respectively, and obtain the image coordinates of the two sets of corner points. and physical coordinates Based on the image coordinates and physical coordinates, the mapping relationships H1 and H2 from physical coordinates to image coordinates can be calculated (H1 and H2 can be calculated using well-known methods such as least squares, weighted least squares, and RANSAC). These relationships satisfy...

[0078]

[0079]

[0080] because and The coordinate system is the same for every point in the second image. If we map this point onto the first image, we only need to map the points in the second image to the physical coordinate system, and then map the points mapped to the physical coordinate system back onto the first image to obtain the coordinates of the final mapped point.

[0081]

[0082] H2 -1 This represents the inverse mapping relationship of H2.

[0083] Using the relation H1H2 -1 This allows you to map all the pixels in the second image onto the first image and merge them with the original pixels in the first image to obtain the final stitched image.

[0084] S103. The electronic device optimizes the splicing relationship to obtain an optimized splicing relationship.

[0085] Specifically, the degree of adaptation between the optimized stitching relationship and the first image is greater than the degree of adaptation between the optimized stitching relationship and the first image, and / or, the degree of adaptation between the optimized stitching relationship and the second image is greater than the degree of adaptation between the optimized stitching relationship and the second image. Optimizing the stitching relationship is equivalent to optimizing H1 and H2 or the entire H1H2. -1 Optimizations have been made to make the calculated optimization relationships more accurate.

[0086] Understandably, the above stitching relationship is determined by the calibration board image, and therefore it is more suitable for stitching calibration board images. Electronic devices can use this relationship to achieve a good stitching of the calibration board images to be stitched. However, if electronic devices use this stitching relationship to stitch the images of the target object, they cannot achieve a good stitching effect. Therefore, it is necessary to optimize this stitching relationship.

[0087] As one possible implementation, the electronic device can determine the overlapping region between the first and second images. Further, based on the overlapping region, the electronic device optimizes the stitching relationship to obtain an optimized stitching relationship. The pose adjustment of the target object in the overlapping region performed by the optimized stitching relationship differs from the pose adjustment performed by the stitching relationship.

[0088] Understandably, using information from overlapping areas in adjacent images to optimize stitching relationships can reduce the processing load on electronic devices and improve optimization efficiency.

[0089] In some embodiments, the stitching relationship includes multiple image transformation parameters; based on the overlapping area, the stitching relationship is optimized to obtain an optimized stitching relationship, including: the electronic device determines the stitching error between the first image and the second image in the overlapping area, and further, adjusts each image transformation parameter according to the stitching error to obtain an optimized stitching relationship.

[0090] For any given image transformation parameter, the electronic device can adjust the image transformation parameter by searching the neighborhood data of the image transformation parameter until the stitching error is less than or equal to a preset threshold.

[0091] In some embodiments, the above problem can be transformed into an optimization problem, where the optimization parameters are the image transformation parameters in the stitching relationship, and the optimization objective is to minimize the error after stitching the common areas (i.e., overlapping areas). This error can be measured using mean error, mean squared error, normalized crosscorrelation (NCC), or other error measurement methods.

[0092] For example, the error in measuring the overlapping regions between images is denoted as the loss function f. i,j Then the loss function can be expressed as:

[0093]

[0094] Where i and j represent the i-th and j-th images to be stitched, respectively, and n is the total number of images to be stitched. The electronic device can optimize this loss function to solve for the final stitching relationship. Optionally, the electronic device can also use optimization strategies to optimize the loss functions of some of the images to be stitched separately, and then merge them; this will not be elaborated further here.

[0095] In some embodiments, the stitching relationship includes multiple image transformation parameters. Since the various image transformation parameters (such as scale changes, angle changes, translation changes, etc.) in the stitching relationship to be optimized only differ from the original stitching relationship by a small range, the electronic device only needs to search in the neighborhood of each image transformation parameter to find the optimal stitching relationship, which can quickly achieve stitching relationship optimization and save optimization time.

[0096] For example, for any two images to be stitched, the electronic device can use template matching to optimize the transformation relationship. The specific template matching method can be chosen based on different features, such as feature point-based matching, contour-based matching, and grayscale-based matching. Using the first image as the stitching reference, after calibration using a calibration board, the transformation relationship (i.e., the stitching relationship) from the second image to the first image can be roughly obtained, including rotation, scaling, shearing, and translation changes. However, these transformation relationships are not accurate. The electronic device can correct the original transformation relationship by searching the neighborhood of each of these transformation relationships. The formula is as follows:

[0097] H correct =H origin H Δ

[0098] Among them, H correct It is the accurate transformation relationship after template matching improvement, H origin This is an inaccurate transformation relationship from the second image to the first image based on the original calculation, H. Δ This is the part about template matching correction. That is, H. correct The fit with the first image is greater than H. origin The degree of adaptation with the first image. Furthermore, for cases involving multiple stitched images, expansion can be made based on two stitched images, which will not be elaborated further in this application's embodiments.

[0099] In other embodiments, the stitching relationship is optimized based on the overlapping region to obtain an optimized stitching relationship, including: the electronic device determining the image transformation parameters to be adjusted. The image transformation parameters to be adjusted are the image transformation parameters that undergo geometric transformations among a plurality of image transformation parameters. Further, the electronic device determines the stitching error between the first image and the second image in the overlapping region, and adjusts the image transformation parameters to be adjusted based on the stitching error to obtain the optimized stitching relationship.

[0100] Understandably, different scenarios may require different image transformation parameters to be searched. Some image transformation parameters that do not change do not need to be searched. Fixing these parameters that do not need to be searched can improve search efficiency.

[0101] It should be noted that this application can use various methods to optimize the stitching relationship, or combine different methods. For example, when the actual model and parameters are known, the electronic device can actually calculate the stitching relationship to be optimized by using the relationship between the camera intrinsic parameters, the calibration plate plane and the target plane. It can also use optical flow to calculate the stitching relationship, or use information from common areas in adjacent images for optimization.

[0102] In some embodiments, in order to improve the optimization calculation speed, in addition to improving the specific method of transformation relationship, the electronic device may also choose to sample the common area of ​​the stitching. In one case, it samples a relatively large image to reduce the image size. In another case, when there are many features in the image, it samples the features to achieve a faster optimization speed.

[0103] S104. The electronic device stitches the first image and the second image together based on the optimized stitching relationship to obtain a stitched image.

[0104] It should be noted that the optimized stitching relationship (i.e., the optimized stitching relationship) is closer to the transformation relationship between the target object plane and the pixel plane.

[0105] Therefore, electronic devices use optimized stitching relationships to stitch the first and second images together to obtain a stitched image. Compared with stitching relationships obtained directly using a calibration board, the stitching effect is more accurate.

[0106] like Figure 6 The image shown is a stitching result using the original stitching relationship (i.e., the stitching relationship determined by the calibration board image). It can be seen that ghosting occurs in the image connection area, resulting in a poor stitching effect. Figure 7 This shows the stitching effect of image stitching using optimized stitching relationships. Figure 7 As you can see, the area where the images are joined is basically the same as other areas of the image, resulting in a better stitching effect.

[0107] The image stitching method provided in this application acquires a first image, a second image, and a calibration board image of a target object. The first image corresponds to a first portion of the target object, and the second image corresponds to a second portion of the target object, with overlapping areas between the two images. The calibration board image is the image corresponding to the calibration board used to calibrate the target object; thus, the calibration of the target object can be completed. Further, based on the calibration board image, a stitching relationship between the first and second images is determined, wherein the stitching relationship is used to indicate the stitching operation of stitching the second image onto the first image. Compared to related technologies that directly utilize this stitching relationship to stitch the first and second images, this application does not directly utilize this stitching relationship but optimizes it to obtain an optimized stitching relationship. The optimized stitching relationship has a higher degree of fit with the first image than the optimized stitching relationship with the first image, and / or, the optimized stitching relationship has a higher degree of fit with the second image than the optimized stitching relationship with the second image. Therefore, this application stitches the first and second images based on the optimized stitching relationship to obtain a stitched image. Understandably, since the calibration board and the actual object are not on the same plane, there is a certain error between the stitching relationship obtained using the calibration board image and the actual stitching relationship required. This application optimizes the original stitching relationship after determining it, and uses the optimized stitching relationship to stitch the image, achieving higher image stitching accuracy and thus improving the image stitching effect.

[0108] In some embodiments, the present application can optimize the stitching relationship at different stages. For example, optimizing the stitching relationship includes a first optimized stitching relationship and a second optimized stitching relationship. The first optimized stitching relationship is determined during the calibration stage of the target object; the second optimized stitching relationship is determined during the stitching stage of the image of the target object.

[0109] As an example, the optimization process for the stitching relationship can be carried out during the calibration stage. After the electronic device optimizes the stitching relationship during the calibration stage, each group of images to be stitched is stitched together using the optimized stitching relationship during the stitching stage.

[0110] Understandably, this method takes less time in the stitching stage and is less sensitive to the time spent optimizing the stitching relationship. However, this method requires that the plane of the target object to be stitched and the equipment used for shooting remain unchanged, and each set of stitched images must be on the same plane.

[0111] Example 2: The optimization process for splicing relationships occurs during the splicing stage. During each splicing process, the electronic device needs to optimize the splicing relationships separately.

[0112] Understandably, this solution takes longer in the splicing stage and is more sensitive to the time required for optimizing the splicing relationship. However, this solution does not require the plane of the target object to remain unchanged, meaning it is more suitable for situations where the splicing plane experiences minor vibrations.

[0113] Example 3: The solution of Example 1 can be combined with the solution of Example 2, that is, the electronic device optimizes the splicing relationship in both the calibration stage and the splicing stage.

[0114] Understandably, the advantage of this approach is that it allows for larger adjustments during the calibration phase, while smaller adjustments are made during the stitching phase. This reduces the time spent in the stitching phase and eliminates minor variations caused by the scene and equipment.

[0115] The foregoing primarily describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the aforementioned functions, it includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, in conjunction with the units and algorithm steps of the various examples described in the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0116] In an exemplary embodiment, this application also provides an image stitching device. Figure 8 This is a schematic diagram illustrating the composition of the image stitching device provided in an embodiment of this application. Figure 8 As shown, the image stitching device includes: an acquisition module 201, a determination module 202, a processing module 203, and a stitching module 204.

[0117] The acquisition module 201 is used to acquire a first image, a second image, and a calibration board image of the target object; the first image corresponds to a first part of the target object, the second image corresponds to a second part of the target object, and the first image and the second image have an overlapping area; the calibration board image is the image corresponding to the calibration board used to calibrate the target object; the determination module 202 is used to determine the stitching relationship between the first image and the second image based on the calibration board image; the stitching relationship is used to indicate the stitching operation of stitching the second image to the first image; the processing module 203 is used to optimize the stitching relationship to obtain an optimized stitching relationship; the degree of adaptation between the optimized stitching relationship and the first image is greater than the degree of adaptation between the stitching relationship and the first image, and / or, the degree of adaptation between the optimized stitching relationship and the second image is greater than the degree of adaptation between the stitching relationship and the second image; the stitching module 204 is used to stitch the first image and the second image based on the optimized stitching relationship to obtain a stitched image.

[0118] In one possible implementation, the processing module 203 is specifically used to: determine the overlapping area between the first image and the second image; optimize the stitching relationship based on the overlapping area to obtain an optimized stitching relationship; the pose adjustment of the target object in the overlapping area by the optimized stitching relationship is different from the pose adjustment of the target object in the overlapping area by the stitching relationship.

[0119] In one possible implementation, the stitching relationship includes multiple image transformation parameters; the processing module 203 is specifically used to: determine the stitching error between the first image and the second image in the overlapping area; and adjust each image transformation parameter according to the stitching error to obtain an optimized stitching relationship.

[0120] In one possible implementation, the processing module 203 is specifically used to: for any image transformation parameter, adjust the image transformation parameter by searching the neighborhood data of the image transformation parameter until the stitching error is less than or equal to a preset threshold.

[0121] In one possible implementation, the stitching relationship includes multiple image transformation parameters; the processing module is specifically used to: determine the image transformation parameters to be adjusted; the image transformation parameters to be adjusted are the image transformation parameters that undergo geometric transformation among the multiple image transformation parameters; determine the stitching error between the first image and the second image in the overlapping area; and adjust the image transformation parameters to be adjusted according to the stitching error to obtain an optimized stitching relationship.

[0122] In one possible implementation, the optimized stitching relationship includes a first optimized stitching relationship and a second optimized stitching relationship; the first optimized stitching relationship is determined during the calibration stage of the target object; the second optimized stitching relationship is determined during the stitching stage of the image of the target object.

[0123] It should be noted that, Figure 8The module division shown is illustrative and represents only one logical functional division; in actual implementation, other division methods are possible. For example, two or more functions can be integrated into a single processing module. These integrated modules can be implemented in hardware or as software functional units.

[0124] In an exemplary embodiment, this application also provides a computer-readable storage medium including software instructions that, when run on an electronic device, cause the electronic device to perform any of the methods provided in the above embodiments.

[0125] In an exemplary embodiment, this application also provides a computer program product containing computer execution instructions, which, when run on an electronic device, causes the electronic device to perform any of the methods provided in the above embodiments.

[0126] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software programs, implementation can be, in whole or in part, in the form of a computer program product. This computer program product includes one or more computer-executable instructions. When these computer-executable instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer-executable instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer-executable instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device containing one or more servers, data centers, etc., that can be integrated with the medium. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), etc.

[0127] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, disclosure, and appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple components. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0128] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made thereto without departing from the spirit and scope of this application. Accordingly, this specification and drawings are merely exemplary illustrations of this application as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from the spirit and scope of this application. Thus, if such modifications and modifications of this application fall within the scope of the claims of this application and their equivalents, this application is also intended to include such modifications and modifications.

[0129] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image stitching method, characterized in that, The method includes: A first image, a second image, and a calibration board image of the target object are acquired; the first image corresponds to a first part of the target object, the second image corresponds to a second part of the target object, and the first image and the second image have an overlapping area; the calibration board image is the image corresponding to the calibration board used to calibrate the target object. Based on the calibration board image, the stitching relationship between the first image and the second image is determined; the stitching relationship is used to indicate the stitching operation of stitching the second image to the first image; The overlapping area between the first image and the second image is determined; based on the overlapping area, the stitching relationship is optimized to obtain an optimized stitching relationship; the pose adjustment of the target object in the overlapping area by the optimized stitching relationship is different from the pose adjustment of the target object in the overlapping area by the stitching relationship. The degree of adaptation between the optimized stitching relationship and the first image is greater than the degree of adaptation between the stitching relationship and the first image; the degree of adaptation between the optimized stitching relationship and the second image is greater than the degree of adaptation between the stitching relationship and the second image. The optimized stitching relationship includes a first optimized stitching relationship and a second optimized stitching relationship; the first optimized stitching relationship is determined during the calibration stage of the target object; the second optimized stitching relationship is determined during the stitching stage of the image of the target object. Based on the optimized stitching relationship, the first image and the second image are stitched together to obtain a stitched image.

2. The method according to claim 1, characterized in that, The stitching relationship includes multiple image transformation parameters; the optimization of the stitching relationship based on the overlapping region to obtain an optimized stitching relationship includes: Determine the stitching error between the first image and the second image in the overlapping area; Based on the stitching error, the transformation parameters of each image are adjusted to obtain the optimized stitching relationship.

3. The method according to claim 2, characterized in that, The step of adjusting the transformation parameters of each image based on the stitching error to obtain the optimized stitching relationship includes: For any image transformation parameter, the image transformation parameter is adjusted by searching the neighborhood data of the image transformation parameter until the stitching error is less than or equal to a preset threshold.

4. The method according to claim 1, characterized in that, The stitching relationship includes multiple image transformation parameters; the optimization of the stitching relationship based on the overlapping region to obtain an optimized stitching relationship includes: Determine the image transformation parameters to be adjusted; the image transformation parameters to be adjusted are the image transformation parameters that undergo geometric transformations among the plurality of image transformation parameters. Determine the stitching error between the first image and the second image in the overlapping area; Based on the splicing error, the transformation parameters of the image to be adjusted are adjusted to obtain the optimized splicing relationship.

5. An image stitching device, characterized in that, The device includes an acquisition module, a determination module, a processing module, and a splicing module; The acquisition module is used to acquire a first image, a second image, and a calibration board image of the target object; the first image corresponds to a first part of the target object, the second image corresponds to a second part of the target object, and the first image and the second image have an overlapping area; the calibration board image is an image corresponding to the calibration board used to calibrate the target object. The determining module is used to determine the stitching relationship between the first image and the second image based on the calibration board image; The stitching relationship is used to indicate the stitching operation of stitching the second image to the first image; The processing module is used to determine the overlapping region between the first image and the second image; based on the overlapping region, optimize the stitching relationship to obtain an optimized stitching relationship; the pose adjustment of the target object in the overlapping region by the optimized stitching relationship is different from the pose adjustment of the target object in the overlapping region by the stitching relationship; the degree of adaptation between the optimized stitching relationship and the first image is greater than the degree of adaptation between the stitching relationship and the first image; the degree of adaptation between the optimized stitching relationship and the second image is greater than the degree of adaptation between the stitching relationship and the second image; the optimized stitching relationship includes a first optimized stitching relationship and a second optimized stitching relationship; the first optimized stitching relationship is determined during the calibration stage of the target object; the second optimized stitching relationship is determined during the image stitching stage of the target object; The stitching module is used to stitch the first image and the second image together based on the optimized stitching relationship to obtain a stitched image.

6. The apparatus according to claim 5, characterized in that, The splicing relationship includes multiple image transformation parameters; the processing module is specifically used for: Determine the stitching error between the first image and the second image in the overlapping area; For any image transformation parameter, the image transformation parameter is adjusted by searching the neighborhood data of the image transformation parameter until the stitching error is less than or equal to a preset threshold.

7. The apparatus according to claim 5, characterized in that, The splicing relationship includes multiple image transformation parameters; the processing module is specifically used for: Determine the image transformation parameters to be adjusted; the image transformation parameters to be adjusted are the image transformation parameters that undergo geometric transformations among the plurality of image transformation parameters. Determine the stitching error between the first image and the second image in the overlapping area; Based on the splicing error, the transformation parameters of the image to be adjusted are adjusted to obtain the optimized splicing relationship.

8. An electronic device, characterized in that, include: Processor and memory; The memory stores instructions that the processor can execute; When the processor is configured to execute the instructions, the electronic device performs the method as described in any one of claims 1-4.

9. A computer-readable storage medium, characterized in that, include: Software instructions; When the software instructions are executed in an electronic device, the electronic device causes the electronic device to perform the method as described in any one of claims 1-4.

Images