Image processing method, device and electronic equipment

Abstract

The present application provides an image processing method, device, and electronic device. After determining a first original image and a second original image that need to be re-stitched from a panoramic image, the present application divides the stitching area of the first original image into a specified number of image areas according to the depth information of each pixel point, and when any image area includes a target stitching object, determines the image area as the area of interest for this stitching, and determines a second area in the second original image that matches the image area, performs feature matching on the pixels in the image area and the second area, and further re-splices the first original image and the second original image based on the matching results; by determining the area where the target stitching object is located and performing matching and stitching only based on the feature points in the area, the panoramic image with poor visual effect can be re-stitched based on the area to which the target stitching object belongs, thereby improving the stitching effect.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to image processing methods, devices and electronic equipment. Background Art

[0002] In real-world scenarios, images captured by multiple image acquisition devices from different perspectives are often stitched together to create a panoramic image with a wider field of view. During this image stitching process, alignment is often performed on the stitched images to minimize image misalignment at the seams. However, even after alignment, the resulting panoramic image can still exhibit poor visual quality due to low alignment or stitching accuracy. Summary of the Invention

[0003] In view of this, the present application provides an image processing method, device and electronic device for re-stitching panoramic images with poor visual effects obtained by stitching to improve the stitching effect.

[0004] The technical solutions provided in this application are as follows:

[0005] According to an embodiment of the first aspect of the present application, there is provided an image processing method, the method comprising:

[0006] Determining, from a panoramic image stitched from a plurality of original images, a first original image and a second original image that need to be stitched again; the first original image and the second original image are misaligned in the panoramic image;

[0007] Dividing the stitching area of the first original image to obtain a specified number of image areas; wherein the stitching area of the first original image refers to an area in the first original image used for image stitching with the second original image, and the image areas are divided based on depth information of each pixel in the stitching area;

[0008] If the target stitching object is included in the first area, determining a second area in the second original image that matches the first area; the first area is one of a specified number of image areas obtained by dividing the stitching area;

[0009] Performing feature matching on pixel points in a first area in the first original image and a second area in the second original image;

[0010] The first original image and the second original image in the panoramic image are re-stitched according to the matching result.

[0011] Optionally, determining the first original image and the second original image that need to be re-stitched from the panoramic image stitched together from the plurality of original images includes:

[0012] For each seam included in the panoramic image, determining two original images corresponding to the seam;

[0013] Determining the similarity between the stitching regions of the two original images;

[0014] If the similarity between the stitching regions of the two original images is less than a first specified threshold, the two original images are determined to be a first original image and a second original image that need to be stitched again.

[0015] Optionally, dividing the spliced area of the first original image to obtain a specified number of image areas includes:

[0016] determining a depth range included in the stitched area of the first original image based on the obtained depth information of each pixel in the stitched area of the first original image; the depth information of each pixel in the stitched area of the first original image is obtained based on the depth estimation module when stitching multiple original images to obtain a panoramic image;

[0017] Dividing the depth range into the specified number of sub-depth ranges;

[0018] A sub-region corresponding to each sub-depth range in the stitching region of the first original image is determined as an image region.

[0019] Optionally, determining a second region in the second original image that matches the first region includes:

[0020] determining, based on each first feature point included in the first area of the first original image, a second feature point in the second original image that matches the first feature point; wherein the matching relationship between the first feature point in the first original image and the second feature point in the second original image refers to the matching relationship between the first feature point in the first original image and the second feature point in the second original image included in the panoramic image;

[0021] The region where all the second feature points are located is determined as a second region in the second original image that matches the first region.

[0022] Optionally, before determining the second region in the second original image that matches the first region, the method further includes:

[0023] If the area ratio of the target stitching object in the first area is less than the second specified threshold, the specified number is updated, and the process returns to the step of dividing the stitching area of the first original image to obtain the specified number of image areas, until the area ratio of the target stitching object in the first area is not less than the second specified threshold; wherein the updated specified number is greater than the specified number before the update.

[0024] Optionally, after performing feature matching on the pixels in the first area in the first original image and the second area in the second original image, the method further includes:

[0025] For each matched feature point pair in the first area and the second area, if the error value between the two feature points in the feature point pair is greater than a third specified threshold, the position of the feature point pair is adjusted so that the error value between the two feature points in the feature point pair is not greater than the third specified threshold.

[0026] Optionally, after performing feature matching on the pixels in the first area in the first original image and the second area in the second original image, the method further includes:

[0027] At least one pair of matched feature points is added to the first area and the second area; the positions of the added feature point pairs are different from the positions of the matched feature point pairs in the first area and the second area.

[0028] According to an embodiment of the second aspect of the present application, there is provided an image processing device, the device comprising:

[0029] A first determining unit is configured to determine, from a panoramic image stitched from a plurality of original images, a first original image and a second original image that need to be stitched together again; the first original image and the second original image are misaligned in the panoramic image;

[0030] a dividing unit, configured to divide the stitching area of the first original image into a specified number of image areas; wherein the stitching area of the first original image refers to an area in the first original image used for image stitching with the second original image, and the image areas are divided based on depth information of each pixel in the stitching area;

[0031] a second determining unit, configured to determine, if the target stitching object is included in the first area, a second area in the second original image that matches the first area; the first area being one of a specified number of image areas obtained by dividing the stitching area;

[0032] The stitching unit is configured to perform feature matching on pixel points in a first area of the first original image and a second area of the second original image; and re-stitch the first original image and the second original image in the panoramic image based on the matching result.

[0033] Optionally, the first determining unit is specifically configured to:

[0034] For each seam included in the panoramic image, determining two original images corresponding to the seam;

[0035] Determining the similarity between the stitching regions of the two original images;

[0036] If the similarity between the stitching areas of the two original images is less than a first specified threshold, determining that the two original images are the first original image and the second original image that need to be re-stitched;

[0037] And / or, the dividing unit is specifically used for:

[0038] determining a depth range included in the stitched area of the first original image based on the obtained depth information of each pixel in the stitched area of the first original image; the depth information of each pixel in the stitched area of the first original image is obtained based on the depth estimation module when stitching multiple original images to obtain a panoramic image;

[0039] Dividing the depth range into the specified number of sub-depth ranges;

[0040] determining a sub-region corresponding to each sub-depth range in the stitching region of the first original image as an image region;

[0041] And / or, the second determining unit is specifically configured to:

[0042] determining, based on each first feature point included in the first area of the first original image, a second feature point in the second original image that matches the first feature point; wherein the matching relationship between the first feature point in the first original image and the second feature point in the second original image refers to the matching relationship between the first feature point in the first original image and the second feature point in the second original image included in the panoramic image;

[0043] determining a region where all second feature points are located as a second region in the second original image that matches the first region;

[0044] And / or, before determining the second region matching the first region in the second original image, the dividing unit is further configured to:

[0045] If the area ratio of the target stitching object in the first region is less than a second specified threshold, the specified number is updated, and the process returns to the step of dividing the stitching region of the first original image to obtain a specified number of image regions, until the area ratio of the target stitching object in the first region is no less than the second specified threshold; wherein the updated specified number is greater than the specified number before the update;

[0046] And / or, after performing feature matching on the pixels in the first area in the first original image and the second area in the second original image, the stitching unit is further configured to:

[0047] For each matched feature point pair in the first area and the second area, if the error value between two feature points in the feature point pair is greater than a third specified threshold, adjust the position of the feature point pair so that the error value between the two feature points in the feature point pair is not greater than the third specified threshold;

[0048] And / or, after performing feature matching on the pixels in the first area in the first original image and the second area in the second original image, the stitching unit is further configured to:

[0049] At least one pair of matched feature points is added to the first area and the second area; the positions of the added feature point pairs are different from the positions of the matched feature point pairs in the first area and the second area.

[0050] According to an embodiment of the third aspect of the present application, an electronic device is provided, comprising: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the method described in the first aspect.

[0051] It can be seen from the above technical solution that after determining the first original image and the second original image that need to be re-stitched from the panoramic image, the present application divides the stitching area of the first original image into a specified number of image areas according to the depth information of each pixel point, and when any image area includes the target stitching object, the image area is determined as the area of focus for this stitching, and a second area in the second original image that matches the image area is determined, feature matching is performed on the pixel points in the image area and the second area, and the first original image and the second original image are further re-stitched based on the matching results; by determining the area where the target stitching object is located and matching and stitching only based on the feature points in the area, the panoramic image with poor visual effect is re-stitched based on the area to which the target stitching object belongs, thereby improving the stitching effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0053] Figure 1 A flowchart of the image processing method provided in an embodiment of the present application;

[0054] Figure 2 A schematic diagram of an image processing system provided in an embodiment of the present application;

[0055] Figure 3 A schematic diagram of the execution flow of the panoramic stitching module provided in an embodiment of the present application;

[0056] Figure 4 A schematic diagram of the execution flow of the panoramic preview module provided in an embodiment of the present application;

[0057] Figure 5 A schematic diagram of the execution flow of the interactive optimization module provided in an embodiment of the present application;

[0058] Figure 6A This is an example of an original image in an indoor track scene provided in an embodiment of the present application;

[0059] Figure 6B This is an example of an original panoramic image in an indoor track scene provided by an embodiment of the present application;

[0060] Figure 6C This is an example of a panoramic image of a target in an indoor track scene provided by an embodiment of the present application;

[0061] Figure 7A This is an example of an original image in an outdoor railway scene provided in an embodiment of the present application;

[0062] Figure 7B This is an example of an original panoramic image of an outdoor railway scene provided in an embodiment of the present application;

[0063] Figure 7C This is an example of a panoramic image of a target in an outdoor railway scene provided by an embodiment of the present application;

[0064] Figure 8 A structural diagram of an image processing device provided in an embodiment of the present application;

[0065] Figure 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0066] In order to enable those skilled in the art to better understand the technical solutions provided by the embodiments of the present application, and to make the above-mentioned purposes, features and advantages of the embodiments of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application are further described in detail below with reference to the accompanying drawings.

[0067] In some practical application fields, such as security monitoring, in order to obtain a monitoring range with a larger viewing angle, it is usually possible to capture images through multiple image acquisition devices with different viewing angles, and stitch the images captured by each image acquisition device to obtain a panoramic image with a wider viewing angle.

[0068] When stitching images to be stitched that are captured by multiple image acquisition devices with different perspectives, it is necessary to find the overlapping areas between the images to be stitched, and to completely align the overlapping areas between the images to be stitched through image registration, and then further fuse the multiple images to be stitched to obtain a panoramic image.

[0069] However, due to the inaccurate image registration results or the presence of multiple depth planes in the captured image, misalignment may occur in the panorama after image stitching, resulting in an inconsistent visual effect of the panorama.

[0070] Based on this, the present application provides an image processing method to re-stitch panoramic images with poor visual effects to improve the image stitching effect.

[0071] Please refer to Figure 1 , Figure 1 This is a flowchart of the image processing method provided in an embodiment of the present application.

[0072] As an embodiment, the method can be applied to a server in an image processing system.

[0073] like Figure 1 As shown, the method may include the following steps:

[0074] Step 101 : Determine a first original image and a second original image that need to be re-stitched from a panoramic image stitched together from a plurality of original images.

[0075] In this embodiment, the original image where the stitching misalignment occurs can be determined from the panoramic image. Since each stitching seam corresponds to two original images, the two original images where the stitching misalignment occurs can be recorded as the first original image and the second original image.

[0076] Among them, the first original image and the second original image are only used to describe the original images on both sides of the seam where the splicing misalignment occurs. For example, the original image on the left side of the seam can be recorded as the first original image, and the original image on the right side of the seam can be recorded as the second original image. Alternatively, the original image on the left side of the seam can be recorded as the second original image, and the original image on the right side of the seam can be recorded as the first original image. This application does not impose any restrictions on this.

[0077] In this embodiment, the original panoramic image may include one or more seams. To determine the first original image and the second original image that need to be re-stitched, it is first necessary to determine which seam is the seam with seam misalignment.

[0078] Specifically, the method for determining the occurrence of stitching misalignment can be to observe the position where the image stitching misalignment occurs in the panoramic image, and manually select the original image corresponding to the stitching misalignment from the original panoramic image; or it can be to automatically detect and select the original image corresponding to the stitching misalignment from each stitching of the server, and this application does not limit this.

[0079] As an embodiment, a method for determining a first original image and a second original image that need to be re-stitched from a panoramic image stitched from multiple original images may include:

[0080] For each seam included in the panoramic image, determining two original images corresponding to the seam;

[0081] Determining the similarity between the stitching regions of the two original images;

[0082] If the similarity between the stitching regions of the two original images is less than a first specified threshold, the two original images are determined to be a first original image and a second original image that need to be stitched again.

[0083] In this embodiment, for each seam included in the panoramic image, the two original images corresponding to the seam can be determined, and the similarity of the stitching area (i.e., the overlapping area) of the two original images can be further calculated, such as calculating the mean squared error (MSE) or structural similarity index (SSIM) of the stitching area of the two original images, to determine the similarity of the stitching area of the two original images. This application does not limit the specific method for calculating the similarity.

[0084] Furthermore, if it is determined that the similarity between the stitching areas of the two original images is less than the first specified threshold, it indicates that the difference between the stitching areas of the two original images is large and the stitching effect is poor. At this time, it can be determined that the two original images are the first original image and the second original image that need to be re-stitched.

[0085] In this embodiment, the server may be connected to a display device to display the original panoramic image and the image stitching misaligned area on the display device for viewing by a user.

[0086] Illustratively, in this embodiment, after obtaining the original panoramic image, each seam included in the original panoramic image can be detected to determine whether there is a seam misalignment at the seam. If it is determined that a seam occurs at the seam, the stitching area of the two original images at the seam can be marked in the original panoramic image, for example, by marking the stitching area of the two original images in the form of an external rectangular frame.

[0087] At this point, the description of step 101 ends, and step 102 is executed next.

[0088] Step 102: Divide the spliced area of the first original image to obtain a specified number of image areas.

[0089] The stitching area of the first original image refers to an area in the first original image used for image stitching with the second original image, and the image area is divided based on depth information of each pixel in the stitching area.

[0090] In this embodiment, the first original image and the second original image may carry feature points included in the image and attribute information of each feature point.

[0091] As an embodiment, the attribute information of the feature points may include depth information of each feature point. The depth information may be obtained based on a deployed depth estimation module when multiple original images are stitched together to obtain a panoramic image.

[0092] Exemplarily, a specific method of dividing the spliced area of the first original image to obtain a specified number of image areas may include:

[0093] determining a depth range included in the stitched area of the first original image based on the obtained depth information of each pixel in the stitched area of the first original image; the depth information of each pixel in the stitched area of the first original image is obtained based on the depth estimation module when stitching multiple original images to obtain a panoramic image;

[0094] Dividing the depth range into the specified number of sub-depth ranges;

[0095] A sub-region corresponding to each sub-depth range in the stitching region of the first original image is determined as an image region.

[0096] In this embodiment, the depth range of the stitched area of the first original image can be determined based on the depth information of each pixel in the stitched area of the first original image. For example, if the depth information of the pixels in the stitched area of the first original image is a minimum of 0 and a maximum of 60, the depth range of the stitched area of the first original image is determined to be [0, 60].

[0097] Furthermore, the depth range can be divided into a specified number of sub-depth ranges. Specifically, the specified number can be a pre-set number, such as 2. In this case, the depth range can be divided into 2 sub-depth ranges. For example, the depth range can be evenly divided into 2 sub-ranges according to the size of the depth range, that is, divided into 2 sub-ranges of [0, 30] and [31, 60]. For another example, the depth range can be divided into 2 sub-ranges according to the depth information of the pixel points. For example, the depth information corresponding to 50% of the pixels is divided into one sub-range, and the depth information corresponding to the other 50% of the pixels is divided into another sub-range. For example, the depth information corresponding to 50% of the pixels is within the range of [0, 40], and the depth information corresponding to the other 50% of the pixels is within the range of [41, 60]. In this case, the depth range [0, 60] can be divided into 2 sub-ranges of [0, 40] and [41, 60]. This application does not limit the method of dividing the sub-ranges.

[0098] After the sub-depth ranges are divided, a sub-region corresponding to each sub-depth range in the stitching area of the first original image may be determined as an image region, and a specified number of image regions may be determined.

[0099] At this point, the description of step 102 ends, and step 103 is executed next.

[0100] Step 103: If the target stitching object is included in the first area, determine a second area in the second original image that matches the first area.

[0101] The first area is one of a specified number of image areas obtained by dividing the stitching area.

[0102] In this embodiment, the target stitching object refers to the stitching object of interest in this stitching. For example, in a scenario where the behavior of pedestrians in an image is monitored, it is only necessary to focus on the area where the pedestrians are located in order to achieve a better stitching effect. In this case, the target stitching object is the pedestrian. For another example, in a scenario where vehicles in an image are monitored, it is only necessary to focus on the area where the vehicles are located in order to achieve a better stitching effect. In this case, the target stitching object is the vehicle.

[0103] For example, if it is determined that any area among the specified number of image areas determined in step 102 completely includes the target stitching object, then the area is recorded as the first area, and the first area is determined to be the area that needs to be paid attention to in this stitching, and the depth range to which the first area belongs is the stitching depth range that is concerned in this stitching.

[0104] Furthermore, a second region matching the first region may be determined in the second original image, and a region of the two images that needs to be focused on in the current stitching may be found.

[0105] Specifically, the method of determining the second region in the second original image that matches the first region may include:

[0106] determining, based on each first feature point included in the first area of the first original image, a second feature point in the second original image that matches the first feature point; wherein the matching relationship between the first feature point in the first original image and the second feature point in the second original image refers to the matching relationship between the first feature point in the first original image and the second feature point in the second original image included in the panoramic image;

[0107] The region where all the second feature points are located is determined as a second region in the second original image that matches the first region.

[0108] In this embodiment, for each feature point included in the first area of the first original image (referred to as a first feature point), the second feature point that matches each first feature point in the first area in the second original image can be found based on the matching relationship between the first feature point in the first original image and the second feature point in the second original image recorded when the panoramic image was previously stitched.

[0109] Furthermore, the region where all the second feature points are located may be determined as the second region in the second original image that matches the first region.

[0110] At this point, the description of step 103 ends, and step 104 is executed next.

[0111] Step 104: Perform feature matching on the pixels in the first area of the first original image and the second area of the second original image.

[0112] In this embodiment, feature matching can be performed on the pixel points included in the first area and the second area obtained in step 103, that is, matching is performed only based on the pixel points in the first area and the second area in the spliced area, and the pixel points in other areas do not participate in the matching process.

[0113] At this point, the description of step 104 ends, and step 105 is executed next.

[0114] Step 105 : Rejoin the first original image and the second original image in the panoramic image according to the matching result.

[0115] In this embodiment, the first original image and the second original image can be re-stitched based on the matching results of the first area and the second area in step 104. Since this stitching is performed on the area where the target stitching object is located, a better stitching effect can be obtained.

[0116] As an embodiment, after performing feature matching on the pixels in the first area in the first original image and the second area in the second original image, the method proposed in this embodiment may further include:

[0117] For each matched feature point pair in the first area and the second area, if the error value between the two feature points in the feature point pair is greater than a third specified threshold, the position of the feature point pair is adjusted so that the error value between the two feature points in the feature point pair is not greater than the third specified threshold.

[0118] In this embodiment, the attribute information of each feature point may also include an error value for the matched feature point pairs. After completing the matching of pixels in the first and second regions, feature point pairs in the matched feature point pairs whose error value exceeds a third specified threshold may be adjusted so that the error value between the two feature points in the adjusted feature point pairs does not exceed the third specified threshold.

[0119] Alternatively, in subsequent stitching, feature point pairs with error values greater than the third specified threshold may be ignored, and only feature point pairs with error values not greater than the third specified threshold may be used for stitching to reduce errors.

[0120] As an embodiment, after performing feature matching on the pixels in the first area in the first original image and the second area in the second original image, the method proposed in this embodiment may further include:

[0121] At least one pair of matched feature points is added to the first area and the second area; the positions of the added feature point pairs are different from the positions of the matched feature point pairs in the first area and the second area.

[0122] In this embodiment, after completing the matching of pixel points between the first area and the second area, at least one pair of matched feature points can be added to the first area and the second area according to actual needs, so that more useful features can be involved in the stitching process, thereby improving the visual effect of the stitched image.

[0123] This concludes the description of step 105.

[0124] In this embodiment, before determining the second region in the second original image that matches the first region, the method may further include:

[0125] If the area ratio of the target stitching object in the first area is less than the second specified threshold, the specified number is updated, and the process returns to the step of dividing the stitching area of the first original image to obtain the specified number of image areas, until the area ratio of the target stitching object in the first area is not less than the second specified threshold; wherein the updated specified number is greater than the specified number before the update.

[0126] In this embodiment, if the area ratio of the target stitching object in the first area is less than the second specified threshold, it indicates that the target stitching object accounts for too small a proportion in the divided first area. Direct stitching may result in matching and stitching a large number of objects that do not need to be paid attention to, while resulting in poor stitching effect of the target stitching objects that need to be paid attention to.

[0127] At this time, it can be considered that the current division of the image area is not appropriate enough, and the specified number can be updated, such as adding a specified value (such as 1) on the basis of the original specified number, and returning to the step of dividing the stitching area of the first original image to obtain a specified number of image areas, until the area proportion of the target stitching object in the first area is not less than the second specified threshold, indicating that the target stitching object now accounts for a large proportion in the first area obtained by division, and the stitching effect of matching stitching for this area is better.

[0128] In this embodiment, if it is found that the target stitching object is not completely contained in any image area, for example, the target stitching object spans multiple image areas, it indicates that the division of the image area may be too detailed and too many areas are divided. At this time, the specified number can be updated, such as reducing the specified value (for example, 1) based on the original specified number, and returning to the step of dividing the stitching area of the first original image to obtain the specified number of image areas until the target stitching object is contained in the first area.

[0129] In this embodiment, the feature points carried by the first original image and the second original image and the attribute information of each feature point are recorded during the process of generating the panoramic image.

[0130] As an example, the panoramic image can be obtained by the following method:

[0131] Obtain K original images, where K is greater than or equal to 2;

[0132] Extract feature points from the K original images, match the same feature points in the original images with overlapping areas, and record the attribute information corresponding to each feature point;

[0133] The K original images are fused according to the matched feature points included in the K original images to obtain the panoramic image.

[0134] In this embodiment, the panoramic image can be obtained by stitching together multiple original images captured by multiple image capture devices with different viewing angles.

[0135] Specifically, feature extraction can be performed on multiple original images to obtain multiple feature points corresponding to each original image. The same feature points in the original images with overlapping areas can be further matched, and image alignment can be performed on the original images with overlapping areas, while recording the attribute information corresponding to each feature point.

[0136] Furthermore, the K original images may be fused according to the matched feature points included in the K original images to obtain an original panoramic image.

[0137] In this embodiment, after re-stitching the first and second original images in the panoramic image based on the matching results to obtain a new panoramic image (referred to as the target panoramic image), the target panoramic image can be further checked for any misaligned seams. If any still exist, this indicates that the current target panoramic image still suffers from a visually inconsistent effect. At this point, the process can return to the step of determining the first and second original images that need to be re-stitched from the panoramic image stitched from the multiple original images until the target panoramic image no longer has any misaligned seams or the target panoramic image meets the user's desired visual effect.

[0138] This concludes Figure 1 Description.

[0139] After determining the first original image and the second original image that need to be re-stitched from the panoramic image, the present application divides the stitching area of the first original image into a specified number of image areas according to the depth information of each pixel point, and when any image area includes the target stitching object, determines the image area as the area of focus for this stitching, and determines a second area in the second original image that matches the image area, performs feature matching on the pixel points in the image area and the second area, and further re-stitches the first original image and the second original image based on the matching results; by determining the area where the target stitching object is located and matching and stitching only based on the feature points in the area, the panoramic image with poor visual effect can be re-stitched based on the area to which the target stitching object belongs, thereby improving the stitching effect.

[0140] Please refer to Figure 2 , Figure 2 A schematic diagram of a display system provided in an embodiment of the present application.

[0141] like Figure 2 As shown, the image processing method proposed in this application can be applied to an image processing system.

[0142] The image processing system may include:

[0143] Panoramic stitching module, panoramic preview module and interactive optimization module.

[0144] Among them, the panoramic stitching module is used to perform image registration, fusion and other stitching processes on the original images to obtain a panoramic image; the panoramic preview module is used to display the panoramic image obtained by the panoramic stitching module and detect the misalignment of each seam in the panoramic image; the interactive optimization module is used to adjust and re-stitch the feature points in the original images on both sides of the misaligned seam according to the misaligned seam determined by the user in the panoramic image, so as to specifically reduce the occurrence of misalignment.

[0145] The following combination Figures 3 to 5 Introduce the above modules.

[0146] Please refer to Figure 3 , Figure 3 A schematic diagram of the execution flow of the panoramic stitching module provided in an embodiment of the present application.

[0147] like Figure 3 As shown, the panoramic stitching module is used to:

[0148] Acquire multiple original images. The multiple original images may be acquired by an image acquisition device included in the image processing system or may be input externally. This application does not impose any restrictions on this.

[0149] Image registration is performed on each original image. The image registration process includes processes such as feature point matching, homography estimation, and projective transformation. During the image registration process, attribute information of each feature point can be obtained. The attribute information can include the error value of each pair of matched feature points and the depth information of each feature point. The error value of each pair of matched feature points can be obtained during the feature point matching, homography estimation, and projective transformation process. The depth information of each feature point can be obtained using a depth estimation module. This depth estimation module can be a reused depth estimation module already existing in the current image processing system or a newly added module in the system. This application does not limit this.

[0150] The original images are fused, for example, to generate a panoramic image by weighted averaging, multi-resolution fusion, or priority region fusion, etc., which is not limited in this application.

[0151] Please refer to Figure 4 , Figure 4 A schematic diagram of the execution flow of the panoramic preview module provided in an embodiment of the present application.

[0152] like Figure 4 As shown, the panoramic preview module is used to:

[0153] Get the stitching result of the panoramic stitching module and display the panoramic image.

[0154] Detect the misalignment at the seam in the panorama and display it overlaid on the panorama (for example, display a rectangular box circumscribing the stitched area of the two original images corresponding to the misaligned seam). The method for detecting misalignment at the seam in the panorama has been described in detail above and will not be repeated here.

[0155] Please refer to Figure 5 , Figure 5 A schematic diagram of the execution flow of the interactive optimization module provided in an embodiment of the present application.

[0156] like Figure 5 As shown, the interactive optimization module is used to:

[0157] Selecting a misaligned seam. In this embodiment, the user can click on a misaligned seam in the panoramic preview module (every two overlapping images will have a seam, and the misaligned seam may be detected by the algorithm or may be caused by the user's perceived visual quality). This will reveal the two original images corresponding to the seam (referred to as the first original image and the second original image).

[0158] The spliced area of the first original image is divided into a specified number of image areas, for example, the spliced area of the first original image is divided into three image areas.

[0159] Determine whether the target stitching object is included in the first region. If not, it indicates that the current division of the stitching region of the first original image is inappropriate and too detailed. In this case, you can reduce the value of the specified number and return to the step of dividing the stitching region of the first original image to obtain the specified number of image regions. For example, you can redivide the stitching region of the first original image into two image regions.

[0160] If the target stitching object is included in the first region, it can be further determined whether the area ratio of the target stitching object in the first region is less than a second specified threshold. If so, it indicates that the current stitching region division of the first original image is inappropriate, and the target stitching object occupies too small a proportion of the region, making stitching unfavorable. In this case, the specified value can be increased, and the process returns to the step of dividing the stitching region of the first original image to obtain the specified number of image regions. For example, the stitching region of the first original image can be re-divided into five image regions.

[0161] If the area proportion of the target stitching object in the first region is not less than the second specified threshold, it indicates that the current stitching region division of the first original image is appropriate. At this time, a second region matching the first region in the second original image can be determined.

[0162] Furthermore, feature matching can be performed on pixels in the first area of the first original image and the second area of the second original image. Since the two original images display the matched feature points stored in the panoramic stitching module (including the error value of each pair of matching points, where the larger the error value, the greater the possibility of inaccurate matching points) and the estimated depth information, the user can adjust the feature points that the algorithm has matched. For example, the user can ignore the feature points in the stitching area other than the feature points in the first and second areas, because these areas are not the areas of interest for this stitching. For example, the user can focus on matching points with large error values. If an incorrectly matched feature point is found, the feature point can be ignored during stitching, or the feature point can be adjusted to reduce the error value of the adjusted feature point. In areas where misalignment occurs or on targets of obvious interest, the feature points can be manually supplemented with reference to the depth information so that the feature points are on the same depth range plane, which can better reduce the misalignment of the plane.

[0163] After adjusting the matched feature points, the first original image and the second original image may be rejoined according to the matching result between the first region and the second region.

[0164] In this embodiment, after the re-stitching is completed, if it is found that there is still a seam that does not meet the stitching requirements, the process can be continued back to the process where the user clicks the misalignment at the seam in the panoramic preview module. If the stitching visual requirements are met, the interactive optimization module is exited and the panoramic image is output.

[0165] The image processing method proposed in this application is described below through two specific embodiments.

[0166] Example 1

[0167] Please refer to Figures 6A to 6C , Figure 6A This is an example of the original image to be stitched in the indoor track scene provided in the embodiment of the present application. Figure 6B This is an example of an original panoramic image in an indoor track scene provided in an embodiment of the present application. Figure 6C This is an example of a panoramic image of a target in an indoor track scene provided in an embodiment of the present application.

[0168] like Figure 6AAs shown in the figure, indoor scene stitching usually uses a wall-mounted camera setup to shoot downward, so the shooting angle is not perpendicular to the ground and the shooting distance is relatively close. The captured image will obviously have multiple depth planes and large parallax.

[0169] like Figure 6B As shown, due to the presence of multiple planes, when these two images are directly stitched together in the panoramic stitching module, the feature matching points are usually fitted on the wall farthest from the camera. In this way, the stitched panorama is obtained with little misalignment of the wall and obvious misalignment of the track. Please refer to Figure 6B The stitched image in the red frame. Since the user is mainly concerned with the middle track, the misalignment of the track needs to be as small as possible. Figure 6B The panorama shown does not meet the requirements.

[0170] like Figure 6C As shown, after the panoramic preview module detects and displays the misalignment at the seam, click on the misalignment at the track in the panoramic preview module of the display system, and then enter the interactive optimization module to display the original image to be stitched. In this embodiment, multiple planes can be divided according to the depth information of the feature points. The user can select the track plane to retain the feature points on the plane and add more feature points to the track plane. Enter the panoramic stitching module again, generate a panoramic image and preview it, and you can see Figure 6C There is no obvious misalignment at the center track.

[0171] In this embodiment, after selecting the track plane, in addition to points with large error values, points not on the plane (such as feature points on the plane where the wall is located) can also be ignored, because these points are correct invalid points. It is precisely because a large number of invalid points are used during matching that poor splicing results are caused.

[0172] In this embodiment, there are multiple methods for detecting misalignment at the seam. This embodiment calculates the structural similarity (SSIM) of the overlapping area of the two images, which is more consistent with the intuitive perception of the human eye. When the SSIM is less than a set threshold, it is considered that there is obvious misalignment, and the panoramic preview module will display the area. This application does not limit this.

[0173] This concludes the description of Example 1.

[0174] Example 2

[0175] Please refer to 7A to 7C , Figure 7A This is an example of the original image to be spliced in the outdoor railway scene provided in the embodiment of the present application. Figure 7B This is an example of an original panoramic image of an outdoor railway scene provided in an embodiment of the present application. Figure 7C This is an example of a panoramic image of a target in an outdoor railway scene provided in an embodiment of the present application.

[0176] like Figure 7A As shown in the figure, the camera setup for the outdoor railway stitching is consistent with the above problem. However, the more serious problem with railway stitching is that there are a large number of repeated textures in the scene. The area in the green box in the figure is the stitching area corresponding to the two images.

[0177] like Figure 7B As shown in the figure, due to the repeated texture, there will be many mismatches after the algorithm automatically matches, and the registration accuracy is very low, so there is obvious misalignment in the panorama itself ( Figure 7B middle red frame area).

[0178] like Figure 7C As shown in the figure, after clicking on the misalignment to enter the interactive optimization module, first ignore the mismatched points that have been matched by the algorithm (the error values of these points will be large), and then add the correct matching points.

[0179] For example, a large number of erroneous points in the algorithm matching can be ignored, and feature points can be added on the plane that needs to be paid attention to, such as the depth information of the feature points on the railway track. For the plane where the slope behind the image is located, since it is not a plane that needs attention, there is no need to add feature points to this plane. The feature points of planes other than the planes that need attention, such as the railway track, can be ignored, and these points will not be used for feature matching. After completing the addition of feature points, you can enter the panoramic stitching module again, generate a panoramic image and preview it, and you can see that the track can correspond correctly.

[0180] In this example, due to the specificity of the scene, the presence of a large number of repeated textures resulted in a large number of mismatched points in the algorithm, resulting in low image registration accuracy and severe misalignment. By ignoring mismatched points and feature points on non-interested planes and supplementing them with correct matching points, the interactive optimization module successfully improved image registration accuracy and reduced panorama misalignment.

[0181] This concludes the description of Example 2.

[0182] Please refer to Figure 8 , Figure 8 This is a structural diagram of an image processing device proposed in an embodiment of the present application. Figure 8 As shown, the apparatus may include a first determining unit 801, a dividing unit 802, a second determining unit 803, and a splicing unit 804. Specifically, the apparatus includes:

[0183] The first determining unit 801 is configured to determine, from a panoramic image stitched from a plurality of original images, a first original image and a second original image that need to be stitched together again; the first original image and the second original image are misaligned in the panoramic image;

[0184] a dividing unit 802 configured to divide the stitching area of the first original image into a specified number of image areas; wherein the stitching area of the first original image refers to an area of the first original image used for image stitching with the second original image, and the image areas are divided based on depth information of each pixel in the stitching area;

[0185] A second determining unit 803 is configured to determine a second region in the second original image that matches the first region if the target stitching object is included in the first region; the first region is one of a specified number of image regions obtained by dividing the stitching region;

[0186] The stitching unit 804 is configured to perform feature matching on pixel points in the first area of the first original image and the second area of the second original image; and re-stitch the first original image and the second original image in the panoramic image based on the matching result.

[0187] Optionally, the first determining unit 801 is specifically configured to:

[0188] For each seam included in the panoramic image, determining two original images corresponding to the seam;

[0189] Determining the similarity between the stitching regions of the two original images;

[0190] If the similarity between the stitching areas of the two original images is less than a first specified threshold, determining that the two original images are the first original image and the second original image that need to be re-stitched;

[0191] And / or, the dividing unit 802 is specifically configured to:

[0192] determining a depth range included in the stitched area of the first original image based on the obtained depth information of each pixel in the stitched area of the first original image; the depth information of each pixel in the stitched area of the first original image is obtained based on the depth estimation module when stitching multiple original images to obtain a panoramic image;

[0193] Dividing the depth range into the specified number of sub-depth ranges;

[0194] determining a sub-region corresponding to each sub-depth range in the stitching region of the first original image as an image region;

[0195] And / or, the second determining unit 803 is specifically configured to:

[0196] determining, based on each first feature point included in the first area of the first original image, a second feature point in the second original image that matches the first feature point; wherein the matching relationship between the first feature point in the first original image and the second feature point in the second original image refers to the matching relationship between the first feature point in the first original image and the second feature point in the second original image included in the panoramic image;

[0197] determining a region where all second feature points are located as a second region in the second original image that matches the first region;

[0198] And / or, before determining the second region matching the first region in the second original image, the dividing unit 802 is further configured to:

[0199] If the area ratio of the target stitching object in the first region is less than a second specified threshold, the specified number is updated, and the process returns to the step of dividing the stitching region of the first original image to obtain a specified number of image regions, until the area ratio of the target stitching object in the first region is no less than the second specified threshold; wherein the updated specified number is greater than the specified number before the update;

[0200] And / or, after performing feature matching on the pixels in the first area in the first original image and the second area in the second original image, the stitching unit 804 is further configured to:

[0201] For each matched feature point pair in the first area and the second area, if the error value between two feature points in the feature point pair is greater than a third specified threshold, adjust the position of the feature point pair so that the error value between the two feature points in the feature point pair is not greater than the third specified threshold;

[0202] And / or, after performing feature matching on the pixels in the first area in the first original image and the second area in the second original image, the stitching unit 804 is further configured to:

[0203] At least one pair of matched feature points is added to the first area and the second area; the positions of the added feature point pairs are different from the positions of the matched feature point pairs in the first area and the second area.

[0204] So far, completed Figure 8 Description of the image processing device.

[0205] The present application also provides Figure 8 The hardware structure of the device is described in the following figure. Figure 9 The structure of the electronic device shown. Figure 9 , Figure 9 This is a structural diagram of an electronic device provided in an embodiment of the present application. Figure 9 As shown, the hardware structure may include: a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the method disclosed in the above example of this application.

[0206] Based on the same application concept as the above method, an embodiment of the present application also provides a machine-readable storage medium, on which a number of computer instructions are stored. When the computer instructions are executed by a processor, the method disclosed in the above example of the present application can be implemented.

[0207] Exemplarily, the machine-readable storage medium may be any electronic, magnetic, optical, or other physical storage device that may contain or store information, such as executable instructions, data, and the like. For example, the machine-readable storage medium may be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, a storage drive (such as a hard disk drive), a solid-state drive, any type of storage disk (such as a CD, DVD, etc.), or similar storage media, or a combination thereof.

[0208] The above are merely embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. An image processing method, characterized in that: The method includes: Determining, from a panoramic image stitched from a plurality of original images, a first original image and a second original image that need to be stitched again; the first original image and the second original image are misaligned in the panoramic image; Dividing the stitching area of the first original image to obtain a specified number of image areas; wherein the stitching area of the first original image refers to an area in the first original image used for image stitching with the second original image, and the image areas are divided based on depth information of each pixel in the stitching area; If the target stitching object is included in the first area, determining a second area in the second original image that matches the first area; the first area is one of a specified number of image areas obtained by dividing the stitching area; and the area ratio of the target stitching object in the first area is not less than a second specified threshold; Performing feature matching on pixel points in a first area in the first original image and a second area in the second original image; The first original image and the second original image in the panoramic image are re-stitched according to the matching result.

2. The method according to claim 1, characterized in that The step of determining a first original image and a second original image that need to be re-stitched from a panoramic image stitched from a plurality of original images includes: For each seam included in the panoramic image, determining two original images corresponding to the seam; Determining the similarity between the stitching regions of the two original images; If the similarity between the stitching regions of the two original images is less than a first specified threshold, the two original images are determined to be a first original image and a second original image that need to be stitched again.

3. The method according to claim 1, characterized in that The spliced area of the first original image is divided to obtain a specified number of image areas, including: determining a depth range included in the stitched area of the first original image based on the obtained depth information of each pixel in the stitched area of the first original image; the depth information of each pixel in the stitched area of the first original image is obtained based on the depth estimation module when stitching multiple original images to obtain a panoramic image; Dividing the depth range into the specified number of sub-depth ranges; A sub-region corresponding to each sub-depth range in the stitching region of the first original image is determined as an image region.

4. The method according to claim 1, wherein The determining a second region in the second original image that matches the first region includes: determining, based on each first feature point included in the first area of the first original image, a second feature point in the second original image that matches the first feature point; wherein the matching relationship between the first feature point in the first original image and the second feature point in the second original image refers to the matching relationship between the first feature point in the first original image and the second feature point in the second original image included in the panoramic image; The region where all the second feature points are located is determined as a second region in the second original image that matches the first region.

5. The method according to claim 1, wherein Before determining a second region in the second original image that matches the first region, the method further includes: If the area ratio of the target stitching object in the first area is less than the second specified threshold, the specified number is updated, and the process returns to the step of dividing the stitching area of the first original image to obtain the specified number of image areas, until the area ratio of the target stitching object in the first area is not less than the second specified threshold; wherein the updated specified number is greater than the specified number before the update.

6. The method according to claim 1, characterized in that After performing feature matching on the pixels in the first area of the first original image and the second area of the second original image, the method further includes: For each matched feature point pair in the first area and the second area, if the error value between the two feature points in the feature point pair is greater than a third specified threshold, the position of the feature point pair is adjusted so that the error value between the two feature points in the feature point pair is not greater than the third specified threshold.

7. The method according to claim 1, characterized in that After performing feature matching on the pixels in the first area of the first original image and the second area of the second original image, the method further includes: At least one pair of matched feature points is added to the first area and the second area; the positions of the added feature point pairs are different from the positions of the matched feature point pairs in the first area and the second area.

8. An image processing device, characterized in that: The device includes: A first determining unit is configured to determine, from a panoramic image stitched from a plurality of original images, a first original image and a second original image that need to be stitched together again; the first original image and the second original image are misaligned in the panoramic image; a dividing unit, configured to divide the stitching area of the first original image into a specified number of image areas; wherein the stitching area of the first original image refers to an area in the first original image used for image stitching with the second original image, and the image areas are divided based on depth information of each pixel in the stitching area; a second determining unit, configured to determine, if the target stitching object is included in the first area, a second area in the second original image that matches the first area; the first area being one of a specified number of image areas obtained by dividing the stitching area; and an area ratio of the target stitching object in the first area being not less than a second specified threshold; The stitching unit is configured to perform feature matching on pixel points in a first area of the first original image and a second area of the second original image; and re-stitch the first original image and the second original image in the panoramic image based on the matching result.

9. The device according to claim 8, characterized in that The first determining unit is specifically configured to: For each seam included in the panoramic image, determining two original images corresponding to the seam; Determining the similarity between the stitching regions of the two original images; If the similarity between the stitching areas of the two original images is less than a first specified threshold, determining that the two original images are the first original image and the second original image that need to be re-stitched; And / or, the dividing unit is specifically used for: determining a depth range included in the stitched area of the first original image based on the obtained depth information of each pixel in the stitched area of the first original image; the depth information of each pixel in the stitched area of the first original image is obtained based on the depth estimation module when stitching multiple original images to obtain a panoramic image; Dividing the depth range into the specified number of sub-depth ranges; determining a sub-region corresponding to each sub-depth range in the stitching region of the first original image as an image region; And / or, the second determining unit is specifically configured to: determining, based on each first feature point included in the first area of the first original image, a second feature point in the second original image that matches the first feature point; wherein the matching relationship between the first feature point in the first original image and the second feature point in the second original image refers to the matching relationship between the first feature point in the first original image and the second feature point in the second original image included in the panoramic image; determining a region where all second feature points are located as a second region in the second original image that matches the first region; And / or, before determining the second region matching the first region in the second original image, the dividing unit is further configured to: If the area ratio of the target stitching object in the first region is less than a second specified threshold, the specified number is updated, and the process returns to the step of dividing the stitching region of the first original image to obtain a specified number of image regions, until the area ratio of the target stitching object in the first region is no less than the second specified threshold; wherein the updated specified number is greater than the specified number before the update; And / or, after performing feature matching on the pixels in the first area in the first original image and the second area in the second original image, the stitching unit is further configured to: For each matched feature point pair in the first area and the second area, if the error value between two feature points in the feature point pair is greater than a third specified threshold, adjust the position of the feature point pair so that the error value between the two feature points in the feature point pair is not greater than the third specified threshold; And / or, after performing feature matching on the pixels in the first area in the first original image and the second area in the second original image, the stitching unit is further configured to: At least one pair of matched feature points is added to the first area and the second area; the positions of the added feature point pairs are different from the positions of the matched feature point pairs in the first area and the second area.

10. An electronic device, characterized in that: include: a processor and a machine-readable storage medium storing machine-executable instructions capable of being executed by the processor; The processor is configured to execute machine-executable instructions to implement the method according to any one of claims 1 to 7.

Images