Image processing method and device and electronic equipment

Abstract

The invention provides an image processing method and device and electronic equipment. According to the invention, after the first original image and the second original image which need to be spliced again are determined from the panoramic image, the splicing area of the first original image is divided into the specified number of image areas according to the depth information of each pixel point, and when any image area comprises the target splicing object, the splicing area of the first original image is divided into the specified number of image areas. Determining the image region as a region concerned by the splicing, determining a second region matched with the image region in the second original image, performing feature matching on pixel points in the image region and the second region, and further performing re-splicing on the first original image and the second original image based on a matching result; by determining the area where the target splicing object is located and performing matching and splicing only based on the feature points in the area, the panoramic image with a poor visual effect is re-spliced based on the area where the target splicing object belongs, and the splicing effect is improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and particularly to an image processing method, apparatus, and electronic device. Background Art

[0002] In an actual scenario, images captured by multiple image acquisition devices from different perspectives are usually stitched together to obtain a panoramic image with a wider perspective. During the image stitching process, in order to reduce the image misalignment phenomenon at the seam, image alignment processing is usually performed on the images to be stitched. However, even for the images to be stitched after image alignment processing, due to the low accuracy of image alignment processing or stitching, there will still be a problem that the visual effect of the stitched panoramic image is not good. Summary of the Invention

[0003] In view of this, this application provides an image processing method, apparatus, and electronic device, which re-stitch the panoramic image with poor visual effect obtained by stitching to improve the stitching effect.

[0004] The technical solutions provided by this application are as follows: According to an embodiment of the first aspect of this application, an image processing method is provided, and the method includes: Determine a first original image and a second original image that need to be re-stitched from a panoramic image stitched by multiple original images; the first original image and the second original image have seam misalignment in the panoramic image; Divide the stitching area of the first original image to obtain a specified number of image areas; wherein, the stitching area of the first original image refers to the area in the first original image used for image stitching with the second original image, and the image areas are divided based on the depth information of each pixel point in the stitching area; If the target stitching object is included in the first area, determine a second area in the second original image that matches the first area; the first area is one of the specified number of image areas obtained by dividing the stitching area; Perform feature matching on the pixel points in the first area of the first original image and the second area of the second original image; Re-stitch the first original image and the second original image in the panoramic image according to the matching result.

[0005] Optionally, the determining a first original image and a second original image that need to be re-stitched from a panoramic image stitched by multiple original images includes: For each seam included in the panoramic image, determine the two original images corresponding to the seam; Determine the similarity between the splicing regions of the two original images; If the similarity between the splicing regions of the two original images is less than the first specified threshold, determine that the two original images are the first original image and the second original image that need to be re-spliced.

[0006] Optionally, divide the splicing region of the first original image to obtain a specified number of image regions, including: According to the depth information of each pixel point in the splicing region of the obtained first original image, determine the depth range included in the splicing region of the first original image; the depth information of each pixel point in the splicing region of the first original image is obtained based on the depth estimation module when stitching multiple original images to obtain a panoramic image; Divide the depth range into the specified number of sub-depth ranges; Determine the sub-region corresponding to each sub-depth range in the splicing region of the first original image as an image region.

[0007] Optionally, the determining the second region in the second original image that matches the first region includes: According to each first feature point included in the first region in the first original image, determine the second feature point in the second original image that matches the first feature point; wherein, the matching relationship between the first feature point in the first original image and the second feature point in the second original image refers to the matching relationship between the first feature point in the first original image and the second feature point in the second original image included in the panoramic image; Determine the region where all the second feature points are located as the second region in the second original image that matches the first region.

[0008] Optionally, before determining the second region in the second original image that matches the first region, the method further includes: If the area ratio of the target splicing object in the first region is less than the second specified threshold, update the specified number and return to the step of dividing the splicing region of the first original image to obtain a specified number of image regions until the area ratio of the target splicing object in the first region is not less than the second specified threshold; wherein, the updated specified number is greater than the updated-before specified number.

[0009] Optionally, after performing feature matching on the pixel points in the first region of the first original image and the second region of the second original image, the method further includes: For each pair of matched feature points in the first region and the second region, if the error value between the two feature points in the pair of feature points is greater than a third specified threshold, adjust the positions of the pair of feature points so that the error value between the two feature points in the pair of feature points is not greater than the third specified threshold.

[0010] Optionally, after performing feature matching on the pixel points in the first region of the first original image and the second region of the second original image, the method further includes: Add at least one pair of matched feature points in the first region and the second region; the added pair of feature points has a different position from the pair of matched feature points in the first region and the second region.

[0011] According to an embodiment of the second aspect of the present application, there is provided an image processing apparatus, the apparatus includes: A first determination unit, configured to determine a first original image and a second original image that need to be re - stitched from a panoramic image stitched by multiple original images; the first original image and the second original image have a seam misalignment in the panoramic image; A division unit, configured to divide the stitching region of the first original image to obtain a specified number of image regions; wherein, the stitching region of the first original image refers to the region of the first original image used for image stitching with the second original image, and the image regions are obtained by dividing based on the depth information of each pixel point in the stitching region; A second determination unit, configured to determine a second region in the second original image that matches the first region if a target stitching object is included in the first region; the first region is one of the specified number of image regions obtained by dividing the stitching region; A stitching unit, configured to perform feature matching on the pixel points in the first region of the first original image and the second region of the second original image; and re - stitch the first original image and the second original image in the panoramic image according to the matching result.

[0012] Optionally, the first determination unit is specifically configured to: For each seam included in the panoramic image, determine the two original images corresponding to the seam; Determine the similarity between the stitching regions of the two original images; If the similarity between the stitching regions of the two original images is less than a first specified threshold, determine the two original images as the first original image and the second original image that need to be re - stitched; And / or, the division unit is specifically configured to: Determine the depth range included in the stitching area of the first original image according to the depth information of each pixel point in the stitching area of the obtained first original image; the depth information of each pixel point in the stitching area of the first original image is obtained based on a depth estimation module when stitching multiple original images to obtain a panoramic image; Divide the depth range into the specified number of sub-depth ranges; Determine the sub-region corresponding to each sub-depth range in the stitching area of the first original image as an image region; And / or, the second determination unit is specifically configured to: Determine the second feature points in the second original image that match each first feature point included in the first region of the first original image; wherein, the matching relationship between the first feature points in the first original image and the second feature points in the second original image refers to the matching relationship between the first feature points in the first original image included in the panoramic image and the second feature points in the second original image; Determine the region where all the second feature points are located as the second region in the second original image that matches the first region; And / or, before determining the second region in the second original image that matches the first region, the dividing unit is further configured to: If the area ratio of the target stitching object in the first region is less than a second specified threshold, update the specified number, and return to the step of dividing the stitching area of the first original image to obtain the specified number of image regions until the area ratio of the target stitching object in the first region is not less than the second specified threshold; wherein, the updated specified number is greater than the specified number before the update; And / or, after performing feature matching on the pixel points in the first region of the first original image and the second region of the second original image, the stitching unit is further configured to: For each pair of matched feature points in the first region and the second region, if the error value between the two feature points in the pair of feature points is greater than a third specified threshold, adjust the positions of the pair of feature points so that the error value between the two feature points in the pair of feature points is not greater than the third specified threshold; And / or, after performing feature matching on the pixel points in the first region of the first original image and the second region of the second original image, the stitching unit is further configured to: Add at least one pair of matched feature points in the first region and the second region; the added pair of feature points has a different position from the pair of matched feature points in the first region and the second region.

[0013] According to an embodiment of the third aspect of the present application, an electronic device is provided, including: a processor and a machine-readable storage medium, where the machine-readable storage medium stores machine-executable instructions that can be executed by the processor; the processor is configured to execute the machine-executable instructions to implement the method described in the first aspect.

[0014] As can be seen from the above technical solutions, after determining the first original image and the second original image that need to be re-stitched from the panoramic image in the present application, the stitching area of the first original image is divided into a specified number of image areas according to the depth information of each pixel point, and when any image area includes the target stitching object, the image area is determined as the area of concern for this stitching, and the second area in the second original image that matches the image area is determined, and feature matching is performed on the pixel points in the image area and the second area, and further, the first original image and the second original image are re-stitched based on the matching result; by determining the area where the target stitching object is located and performing matching and stitching only based on the feature points within this area, the panoramic image with poor visual effect is re-stitched based on the area to which the target stitching object belongs, improving the stitching effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings herein are incorporated into the specification and form a part of the specification, showing embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application.

[0016] Figure 1 It is a flowchart of the image processing method provided by the embodiment of the present application; Figure 2 It is a schematic diagram of an image processing system provided by the embodiment of the present application; Figure 3 It is a schematic diagram of the execution process of the panoramic stitching module provided by the embodiment of the present application; Figure 4 It is a schematic diagram of the execution process of the panoramic preview module provided by the embodiment of the present application; Figure 5 It is a schematic diagram of the execution process of the interaction optimization module provided by the embodiment of the present application; Figure 6A It is an example diagram of the original image in the indoor track scene provided by the embodiment of the present application; Figure 6B It is an example diagram of the original panoramic image in the indoor track scene provided by the embodiment of the present application; Figure 6C It is an example diagram of the target panoramic image in the indoor track scene provided by the embodiment of the present application; Figure 7A It is an example diagram of the original image in the outdoor railway scene provided by the embodiment of the present application; Figure 7BAn example diagram of the original panoramic image in the outdoor railway scenario provided by the embodiments of the present application; Figure 7C An example diagram of the target panoramic image in the outdoor railway scenario provided by the embodiments of the present application; Figure 8 A structural diagram of an image processing device provided by the embodiments of the present application; Figure 9 A schematic structural diagram of an electronic device provided by the embodiments of the present application. Detailed implementation manners

[0017] In order to enable those skilled in the art to better understand the technical solutions provided by the embodiments of the present application, and to make the above-mentioned objects, features, and advantages of the embodiments of the present application more obvious and understandable, the technical solutions in the embodiments of the present application will be further described in detail below with reference to the accompanying drawings.

[0018] In some actual application fields such as the security monitoring field, in order to obtain a monitoring range with a larger viewing angle, image acquisition is usually performed by multiple image acquisition devices with different viewing angles, and the images acquired by each image acquisition device are stitched together to obtain a panoramic image with a wider viewing angle.

[0019] When stitching the to-be-stitched images acquired by multiple image acquisition devices with different viewing angles, it is necessary to find the overlapping areas between the to-be-stitched images, and through image registration, the overlapping areas between the to-be-stitched images are completely aligned, and further, multiple to-be-stitched images are fused to obtain a panoramic image.

[0020] However, due to the inaccurate result of image registration or the existence of multiple-depth planes in the acquired images, there may be misalignment in the panoramic image after image stitching, resulting in an uncoordinated visual effect of the panoramic image.

[0021] Based on this, the present application provides an image processing method to re-stitch the panoramic image with poor visual effect and improve the effect of image stitching.

[0022] Please refer to Figure 1 , Figure 1 A flowchart of the image processing method provided by the embodiments of the present application.

[0023] As an embodiment, this method can be applied to a server in an image processing system.

[0024] As Figure 1 shown, this method may include the following steps: Step 101, determine a first original image and a second original image that need to be re-stitched from the panoramic image stitched by multiple original images.

[0025] In this embodiment, the original images with seam misalignment can be determined from the panoramic image. Since each seam corresponds to two original images, the two original images at the seam with misalignment can be denoted as the first original image and the second original image.

[0026] Herein, the first original image and the second original image are only used to describe the original images on both sides of the seam with splicing misalignment. For example, the original image on the left side of the seam can be denoted as the first original image, and the original image on the right side of the seam can be denoted as the second original image. Alternatively, the original image on the left side of the seam can be denoted as the second original image, and the original image on the right side of the seam can be denoted as the first original image. This application does not limit this.

[0027] In this embodiment, the original panoramic image may include one or more seams. To determine the first original image and the second original image that need to be re - spliced, it is first necessary to determine which seam is the one with seam misalignment.

[0028] Specifically, the method for determining the seam with misalignment can be to manually select the original images corresponding to the seam with image splicing misalignment in the original panoramic image by observing the positions with image splicing misalignment in the panoramic image; or it can be to automatically detect and select the original images corresponding to the seam with misalignment among the seams of the server. This application does not limit this.

[0029] As an embodiment, the method for determining the first original image and the second original image that need to be re - spliced from a panoramic image formed by splicing multiple original images may include: For each seam included in the panoramic image, determine the two original images corresponding to the seam; Determine the similarity between the splicing regions of the two original images; If the similarity between the splicing regions of the two original images is less than the first specified threshold, determine that the two original images are the first original image and the second original image that need to be re - spliced.

[0030] In this embodiment, for each seam included in the panoramic image, the two original images corresponding to the seam can be determined, and further, the similarity of the splicing regions (i.e., the overlapping regions) of the two original images can be calculated. For example, calculate the mean squared error (MSE, Mean Squared Error) or the structural similarity index (SSIM, Structural Similarity Index) of the splicing regions of the two original images, etc., to determine the similarity of the splicing regions of the two original images. This application does not limit the specific method for calculating the similarity.

[0031] Further, if it is determined that the similarity between the splicing regions of the two original images is less than the first specified threshold, it indicates that the difference between the splicing regions of the two original images is large and the splicing effect is poor. At this time, it can be determined that these two original images are the first original image and the second original image that need to be re-spliced.

[0032] In this embodiment, the server can be connected to the display device to display the original panoramic image and the image splicing misalignment area on the display device for the user to view.

[0033] Exemplarily, in this embodiment, after obtaining the original panoramic image, each seam included in the original panoramic image can be detected to determine whether there is a seam misalignment at the seam. If it is determined that there is a seam at the seam, the splicing regions of the two original images at the seam can be marked in the original panoramic image, for example, the splicing regions of the two original images are marked in the form of an external rectangular frame.

[0034] So far, the description of step 101 ends, and then step 102 is executed.

[0035] Step 102, divide the splicing region of the first original image to obtain a specified number of image regions.

[0036] Among them, the splicing region of the first original image refers to the region of the first original image used for image splicing with the second original image, and the image regions are obtained by dividing based on the depth information of each pixel point in the splicing region.

[0037] In this embodiment, the first original image and the second original image can carry the feature points included in the image and the attribute information of each feature point.

[0038] As an embodiment, the attribute information of the feature points can include the depth information of each feature point, and this depth information can be obtained based on the deployed depth estimation module when multiple original images are spliced to obtain a panoramic image.

[0039] Exemplarily, the specific method for dividing the splicing region of the first original image to obtain a specified number of image regions can include: According to the depth information of each pixel point in the splicing region of the obtained first original image, determine the depth range included in the splicing region of the first original image; the depth information of each pixel point in the splicing region of the first original image is obtained based on the depth estimation module when multiple original images are spliced to obtain a panoramic image; Divide the depth range into the specified number of sub-depth ranges; Determine the sub-region corresponding to each sub-depth range in the splicing region of the first original image as an image region.

[0040] In this embodiment, the depth range included in the splicing area of the first original image can be determined according to the depth information of each pixel point in the splicing area of the first original image. For example, if the minimum depth information of the pixel points included in the splicing area of the first original image is 0 and the maximum is 60, then it is determined that the depth range included in the splicing area of the first original image is [0, 60].

[0041] Furthermore, the depth range can be divided into a specified number of sub-depth ranges. Specifically, the specified number can be a preset number, such as 2. Then the depth range can be divided into 2 sub-depth ranges. For example, it can be evenly divided into 2 sub-ranges according to the size of the depth range, that is, divided into 2 sub-ranges of [0, 30] and [31, 60]; again, for example, the depth range can also be divided into 2 sub-ranges according to the depth information of the pixel points. For example, the depth information corresponding to 50% of the pixel points is divided into one sub-range, and the depth information corresponding to the other 50% of the pixel points is divided into another sub-range. For example, the depth information corresponding to 50% of the pixel points is all within the range of [0, 40], and the depth information corresponding to the other 50% of the pixel points is all within the range of [41, 60]. At this time, the depth range [0, 60] can be divided into 2 sub-ranges of [0, 40] and [41, 60]. The method for dividing the sub-ranges in this application is not limited.

[0042] After the division of the sub-depth ranges is completed, each sub-depth range corresponding sub-region in the splicing area of the first original image can be determined as an image region, and a specified number of image regions are determined.

[0043] So far, the description of step 102 ends, and then step 103 is executed.

[0044] Step 103, if the target splicing object is included in the first region, then determine the second region in the second original image that matches the first region.

[0045] Wherein, the first region is one of the specified number of image regions obtained by dividing the splicing region.

[0046] In this embodiment, the target splicing object refers to the splicing object concerned in this splicing. For example, in the scenario of monitoring the behavior of pedestrians in an image, only the area where the pedestrians are located needs to be concerned during splicing to achieve a better splicing effect. At this time, the target splicing object is the pedestrian; again, for example, in the scenario of monitoring vehicles in an image, only the area where the vehicles are located needs to be concerned during splicing. At this time, the target splicing object is the vehicle.

[0047] Exemplarily, if any one of the specified number of image regions determined in step 102 completely includes the target splicing object, then this region is denoted as the first region, and it is determined that this first region is the region that needs to be concerned about in this splicing, and the depth range to which this first region belongs is the splicing depth range that needs to be concerned about in this splicing.

[0048] Further, a second region matching the first region can be determined in the second original image to find the regions that need to be concerned about in this splicing in the two images.

[0049] Specifically, the method for determining the second region matching the first region in the second original image may include: For each first feature point included in the first region in the first original image, a second feature point matching this first feature point is determined in the second original image; wherein, the matching relationship between the first feature point in the first original image and the second feature point in the second original image refers to the matching relationship between the first feature point in the first original image and the second feature point in the second original image included in the panoramic image; The region where all the second feature points are located is determined as the second region matching the first region in the second original image.

[0050] In this embodiment, for each feature point (denoted as the first feature point) included in the first region in the first original image, the second feature point matching each first feature point in the first region in the second original image can be found according to the matching relationship between the first feature point in the first original image and the second feature point in the second original image recorded when splicing the panoramic image before.

[0051] Further, the region where all the second feature points are located can be determined as the second region matching the first region in the second original image.

[0052] So far, the description of step 103 ends, and step 104 is executed below.

[0053] Step 104, perform feature matching on the pixel points in the first region in the first original image and the second region in the second original image.

[0054] In this embodiment, the pixel points included in the first region and the second region obtained in step 103 can be subjected to feature matching, that is, only the pixel points in the first region and the second region are matched in the splicing region, and the pixel points in other regions do not participate in the matching process.

[0055] So far, the description of step 104 ends, and step 105 is executed below.

[0056] Step 105: Re - splice the first original image and the second original image in the panoramic image according to the matching result.

[0057] In this embodiment, the first original image and the second original image can be re - spliced according to the matching results of the first area and the second area in step 104. Since this splicing is for the area where the target splicing object to be concerned is located, a better splicing effect can be obtained.

[0058] As an embodiment, after performing feature matching on the pixel points in the first area of the first original image and the second area of the second original image, the method proposed in this embodiment may further include: For each pair of matched feature points in the first area and the second area, if the error value between the two feature points in the pair is greater than the third specified threshold, adjust the positions of the pair of feature points so that the error value between the two feature points in the pair is not greater than the third specified threshold.

[0059] In this embodiment, the attribute information of each feature point may further include the error value of the pair of matched feature points. After completing the matching of the pixel points in the first area and the second area, the pair of feature points with an error value greater than the third specified threshold in the pair of matched feature points can also be adjusted so that the error value between the two feature points in the adjusted pair of feature points is not greater than the third specified threshold.

[0060] Alternatively, when splicing subsequently, the pairs of feature points with an error value greater than the third specified threshold can be ignored, and only the pairs of feature points with an error value not greater than the third specified threshold are used for splicing to reduce errors.

[0061] As an embodiment, after performing feature matching on the pixel points in the first area of the first original image and the second area of the second original image, the method proposed in this embodiment may further include: Add at least one pair of matched feature points in the first area and the second area; the added pair of feature points has a different position from the pairs of matched feature points in the first area and the second area.

[0062] In this embodiment, after completing the matching of the pixel points between the first area and the second area, at least one pair of matched feature points can also be added in the first area and the second area according to actual needs, so that more useful features can participate in the splicing process and the visual effect of the spliced image can be improved.

[0063] So far, the description of step 105 ends.

[0064] In this embodiment, before determining the second region in the second original image that matches the first region, the method may further include: If the area ratio of the target splicing object in the first region is less than a second specified threshold, update the specified number, and return to the step of dividing the splicing region of the first original image to obtain a specified number of image regions until the area ratio of the target splicing object in the first region is not less than the second specified threshold; wherein, the updated specified number is greater than the specified number before the update.

[0065] In this embodiment, if the area ratio of the target splicing object in the first region is less than the second specified threshold, it indicates that the target splicing object accounts for too small a proportion in the divided first region. Direct splicing may result in matching and splicing a large number of objects that do not need attention, and at the same time, the splicing effect of the target splicing object that needs attention is poor.

[0066] At this time, it can be considered that the current division of the image region is not appropriate enough. The specified number can be updated, for example, adding a specified value (such as 1) on the basis of the original specified number, and returning to the step of dividing the splicing region of the first original image to obtain a specified number of image regions until the area ratio of the target splicing object in the first region is not less than the second specified threshold, indicating that the target splicing object accounts for a relatively large proportion in the divided first region at this time, and the splicing effect of matching and splicing for this region is better at this time.

[0067] In this embodiment, if it is found that the target splicing object is not completely contained in any image region, for example, the target splicing object spans multiple image regions, it indicates that the division of the image region may be too detailed and too many regions are divided. At this time, the specified number can be updated, for example, reducing the specified value (such as 1) on the basis of the original specified number, and returning to the step of dividing the splicing region of the first original image to obtain a specified number of image regions until the target splicing object is contained in the first region.

[0068] In this embodiment, the feature points carried by the first original image and the second original image and the attribute information of each feature point are recorded during the process of generating the panoramic image.

[0069] As an embodiment, the panoramic image can be obtained by the following method: Obtain K original images, where K is greater than or equal to 2; Extract the feature points of the K original images, match the same feature points in the original images with overlapping regions, and record the attribute information corresponding to each feature point; Fuse the K original images based on the matched feature points included in the K original images to obtain the panoramic image.

[0070] In this embodiment, the panoramic image can be obtained by stitching multiple original images captured by image acquisition devices with different perspectives.

[0071] Specifically, feature extraction can be performed on multiple original images to obtain multiple feature points corresponding to each original image. Further, the same feature points in the original images with overlapping regions are matched, and the original images with overlapping regions are aligned, while recording the attribute information corresponding to each feature point.

[0072] Further, the K original images can be fused based on the matched feature points included in the K original images to obtain the original panoramic image.

[0073] In this embodiment, after re - stitching the first original image and the second original image in the panoramic image according to the matching result to obtain a new panoramic image (denoted as the target panoramic image), it can be further detected whether there are misaligned seams in the target panoramic image. If there are still misaligned seams, it indicates that the current target panoramic image is still a panoramic image with uncoordinated visual effects. At this time, the step of determining the first original image and the second original image that need to be re - stitched from the panoramic image stitched by multiple original images can be returned until there are no misaligned seams in the target panoramic image, or the target panoramic image meets the visual effect requirements of the user.

[0074] So far, the description of Figure 1 ends.

[0075] After determining the first original image and the second original image that need to be re - stitched from the panoramic image in this application, the stitching area of the first original image is divided into a specified number of image areas according to the depth information of each pixel point. When any image area includes the target stitching object, the image area is determined as the area of concern for this stitching, and the second area in the second original image that matches the image area is determined. Feature matching is performed on the pixel points in the image area and the second area, and then the first original image and the second original image are re - stitched based on the matching result; by determining the area where the target stitching object is located and performing matching and stitching only based on the feature points within this area, the re - stitching of the panoramic image with poor visual effects is realized based on the area to which the target stitching object belongs, improving the stitching effect.

[0076] Please refer to Figure 2 , Figure 2 which is a schematic diagram of a display system provided by an embodiment of this application.

[0077] As shown in Figure 2 the figure, the image processing method proposed in this application can be applied to an image processing system.

[0078] The image processing system may include: a panoramic stitching module, a panoramic preview module, and an interactive optimization module.

[0079] Among them, the panoramic stitching module is used to perform stitching processes such as image registration and fusion on the original images to obtain a panoramic image; the panoramic preview module is used to display the panoramic image obtained by the panoramic stitching module and detect the misalignment at each seam in the panoramic image; the interactive optimization module is used to adjust the feature points in the original images on both sides of the misaligned seam determined by the user in the panoramic image and re-stitch them to specifically reduce the occurrence of misalignment.

[0080] Next, the above modules will be introduced in conjunction with Figures 3 to 5 the following.

[0081] Please refer to Figure 3 , Figure 3 which is a schematic diagram of the execution process of the panoramic stitching module provided in the embodiment of this application.

[0082] As shown in Figure 3 the figure, the panoramic stitching module is used for: Obtain multiple original images. The multiple original images can be collected by an image acquisition device included in the image processing system or input externally. This application does not limit this.

[0083] Perform image registration on each original image. The image registration process includes processes such as feature point matching, homography estimation, and projective transformation; during the image registration process, the attribute information of each feature point can be obtained. Among them, the attribute information may include the error value of each pair of matching feature points and the depth information of each feature point. The error value of each pair of matching feature points can be obtained during the processes of feature point matching, homography estimation, and projective transformation. The depth information of each feature point can be obtained according to a depth estimation module. The depth estimation module can be a depth estimation module that reuses an existing one in the current image processing system or a newly added module in the system. This application does not limit this.

[0084] Perform image fusion on the original images, such as generating a panoramic image through methods such as weighted average method, multi-resolution fusion, or priority region fusion. This application does not limit this.

[0085] Please refer to Figure 4 , Figure 4 which is a schematic diagram of the execution process of the panoramic preview module provided in the embodiment of this application.

[0086] As shown in Figure 4As shown, the panoramic preview module is used for: Obtain the stitching result of the panoramic stitching module and display the panoramic image.

[0087] Detect the misalignment at the seam in the panoramic image and superimpose and display it on the panoramic image (for example, display the circumscribed rectangle of the stitching area of the two original images corresponding to the misaligned seam). The method for detecting the misalignment at the seam in the panoramic image has been described in detail above and will not be elaborated here.

[0088] Please refer to Figure 5 , Figure 5 which is the schematic flowchart of the execution process of the interaction optimization module provided by the embodiment of the present application.

[0089] As Figure 5 shown, the interaction optimization module is used for: Select the misaligned seam. In this embodiment, after the user clicks on the misalignment at the seam in the panoramic preview module (each pair of overlapping images has a seam, and the misaligned seam can be detected by an algorithm or considered by the user to have a poor visual effect), the two original images corresponding to the seam can be obtained (denoted as the first original image and the second original image).

[0090] Divide the stitching area of the first original image to obtain a specified number of image areas. For example, divide the stitching area of the first original image into 3 image areas.

[0091] Determine whether the target stitching object is included in the first area. If not, it indicates that the current division of the stitching area of the first original image is inappropriate and too detailed. At this time, the value of the specified number can be reduced, and the step of dividing the stitching area of the first original image to obtain a specified number of image areas can be returned. For example, the stitching area of the first original image can be re-divided into 2 image areas.

[0092] If the target stitching object is included in the first area, it can be further determined whether the area ratio of the target stitching object in the first area is less than a second specified threshold. If so, it indicates that the current division of the stitching area of the first original image is inappropriate, and the target stitching object accounts for too small a proportion in this area, which is not conducive to stitching. At this time, the value of the specified number can be increased, and the step of dividing the stitching area of the first original image to obtain a specified number of image areas can be returned. For example, the stitching area of the first original image can be re-divided into 5 image areas.

[0093] If the area ratio of the target stitching object in the first area is not less than the second specified threshold, it indicates that the current division of the stitching area of the first original image is appropriate. At this time, the second area in the second original image that matches the first area can be determined.

[0094] Further, feature matching can be performed on the pixel points in the first region of the first original image and the second region of the second original image. Since the matched feature points (including the error values of each pair of matched points, the larger the error value, the greater the possibility that the matched points are inaccurate) and the estimated depth information saved in the panoramic stitching module will be displayed on the two original images, the user can adjust the feature points that have been matched by the algorithm. For example, first, the feature points in other regions except those in the first region and the second region in the stitching area can be ignored because these regions are not the regions of concern for this stitching. For another example, the matched points with larger error values can be focused on. If it is found that there are mis-matched feature points, these feature points can be ignored during stitching, or the feature points can be adjusted so that the error values of the adjusted feature points are smaller. In the regions with dislocation or obvious targets of concern, the depth information can be referred to to manually supplement feature points so that the feature points are on the planes in the same depth interval, which can better reduce the dislocation of this plane.

[0095] After adjusting the matched feature points, the first original image and the second original image can be re-stitched according to the matching result between the first region and the second region.

[0096] In this embodiment, after completing the re-stitching, if it is found that there are still seams where the stitching effect does not meet the requirements, the process of the dislocation where the user clicks on the seam in the panoramic preview module can be returned. If the stitching visual requirements are met, the interactive optimization module is exited and the panoramic image is output.

[0097] The image processing method proposed in this application will be described below through two specific embodiments.

[0098] Embodiment 1 Please refer to Figures 6A to 6C , Figure 6A which is an example diagram of the original image to be stitched in the indoor track scene provided by the embodiment of the present application, Figure 6B which is an example diagram of the original panoramic image in the indoor track scene provided by the embodiment of the present application, Figure 6C which is an example diagram of the target panoramic image in the indoor track scene provided by the embodiment of the present application.

[0099] As Figure 6A shown, in indoor scene stitching, the camera is usually mounted on the wall and shoots downward at an angle, so the shooting angle is not perpendicular to the ground and the shooting distance is relatively close. The images collected will obviously have multiple planes with different depths and large parallax.

[0100] As Figure 6BAs shown, due to the existence of multiple planes, when directly stitching these two images in the panoramic stitching module, the feature matching points usually fit on the wall surface farthest from the camera. In this way, the misalignment of the wall surface in the stitched panoramic image is not significant, while the misalignment at the track is obvious. Please refer to Figure 6B the stitched images within the red frame. Since the user mainly focuses on the middle track and requires the misalignment at the track to be as small as possible, Figure 6B the panoramic image shown does not meet the requirements.

[0101] As Figure 6C shown, after the panoramic preview module detects and displays the misalignment at the seam, clicking on the misalignment at the track in the panoramic preview module of the display system will enter the interactive optimization module, and the original images to be stitched will be displayed. In this embodiment, multiple planes can be divided according to the depth information of the feature points. The user can select the track plane to retain the feature points on this plane and can supplement more feature points on the track plane. Re-entering the panoramic stitching module, generating and previewing the panoramic image, it can be seen that Figure 6C there is no obvious misalignment at the track in

[0102] In this embodiment, after selecting the track plane, except for the points with large inherent error values, the points not on this plane (such as the feature points on the wall surface plane) can also be ignored because these points are correctly invalid points. It is precisely because a relatively large number of invalid points are used during matching that the suboptimal stitching result is caused.

[0103] In this embodiment, there are multiple methods for detecting the misalignment at the seam. In this embodiment, the structural similarity SSIM of the overlapping region of the two images is calculated. This index is more in line with the intuitive perception of the human eye. When SSIM is less than the set threshold, it is considered that there is an obvious misalignment, and the panoramic preview module will display this region. This application does not limit this.

[0104] So far, the description of Embodiment 1 is completed.

[0105] Embodiment 2 Please refer to Figures 7A to 7C , Figure 7A which is an example diagram of the original images to be stitched in the outdoor railway scenario provided by this application embodiment, Figure 7B which is an example diagram of the original panoramic image in the outdoor railway scenario provided by this application embodiment, Figure 7C which is an example diagram of the target panoramic image in the outdoor railway scenario provided by this application embodiment.

[0106] As Figure 7A shown, the camera setup for stitching outdoor railways is the same as the above problem. However, a more serious problem with railway stitching is that there are a large number of repetitive textures in this scenario. The area within the green square in the figure is the stitching area corresponding to the two images.

[0107] As Figure 7B shown, due to the repeated texture, there will be many mis-matches after the algorithm automatically matches, and the registration accuracy is very low. Therefore, there are obvious misalignments in the panoramic image itself ( Figure 7B the red frame area in).

[0108] As Figure 7C shown, after clicking on the misalignment and entering the interactive optimization module, first ignore all the mis-matched points that the algorithm has already matched (the error values of these points will be relatively large), and then supplement the correct matching points.

[0109] For example, a large number of incorrect points in the algorithm matching can be ignored, and according to the depth information of the feature points on the plane that needs to be concerned, such as the railway track, feature points are supplemented on this plane. For the plane where the slope is located behind the image, since it is not the plane that needs to be concerned, there is no need to supplement feature points for this plane. The feature points of the planes other than the plane that needs to be concerned, such as the railway track, can be ignored and not used for feature matching. After completing the supplement of the feature points, the panoramic stitching module can be entered again to generate a panoramic image and preview, and it can be seen that the track can be correctly corresponded.

[0110] In this embodiment, due to the particularity of the scene, there are a large number of repeated textures, resulting in a large number of mis-matched points in the algorithm. Therefore, the accuracy of image registration is very low and the misalignment is serious. Here, by ignoring the mis-matched points and the feature points on the non-concerned plane through the interactive optimization module and supplementing the correct matching points, the accuracy of image registration is successfully improved and the misalignment of the panoramic image is reduced.

[0111] So far, the description of Embodiment 2 is ended.

[0112] Please refer to Figure 8 , Figure 8 which is a structural diagram of an image processing device proposed in an embodiment of the present application. As Figure 8 shown, the device may include a first determination unit 801, a division unit 802, a second determination unit 803, and a stitching unit 804. Specifically, the device includes: The first determination unit 801 is configured to determine a first original image and a second original image that need to be re-stitched from a panoramic image stitched by multiple original images; there is a seam misalignment between the first original image and the second original image in the panoramic image; The division unit 802 is configured to divide the stitching area of the first original image to obtain a specified number of image areas; wherein, the stitching area of the first original image refers to the area in the first original image used for image stitching with the second original image, and the image areas are divided based on the depth information of each pixel point in the stitching area; A second determination unit 803, configured to determine a second region in the second original image that matches the first region if the target splicing object is included in the first region; the first region is one of a specified number of image regions obtained by dividing the splicing region. A splicing unit 804, configured to perform feature matching on pixel points in the first region of the first original image and the second region of the second original image; and re-splice the first original image and the second original image in the panoramic image according to the matching result.

[0113] Optionally, the first determination unit 801 is specifically configured to: For each seam included in the panoramic image, determine two original images corresponding to the seam; Determine the similarity between the splicing regions of the two original images; If the similarity between the splicing regions of the two original images is less than a first specified threshold, determine the two original images as the first original image and the second original image that need to be re-spliced; And / or, the dividing unit 802 is specifically configured to: According to the depth information of each pixel point in the splicing region of the obtained first original image, determine the depth range included in the splicing region of the first original image; the depth information of each pixel point in the splicing region of the first original image is obtained based on a depth estimation module when splicing multiple original images to obtain a panoramic image; Divide the depth range into the specified number of sub-depth ranges; Determine a sub-region corresponding to each sub-depth range in the splicing region of the first original image as an image region; And / or, the second determination unit 803 is specifically configured to: According to each first feature point included in the first region of the first original image, determine a second feature point in the second original image that matches the first feature point; wherein, the matching relationship between the first feature point in the first original image and the second feature point in the second original image refers to the matching relationship between the first feature point in the first original image and the second feature point in the second original image included in the panoramic image; Determine the region where all the second feature points are located as the second region in the second original image that matches the first region; And / or, before determining the second region in the second original image that matches the first region, the dividing unit 802 is further configured to: If the area ratio of the target splicing object in the first region is less than the second specified threshold, update the specified quantity, and return the step of dividing the splicing region of the first original image to obtain a specified number of image regions until the area ratio of the target splicing object in the first region is not less than the second specified threshold; wherein, the updated specified quantity is greater than the specified quantity before the update; And / or, after performing feature matching on the pixel points in the first region of the first original image and the second region of the second original image, the splicing unit 804 is further configured to: For each pair of matched feature points in the first region and the second region, if the error value between the two feature points in the pair of feature points is greater than the third specified threshold, adjust the positions of the pair of feature points so that the error value between the two feature points in the pair of feature points is not greater than the third specified threshold; And / or, after performing feature matching on the pixel points in the first region of the first original image and the second region of the second original image, the splicing unit 804 is further configured to: Add at least one pair of matched feature points in the first region and the second region; the added pair of feature points has a different position from the pair of matched feature points in the first region and the second region.

[0114] So far, the description of the image processing device is completed. Figure 8 in the description of the image processing device.

[0115] The embodiments of the present application also provide Figure 8 a description of the hardware structure of the device shown. The hardware structure is Figure 9 the structure in the electronic device shown. Please refer to Figure 9 , Figure 9 which is the structural diagram of the electronic device provided by the embodiments of the present application. As Figure 9 shown, the hardware structure may include: a processor and a machine-readable storage medium, and the machine-readable storage medium stores machine-executable instructions that can be executed by the processor; the processor is configured to execute the machine-executable instructions to implement the method disclosed in the above examples of the present application.

[0116] Based on the same application concept as the above method, the embodiments of the present application also provide a machine-readable storage medium, on which several computer instructions are stored, and when the computer instructions are executed by a processor, the method disclosed in the above examples of the present application can be implemented.

[0117] Exemplarily, the above machine-readable storage medium can be any electronic, magnetic, optical or other physical storage device that can contain or store information such as executable instructions, data, and so on. For example, the machine-readable storage medium can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or a combination thereof.

[0118] The above are only embodiments of the present application and are not used to limit the present application. For those skilled in the art, various changes and modifications can be made to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the scope of the claims of the present application.

Claims

1. An image processing method, characterized in that, The method includes: Determining a first original image and a second original image that need to be re - stitched from a panoramic image stitched by multiple original images; there is a seam misalignment between the first original image and the second original image in the panoramic image; Dividing the stitching area of the first original image to obtain a specified number of image areas; wherein, the stitching area of the first original image refers to the area in the first original image used for image stitching with the second original image, and the image areas are obtained by dividing based on the depth information of each pixel point in the stitching area; If a target stitching object is included in the first area, determining a second area in the second original image that matches the first area; the first area is one of the specified number of image areas obtained by dividing the stitching area; Performing feature matching on the pixel points in the first area of the first original image and the second area of the second original image; Re - stitching the first original image and the second original image in the panoramic image according to the matching result.

2. The method according to claim 1, wherein The determining a first original image and a second original image that need to be re - stitched from a panoramic image stitched by multiple original images includes: For each seam included in the panoramic image, determining the two original images corresponding to the seam; Determining the similarity between the stitching areas of the two original images; If the similarity between the stitching areas of the two original images is less than a first specified threshold, determining the two original images as the first original image and the second original image that need to be re - stitched.

3. The method according to claim 1, characterized in that The dividing the stitching area of the first original image to obtain a specified number of image areas includes: According to the depth information of each pixel point in the stitching area of the first original image already obtained, determining the depth range included in the stitching area of the first original image; the depth information of each pixel point in the stitching area of the first original image is obtained based on a depth estimation module when stitching multiple original images into a panoramic image; Dividing the depth range into the specified number of sub - depth ranges; Determining the sub - area corresponding to each sub - depth range in the stitching area of the first original image as an image area.

4. The method according to claim 1, characterized in that The determining a second area in the second original image that matches the first area includes: According to each first feature point included in the first area in the first original image, determining a second feature point in the second original image that matches the first feature point; wherein, the matching relationship between the first feature point in the first original image and the second feature point in the second original image refers to the matching relationship between the first feature point in the first original image and the second feature point in the second original image included in the panoramic image; Determining the area where all the second feature points are located as the second area in the second original image that matches the first area.

5. The method according to claim 1, characterized in that Before determining the second area in the second original image that matches the first area, the method further includes: If the area ratio of the target splicing object in the first region is less than the second specified threshold, update the specified quantity, and return the step of dividing the splicing region of the first original image to obtain a specified quantity of image regions until the area ratio of the target splicing object in the first region is not less than the second specified threshold; wherein, the updated specified quantity is greater than the specified quantity before the update.

6. The method according to claim 1, characterized in that, After performing feature matching on the pixel points in the first region of the first original image and the second region of the second original image, the method further includes: For each pair of matched feature points in the first region and the second region, if the error value between the two feature points in the pair of feature points is greater than the third specified threshold, adjust the positions of the pair of feature points so that the error value between the two feature points in the pair of feature points is not greater than the third specified threshold.

7. The method according to claim 1, characterized in that After performing feature matching on the pixel points in the first region of the first original image and the second region of the second original image, the method further includes: Add at least one pair of matched feature points in the first region and the second region; the added pair of feature points has a different position from the pair of matched feature points in the first region and the second region.

8. An image processing apparatus, characterized in that, The device includes: A first determination unit, configured to determine a first original image and a second original image that need to be re-spliced from a panoramic image formed by splicing multiple original images; there is a seam misalignment between the first original image and the second original image in the panoramic image. A division unit, configured to divide the splicing region of the first original image to obtain a specified quantity of image regions; wherein, the splicing region of the first original image refers to the region in the first original image used for image splicing with the second original image, and the image regions are obtained by dividing based on the depth information of each pixel point in the splicing region. A second determination unit, configured to determine a second region in the second original image that matches the first region if the target splicing object is included in the first region; the first region is one of the specified quantity of image regions obtained by dividing the splicing region. A splicing unit, configured to perform feature matching on the pixel points in the first region of the first original image and the second region of the second original image; re-splice the first original image and the second original image in the panoramic image according to the matching result.

9. The device according to claim 8, characterized in that, The first determination unit is specifically configured to: For each seam included in the panoramic image, determine the two original images corresponding to the seam; Determine the similarity between the splicing regions of the two original images; If the similarity between the splicing regions of the two original images is less than the first specified threshold, determine the two original images as the first original image and the second original image that need to be re-spliced; And / or, the division unit is specifically configured to: Determine the depth range included in the stitching area of the first original image according to the depth information of each pixel point in the stitching area of the obtained first original image; the depth information of each pixel point in the stitching area of the first original image is obtained based on a depth estimation module when stitching multiple original images to obtain a panoramic image; Divide the depth range into the specified number of sub-depth ranges; Determine the sub-region corresponding to each sub-depth range in the stitching area of the first original image as an image region; And / or, the second determining unit is specifically configured to: Determine the second feature point in the second original image that matches the first feature point according to each first feature point included in the first region in the first original image; wherein, the matching relationship between the first feature point in the first original image and the second feature point in the second original image refers to the matching relationship between the first feature point in the first original image included in the panoramic image and the second feature point in the second original image; Determine the region where all the second feature points are located as the second region in the second original image that matches the first region; And / or, before determining the second region in the second original image that matches the first region, the dividing unit is further configured to: If the area ratio of the target stitching object in the first region is less than a second specified threshold, update the specified number and return to the step of dividing the stitching area of the first original image to obtain the specified number of image regions until the area ratio of the target stitching object in the first region is not less than the second specified threshold; wherein, the updated specified number is greater than the updated-before specified number; And / or, after performing feature matching on the pixel points in the first region of the first original image and the second region of the second original image, the stitching unit is further configured to: For each pair of matched feature points in the first region and the second region, if the error value between the two feature points in the pair of feature points is greater than a third specified threshold, adjust the positions of the two feature points in the pair of feature points so that the error value between the two feature points in the pair of feature points is not greater than the third specified threshold; And / or, after performing feature matching on the pixel points in the first region of the first original image and the second region of the second original image, the stitching unit is further configured to: Add at least one pair of matched feature points in the first region and the second region; the added pair of feature points has a different position from the pair of matched feature points in the first region and the second region.

10. An electronic device, characterized in that, Includes: A processor and a machine-readable storage medium, the machine-readable storage medium stores machine-executable instructions that can be executed by the processor; The processor is configured to execute the machine-executable instructions to implement the method according to any one of claims 1 to 7.

Images