An image processing method, apparatus, electronic device, and storage medium

Abstract

The present disclosure relates to an image processing method and device, electronic equipment and storage medium. The method comprises: continuously capturing images of a target scene to obtain a sequence of original image frames; determining a first image frame of the sequence of original image frames as a current key image frame; determining a current associated image set corresponding to the current key image frame; determining, from the current associated image set, an image frame with the smallest area of a common view region of the current key image frame as the current key image frame; repeatedly performing the steps of determining a current associated image set corresponding to the current key image frame, and determining an image frame with the smallest area as the current key image frame; performing image stitching processing based on each key image frame to obtain a target panoramic image of the target scene. The present disclosure can improve the stitching quality of the target panoramic image.

Description

Technical Field

[0001] This disclosure relates to the field of image processing, and more particularly to an image processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] Most existing panoramic image stitching methods use equal-interval frame counts, equal-interval rotation angles, or fixed camera parameters to select stitched images and complete the stitching. They do not consider whether the content of the images can be stitched, and cannot adapt well to scenarios such as the uncertainty of image source motion, camera intrinsic parameters, and rotation parameters, making it difficult to guarantee the stitching quality of panoramic images. Summary of the Invention

[0003] This disclosure provides an image processing method, apparatus, electronic device, and storage medium, which can improve the quality of panoramic image stitching. The technical solution of this disclosure is as follows:

[0004] According to a first aspect of the present disclosure, an image processing method is provided, the method comprising:

[0005] Continuous image acquisition is performed on the target scene to obtain the original image frame sequence;

[0006] The first image frame in the original image frame sequence is determined as the current key image frame;

[0007] Determine the current associated image set corresponding to the current key image frame; any image frame in the current associated image set is an image frame in the original image frame sequence, and the area of ​​the region co-viewed with the current key image frame is greater than a first threshold.

[0008] The image frame with the smallest shared viewing area with the current key image frame in the current associated image set is determined as the current key image frame;

[0009] Repeat the following steps: determine the current associated image set corresponding to the current key image frame; determine the image frame with the smallest area of ​​the shared viewing region with the current key image frame in the current associated image set as the current key image frame; until the current key image frame is the last image frame of the original image;

[0010] Image stitching is performed on each key image frame to obtain a panoramic image of the target scene.

[0011] Further, determining the current associated image set corresponding to the current key image frame includes:

[0012] The image frame in the original image frame sequence that is adjacent to the current key image frame and is located after the current key image frame is determined as the current target image frame;

[0013] Determine the current area of ​​the shared viewing region between the current key image frame and the current target image frame;

[0014] If the area of ​​the current region is greater than a first threshold, the current target image frame is determined as an associated image frame;

[0015] The next image frame of the current target image frame is determined as the current target image frame;

[0016] Repeat the following steps: determine the area of ​​the shared viewing region between the current key image frame and the current target image frame; until the next image frame of the current target image frame is determined as the current target image frame; until the area of ​​the shared viewing region between the current key image frame and the current target image frame is less than or equal to the first threshold.

[0017] Each associated image frame is determined as the current associated image set corresponding to the current key image frame.

[0018] Furthermore, the method also includes:

[0019] If the area between the current key image frame and the next image frame is less than or equal to the first threshold, the next image frame is determined as the current key image frame.

[0020] Furthermore, both the current key image frame and the current target image frame include multiple tracking corner points;

[0021] Determining the current region area of ​​the shared viewing area between the current key image frame and the current target image frame includes:

[0022] Optical flow estimation is performed on the plurality of tracking corner points to determine the displacement of the plurality of tracking corner points in the current key image frame to the plurality of tracking corner points in the current target image frame;

[0023] The average displacement is determined based on the displacement of each of the multiple tracking corner points.

[0024] The area of ​​the current region is determined based on the average displacement.

[0025] Further, the image stitching process based on key image frames to obtain the target panoramic image of the target scene includes:

[0026] The key image frames are stitched together to obtain a first panoramic image;

[0027] Based on the area of ​​the shared viewing region of adjacent key image frames, the key image frames in the first panoramic image are shifted to obtain the second panoramic image.

[0028] Based on the number of target pixels in the second panoramic image, the cropping area of ​​the second panoramic image is determined; the target pixels are pixels with a pixel value of 0; the target pixels are located at the edge of the second panoramic image.

[0029] The area to be cropped is cropped to obtain the target panoramic image.

[0030] Furthermore, the second panoramic image comprises multiple rows of pixels; determining the cropping region of the second panoramic image based on the number of target pixels includes:

[0031] Determine the number of target pixels corresponding to the target pixel row; the target pixel row is a pixel row located at the edge of the second panoramic image and has a preset number of rows;

[0032] Based on the number of target pixels, the region to be cropped is determined. The number of pixel rows in the region to be cropped is less than or equal to the preset number of rows, and the number of target pixels in each pixel row of the region to be cropped is greater than a third threshold.

[0033] Furthermore, the common viewing region of the adjacent key image frames includes the target reference point;

[0034] The method of shifting the keyframes in the first panoramic image based on the area of ​​the shared viewing region of adjacent key image frames to obtain the second panoramic image includes:

[0035] The first panoramic image is determined as the current mapped image;

[0036] Determine the first displacement vector of the target reference point in the current key image frame and the next key image frame;

[0037] Map the current first displacement vector to the current mapped image to obtain the current mapped displacement vector of the current displacement vector in the current mapped image;

[0038] Based on the current mapping displacement vector and the area of ​​the current shared region, the current second displacement vector of the next key image frame relative to the current key frame is obtained;

[0039] Based on the current second displacement vector, the next key image frame is displaced to obtain the current second panoramic image;

[0040] The current initial second panoramic image is determined as the current mapped image;

[0041] The next key image frame is determined as the current key frame image;

[0042] Repeat the following steps: determine the target reference point in the current first relative displacement vector between the current key image frame and the next key image frame, until the next key image frame is determined as the current key image frame, until there is no next key image frame.

[0043] Furthermore, after shifting the next key image frame based on the current second displacement vector to obtain the current second panoramic image, the method further includes:

[0044] Based on preset weights, the common viewing areas between adjacent key image frames are fused to obtain the second panoramic image; the preset weights represent the common viewing areas corresponding to any one of the key image frames in the adjacent key image frames.

[0045] According to a second aspect of the present disclosure, an image processing apparatus is provided, the apparatus comprising:

[0046] The image acquisition module is used to continuously acquire images of the target scene to obtain the original image frame sequence;

[0047] The first determining module is used to determine the first image frame of the original image frame sequence as the current key image frame;

[0048] The second determining module is used to determine the current associated image set corresponding to the current key image frame; any image frame in the current associated image set is an image frame in the original image frame sequence, and the area of ​​the region co-viewed with the current key image frame is greater than a first threshold.

[0049] The third determining module is used to determine the image frame with the smallest area of ​​the shared viewing region with the current key image frame in the current associated image set as the current key image frame;

[0050] The repetitive execution module is used to repeatedly execute the following steps: determining the current associated image set corresponding to the current key image frame; determining the image frame with the smallest co-view area with the current key image frame in the current associated image set as the current key image frame; until the current key image frame is the last image frame of the original image.

[0051] The image stitching module is used to perform image stitching processing based on each key image frame to obtain a target panoramic image of the target scene.

[0052] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement an image processing method as described in any one of the first aspects above.

[0053] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein when instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform an image processing method as described in any one of the first aspects of the present disclosure.

[0054] According to a fifth aspect of the present disclosure, a computer program product including instructions is provided, which, when run on a computer, causes the computer to perform an image processing method as described in any one of the first aspects of the present disclosure.

[0055] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects:

[0056] This application determines the set of currently associated image frames corresponding to the current key frame image in the original image frame sequence based on the area of ​​the shared viewing region with the current key image frame, and determines the current key image frame in the set of associated image frames. This eliminates the need for prior knowledge of the image acquisition device parameters. Key image frames are selected and adjusted according to the image frame content to ensure the stitching quality of the target panoramic image. The number of images participating in the target panoramic image stitching is reduced in the original image frame sequence, improving stitching efficiency. Furthermore, multiple key image frames are selected based on the shared viewing region area. Considering whether there are shared viewing regions between images that can be stitched together, the stitching effect of the target panoramic image is further guaranteed. Among the key image frames participating in the target panoramic image stitching, the first and last key image frames correspond to the first and last image frames of the original image frame sequence, respectively, ensuring that the head and tail information of the target scene is preserved, thus achieving a closed loop for target panoramic image stitching.

[0057] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0058] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0059] Figure 1 This is a schematic diagram illustrating an application environment according to an exemplary embodiment;

[0060] Figure 2This is a flowchart illustrating an image processing method according to an exemplary embodiment;

[0061] Figure 3 This is a schematic flowchart illustrating a method for determining the combination of currently associated images according to an exemplary embodiment;

[0062] Figure 4 This is a flowchart illustrating a method for determining the area of ​​a current region according to an exemplary embodiment;

[0063] Figure 5 This is a flowchart illustrating a method for determining a target panoramic image according to an exemplary embodiment;

[0064] Figure 6 This is a flowchart illustrating a method for shifting key image frames according to an exemplary embodiment;

[0065] Figure 7 This is a schematic diagram illustrating the location of the common viewing area in adjacent image frames according to an exemplary embodiment;

[0066] Figure 8 This is a schematic diagram illustrating the location of the common viewing area in another adjacent image frame according to an exemplary embodiment;

[0067] Figure 9 This is a schematic diagram illustrating an image stitching effect by extracting key image frames at equal intervals, according to an exemplary embodiment.

[0068] Figure 10 This is a schematic diagram illustrating the image stitching effect corresponding to an image processing method according to an exemplary embodiment;

[0069] Figure 11 This is a schematic diagram illustrating the image stitching effect of matching target reference points across the entire image, according to an exemplary embodiment.

[0070] Figure 12 This is a schematic diagram illustrating the image stitching effect of target reference point matching in a shared viewing area according to an exemplary embodiment;

[0071] Figure 13 This is a detailed schematic diagram illustrating target reference point matching across an entire image according to an exemplary embodiment;

[0072] Figure 14 This is a detailed schematic diagram illustrating target reference point matching in a shared viewing area according to an exemplary embodiment;

[0073] Figure 15 This is a block diagram of an image processing apparatus according to an exemplary embodiment;

[0074] Figure 16This is a block diagram illustrating an electronic device for image processing according to an exemplary embodiment. Detailed Implementation

[0075] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0076] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0077] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0078] Please see Figure 1 , Figure 1 This is a schematic diagram illustrating an application environment according to an exemplary embodiment, such as... Figure 1 As shown, the application environment may include terminal 100 and server 200.

[0079] Terminal 100 can be used to provide image information display services to any user. Specifically, terminal 100 can be, but is not limited to, electronic devices such as smartphones, desktop computers, tablets, laptops, smart speakers, digital assistants, augmented reality (AR) / virtual reality (VR) devices, and smart wearable devices, or software running on the aforementioned electronic devices, such as applications. Optionally, the operating system running on the electronic device can be, but is not limited to, Android, iOS, Linux, Windows, etc.

[0080] In an optional embodiment, server 200 can provide background services to terminal 100, generating image information to be displayed by terminal 100. Specifically, server 200 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0081] In addition, it should be noted that, Figure 1 The example shown is merely one application environment provided by this disclosure. In practical applications, other application environments may also be included, such as more terminals.

[0082] In the embodiments described in this specification, the terminal 100 and the server 200 can be directly or indirectly connected through wired or wireless communication, and this disclosure does not impose any restrictions.

[0083] Figure 2 This is a flowchart illustrating an image processing method according to an exemplary embodiment. Figure 2 As shown, the method may include the following steps.

[0084] Step S210: Continuously acquire images of the target scene to obtain the original image frame sequence;

[0085] In this embodiment, the execution entity is a server, which needs to continuously acquire images of the target scene, process the acquired images into grayscale, and obtain an original image frame sequence. The original image frame sequence contains at least two image frames. In this embodiment, the image acquisition device for acquiring the original image frame sequence, such as a camera, can be the same image acquisition device, or multiple image acquisition devices can be used to continuously acquire images of the target scene. The target scene may include at least one image acquisition device acquiring images of the surrounding environment. For example, images captured by cameras installed at various positions on a vehicle in a 360° loop can be stitched together to form a panoramic image. During the application, the target scene can be acquired as completely as possible by moving the position of the image acquisition device or rotating the image acquisition device.

[0086] Step S220: Determine the first image frame of the original image frame sequence as the current key image frame;

[0087] Step S230: Determine the current associated image set corresponding to the current key image frame; any image frame in the current associated image set is an image frame in the original image frame sequence, and the area of ​​the region co-viewed with the current key image frame is greater than the first threshold.

[0088] In the original image frame sequence, for panoramic image stitching of the same target scene, the more images involved in the stitching without missing frames, the weaker the stitching effect of the panoramic image will be. In order to improve the stitching effect of the panoramic image, a key image frame is determined in the original image frame sequence. In this embodiment, the key image frame is the last image to participate in the target panoramic image stitching. The first image frame of the original image sequence is determined as the current key image frame. The current associated image set of the current key image frame is determined. In two images, the area where the same image information can be observed is the common viewing area. Among them, the image frames in the current associated image set are the image frames in the original image frames that have a common viewing area with the current key image frame, and the area of ​​the common viewing area is greater than a first threshold. Specifically, in order to facilitate the calculation of the common viewing area, the width of the common viewing area in any image frame is set to be consistent. For example, if the image size of an image frame in the original image frame series is consistent, all of them are (a*b), the size of the common viewing area is c*b, where b is constant and consistent with the width of any image frame. The area of ​​the common viewing area changes based on the change of c.

[0089] Step S240: Determine the image frame with the smallest co-view area with the current key image frame in the current associated image set as the current key image frame;

[0090] To reduce the number of image frames that will ultimately be used in panoramic image stitching, a key frame can be determined from the current set of associated images and used as the next key image frame. Specifically, the image frame with the smallest common viewing area is determined as the current key image frame. The smaller the common viewing area, the larger the area of ​​the non-common viewing area, which can reduce the number of image stitching steps and improve the image stitching effect.

[0091] Step S250, repeat the following steps: determine the current associated image set corresponding to the current key image frame; until the image frame with the smallest area of ​​the shared viewing region with the current key image frame in the current associated image set is determined as the current key image frame; until the current key image frame is the last image frame of the original image.

[0092] Step S260: Perform image stitching processing based on each key image frame to obtain a target panoramic image of the target scene.

[0093] The next key image frame is determined in the current associated image set and designated as the current key image frame. Then, the next key image frame is determined in the current key image set corresponding to the current key image frame, and so on, until the current key image frame is the last image frame of the original image. This determines all the key frame images that will participate in the stitching of the target panoramic image. The first key frame image and the last key frame image correspond to the first and last frame images of the original image frame sequence, respectively, ensuring that the head and tail information of the target panoramic stitched image is preserved and the stitching loop is closed.

[0094] This embodiment determines the set of currently associated image frames corresponding to the current key frame image in the original image frame sequence based on the area of ​​the shared viewing region with the current key image frame, and determines the current key image frame in the set of associated image frames. This eliminates the need for prior knowledge of the image acquisition device parameters. Key image frames are selected and adjusted according to the image frame content to ensure the stitching quality of the target panoramic image. Multiple key image frames are selected from the original image frame sequence, reducing the number of images participating in the target panoramic image stitching and ensuring both stitching effect and efficiency. Among the key image frames participating in the target panoramic image stitching, the first and last key image frames correspond to the first and last image frames of the original image frame sequence, respectively, ensuring the preservation of the beginning and end information of the target scene and achieving a closed loop for target panoramic image stitching.

[0095] Figure 3 This is a schematic flowchart illustrating a method for determining the combination of currently associated images according to an exemplary embodiment. Figure 3 As shown, the set of currently associated images corresponding to the current key image frame is determined, including:

[0096] Step S310: Determine the current target image frame as the image frame that is adjacent to the current key image frame in the original image frame sequence and is located after the current key image frame;

[0097] Step S320: Determine the current area of ​​the shared viewing region between the current key image frame and the current target image frame;

[0098] Step S330: If the area of ​​the current region is greater than the first threshold, the current target image frame is determined as the associated image frame;

[0099] Step S340: Determine the next image frame of the current target image frame as the current target image frame;

[0100] Step S350, repeat the following steps: determine the area of ​​the shared viewing region between the current key image frame and the current target image frame; until the next image frame of the current target image frame is determined as the current target image frame; until the area of ​​the shared viewing region between the current key image frame and the current target image frame is less than or equal to the first threshold.

[0101] Step S360: Determine each associated image frame as the current associated image set corresponding to the current key image frame.

[0102] In this embodiment, in the original image sequence, if the shared viewing area between the current key image frame and the current target image frame is greater than a first threshold, the current target image frame is determined as an associated image frame, and the next image frame in the original image frame sequence is determined as the current target image frame. It should be noted that only the shared viewing area between the current key image frame and the current target image frame is determined. For example, if the original image frame sequence includes image frame 1, image frame 2, image frame 3, and image frame 4 in sequence, and image frame 1 is the current key image frame, and image frame 2 is the current target image frame, then if the shared viewing area between the current key image frame and the current target image frame is greater than the first threshold, the current target image frame (image frame 2) is determined as an associated image frame, and the next image frame (image frame 3) is determined as the current target image frame. Given the current target image frame, if the shared viewing area between the current key image frame (image frame 1) and image frame 3 is greater than a first threshold, then image frame 3 is determined as an associated image frame, and image frame 4 is determined as the current target image frame. If the shared viewing area between the current key image frame (image frame 1) and image frame 4 is less than or equal to the first threshold, then the associated image set corresponding to the current key image frame is determined to include image frame 2 and image frame 3 in sequence. The next key image frame can be determined from image frame 2 and image frame 3. The image frame with the smallest shared viewing area with the current key image frame is determined from image frame 2 and image frame 3 as the current key image frame for the next cycle. Generally, the image frame that is farther away from the current key image frame has the smallest shared viewing area with the current key image frame. Therefore, image frame 3 can also be directly determined as the current key image frame for the next cycle.

[0103] In another implementation, when the area of ​​the shared viewing region between the current key image frame and the current target image frame is less than or equal to the first threshold, the area of ​​the shared viewing region between the current target image frame and the current key image frame in the previous image frame of the original image frame sequence is the smallest. The image acquisition device can change the moving direction of the image acquisition device while continuously acquiring images without affecting the selection of the key image frame.

[0104] In this embodiment, a shared viewing area based on the current key image frame is determined, and a set of associated images for the current key image frame is determined. Each image frame in the associated image set has a shared viewing area with the current key image frame that is greater than a first threshold. Each image frame in the key image set can become the next key image frame. This embodiment uses the area of ​​the shared viewing area to dynamically select key image frames, which is not limited to fixed image content or fixed image acquisition device placement. It does not require prior calculation of image frame overlap using image acquisition device parameters, thus providing conditions for reducing panoramic image stitching costs and improving panoramic image stitching effects.

[0105] In one implementation, the method further includes:

[0106] If the area between the current key image frame and the next image frame is less than or equal to a first threshold, the next image frame is determined as the current key image frame.

[0107] If the area of ​​the next image frame of the current key image frame is less than or equal to the first threshold, that is, the current key image frame does not have a corresponding current associated image set, the next image frame is directly determined as the current key image frame. For example, if the original image frame sequence includes image frame 1, image frame 2, image frame 3, and image frame 4 in sequence, and image frame 1 is the current key image frame, if the area of ​​the shared viewing region of the current key image frame and image frame 2 is less than or equal to the first threshold, then image frame 2 is directly determined as the current key image frame of the next cycle. It can be considered that when the associated image set corresponding to the current key image frame is an empty set, the next image frame of the current key image frame in the original image frame sequence is determined as the current key image frame of the next cycle, so as to ensure that the target scene is presented in the target panoramic image as much as possible and improve the integrity of the target panoramic image stitching.

[0108] In one implementation, both the current key image frame and the current target image frame include multiple tracking corner points;

[0109] In this context, tracking corner points are visually prominent points within an image frame, such as edge points or other easily referenced points. Corner point detection offers advantages like real-time performance and stability, allowing for optical flow estimation of pixels within the image frame. Optical flow is the "instantaneous velocity" of pixels moving on the imaging plane of a spatially moving object. Optical flow estimation calculates the displacement of pixels based on their velocity vector characteristics, thereby determining the object's movement distance and direction. This embodiment fully considers the principles of optical flow estimation. In situations of varying brightness or large displacement, tracking corner points are easily lost. This embodiment monitors changes in tracking corner points in real time and resets them based on the refresh of the current key image frame. Furthermore, it considers corner point density distribution, adjusting the maximum number of tracking corner points and the quality level parameters of detectable tracking corner points to maintain as many detected corner points as possible.

[0110] Figure 4 This is a flowchart illustrating a method for determining the area of ​​a current region according to an exemplary embodiment, such as... Figure 4 As shown, the current region area of ​​the shared viewing area between the current key image frame and the current target image frame is determined, including:

[0111] Step S410: Perform optical flow estimation on multiple tracking corner points to determine the displacement of multiple tracking corner points in the current key image frame to multiple tracking corner points in the current target image frame.

[0112] Step S420: Determine the average displacement based on the displacement corresponding to each of the multiple tracking corner points;

[0113] Step S430: Determine the area of ​​the current region based on the average displacement.

[0114] Specifically, the information stored for each key image frame includes the pixel value of that frame, the percentage of the shared viewing area with the preceding and following key frames, and the tracking corner point pair information corresponding to the preceding and following key frames. The percentage of the shared viewing area refers to the percentage of the area of ​​the shared viewing area between images to the entire image frame.

[0115] The area of ​​any given image frame is fixed. The current region is represented by the percentage of the image frame it occupies. Subsequent image frames are read sequentially. Optical flow estimation is used to obtain the optical flow tracing corner point pairs between the current key image frame and the current target image frame. The average displacement avg_move[i] of all point pairs is calculated. The current region percentage is determined by the following formula:

[0116] Pi=1-abs(avg_move[i]÷img_width)

[0117] Wherein, img_width is the image width, Pi is the percentage of the shared viewing area between the current key image frame and the current target image frame, and i is the i-th frame of the original image frame sequence. Correspondingly, a shared viewing area percentage threshold of Plimit can be set to determine whether the current target image frame is an associated image frame in the associated image set. If Pi is greater than or equal to Plimit, and the number and distribution of tracking corner points meet the requirements, the tracking corner points are monitored in real time until Pi is less than Plimit. In this embodiment, the changes in tracking corner points are monitored in real time, and the tracking corner points are reset according to the refresh of the current key image frame. At the same time, the density distribution of tracking corner points is also considered, and the maximum number of tracking corner points and the quality level parameters of detectable tracking corner points are adjusted to maintain as many detection corner points as possible. The changes in brightness and darkness, and the loss of tracking corner points due to large displacement may occur in optical flow estimation. The changes in tracking corner points are monitored in real time, which can ensure the accuracy of optical flow estimation in scenarios with rapid rotation of the image acquisition device and obvious changes in brightness and darkness, and further improve the accuracy of determining the area of ​​the shared viewing area.

[0118] Figure 5 This is a flowchart illustrating a method for determining a target panoramic image according to an exemplary embodiment, such as... Figure 5 As shown, image stitching is performed based on key image frames to obtain a panoramic image of the target scene, including:

[0119] Step S510: Stitch together the key image frames to obtain the first panoramic image;

[0120] Based on the order of each key image frame in the original image frame sequence, the images are stitched together to obtain the first panoramic image. The first key image frame of each key image frame is the first image frame of the original image frame sequence, and the last key image frame of each key image frame is the last image frame of the original image frame sequence, ensuring that the start and end information of the panoramic stitched image is preserved and the stitching is closed loop. The edges of the first panoramic image are straight lines, and each key image frame is placed in parallel.

[0121] Step S520: Based on the area of ​​the common viewing region of adjacent key image frames, shift each key image frame in the first panoramic image to obtain the second panoramic image.

[0122] In the first panoramic image, the common viewing areas of adjacent keyframes overlap. For example, the first panoramic image includes keyframe 1, keyframe 2, and keyframe 3. The common viewing areas between keyframe 1 and keyframe 2 overlap, and the common viewing areas between keyframe 2 and keyframe 3 overlap. To improve the stitching effect of the target panoramic image, the content overlap between adjacent keyframes should be minimized. Keyframe 2 and keyframe 3 can be shifted. For example, keyframe 2 is moved so that the common viewing areas of keyframe 1 and keyframe 2 are aligned. Common viewing area alignment means maximizing the overlap of content between adjacent keyframes. After keyframe 2 is moved, keyframe 3 is shifted based on the current position of keyframe 2. After the movement is completed, the second panoramic image is obtained. It can be considered that the common viewing areas of each keyframe in the second panoramic image overlap.

[0123] Step S530: Based on the number of target pixels in the second panoramic image, determine the cropping area of ​​the second panoramic image; the target pixels are pixels with a pixel value of 0; the target pixels are located at the edge of the second panoramic image;

[0124] Step S540: Crop the area to be cropped to obtain the target panoramic image.

[0125] After the key image frames are moved, the edges of the second panoramic image are jagged and not perfectly parallel. It is necessary to determine the cropping area to eliminate the jagged edges as much as possible while ensuring the integrity of the second panoramic image. The number of target pixels can characterize the jaggedness of the second panoramic image edges to a certain extent. The fewer the number of target pixels, the less jagged the edges of the second panoramic image, and the smaller the area to be cropped. The more target pixels, the larger the area to be cropped. By traversing the pixels with a value of 0 in the second panoramic image and determining the cropping area of ​​the second panoramic image based on the number of target pixels, the jagged edges of the second panoramic image can be cropped, which can improve the presentation effect of the second panoramic image on the target scene.

[0126] In one implementation, the second panoramic image comprises multiple rows of pixels; determining the cropping region of the second panoramic image based on the number of target pixels includes:

[0127] Determine the number of target pixels corresponding to the target pixel row; the target pixel row is a pixel row located at the edge of the second panoramic image and has a preset number of rows;

[0128] Based on the number of target pixels, the cropping region is determined. The number of pixel rows in the cropping region is less than or equal to the preset number of rows, and the number of target pixels in each pixel row of the cropping region is greater than the third threshold.

[0129] In this embodiment, the number of pixel rows in the area to be cropped cannot exceed a preset number of rows. The preset number of rows can be determined based on the degree of shaking during image acquisition. For example, if the degree of shaking is small, the shaking influence factor can be 0.01. If the number of pixel rows in each key image frame is 1000, then the preset number of rows is 10 = 0.01 × 1000, and the number of rows in the area to be cropped does not exceed 10 rows. The target pixels within the preset number of rows are traversed to determine the number of target pixels in each row. The number of rows where the number of target pixels is greater than the third threshold is equal to the number of pixel rows in the area to be cropped. For example, if the preset number of rows is 10, the number of target pixels in each of the 10 rows is determined. The first row is located at the edge of the second panoramic image. If the number of target pixels in the first row is greater than the third threshold, the number of target pixels in the second row is determined. If the number of target pixels in the second row is greater than the third threshold, the number of target pixels in the third row is determined. If the number of target pixels in 3 rows is less than or equal to the third threshold, then the number of pixel rows in the cropping area can be determined to be 3. If the number of target pixels in each row within the preset number of rows is greater than the third threshold, then the number of rows in the cropping area can be determined to be the preset number of rows. The third threshold can be determined based on factors such as the proportion of black objects in the target scene or the second panoramic image, the stillness of black objects in the image, and the brightness of the image. The above method of determining the number of rows in the cropping area can be used for the upper or lower edge of the second panoramic image. In this embodiment, the area of ​​the cropping area does not exceed the preset number of rows, which can prevent over-cropping of the second panoramic image and prevent the second panoramic image from completely presenting the target scene. It is only necessary to traverse the pixels in each row within the preset number of rows to determine the number of rows in the cropping area, which can improve the cropping efficiency of the second panoramic image. Cropping the jagged edges of the second panoramic image can improve the presentation effect of the second panoramic image on the target scene.

[0130] Figure 6 This is a flowchart illustrating a method for shifting key image frames according to an exemplary embodiment, such as... Figure 6 As shown, the common viewing area of ​​adjacent key image frames includes the target reference point;

[0131] Based on the area of ​​the shared viewing region of adjacent keyframes, the keyframes in the first panoramic image are shifted to obtain the second panoramic image, including:

[0132] If the content of the shared viewing area of ​​adjacent key image frames is repeated, this embodiment does not crop the repeated area, but extracts the shared viewing area into the first panoramic image, determines the movement vector of the next key image frame, and moves the next key image frame so that the feature points in the shared viewing area can overlap.

[0133] Step S610: Determine the first panoramic image as the current mapped image;

[0134] Step S620: Determine the current first displacement vector of the target reference point in the current key image frame and the next key image frame;

[0135] Step S630: Map the current first displacement vector onto the current mapped image to obtain the current mapped displacement vector of the current displacement vector in the current mapped image;

[0136] Here, the target reference point is a feature point of the target scene displayed in the shared viewing area. The shared viewing area of ​​the current keyframe and the next keyframe includes the target reference point, and the two target reference points are located at different positions in the previous keyframe and the next keyframe. Specifically, the percentage value or area of ​​the shared viewing area is determined. A negative percentage value represents the percentage of shared viewing area on the left side of the image, and a positive value represents the percentage of shared viewing area on the right side of the image. Except for the first and last keyframes, all other keyframes include the percentage of shared viewing area with the left adjacent keyframe image and the percentage of shared viewing area with the right adjacent keyframe image.

[0137] Target detection and matching of shared viewing area images of adjacent keyframes, image registration transformation, and image alignment are completed. Through local image feature matching, firstly, the detection, description, and matching of features of adjacent keyframe images are established at a coarse-grained level. Then, at a fine-grained level, sub-pixel level dense matching is refined to generate dense matching in areas with less texture. The above feature matching calculations yield the feature matching point pairs mkpts0_c and mkpts1_c of the shared viewing area images of adjacent keyframes. Since camera rotation can be either counter-clockwise or clockwise, the shared viewing area positions correspond to the following two cases: First, the shared viewing area images of consecutive keyframes are located on the right side of the previous image frame and the left side of the subsequent image frame, respectively, such as... Figure 7 As shown, the shaded area represents the shared viewing area; in the second case, the shared viewing area images of adjacent keyframes are located to the left of the previous frame and to the right of the subsequent frame, respectively, as shown... Figure 8As shown; in the two cases above, the coordinate transformation is calculated using the percentage of the shared viewing area: In the first case, the feature point coordinate offset of the left frame is w0_offset, and the feature point coordinate offset of the right frame is w1_offset, then w0_offset = img_width × (1 - abs(w_ratios)), w1_offset = 0; In the second case, the feature point coordinate offset of the left frame is w0_offset, and the feature point coordinate offset of the right frame is w1_offset, then w0_offset = 0, w1_offset = img_width × (1 - abs(w_ratios)), w1_offset = 0. idth×(1-abs(w_ratios)), where img_width is the width of the image and w_ratios is the percentage of the shared viewing area between adjacent key image frames. The translation transformation of the image is calculated based on the coordinates of the target reference points of adjacent key image frames in the coordinate system of the current mapped image, thus completing the alignment of the target reference points of adjacent key image frames. The above method maps the target reference points of adjacent key image frames to the current mapped image through coordinate system transformation. The current mapped displacement vector of the target reference point of the next key image frame relative to the target reference point in the current mapped image is calculated.

[0138] In another implementation, the current first displacement vector of the target reference point in the current key image frame and the next key image frame can be determined, and the first displacement vector can be mapped to the current mapped image through image coordinate system transformation to obtain the current mapped displacement vector.

[0139] Step S640: Based on the current mapped displacement vector and the area of ​​the current shared region, obtain the current second displacement vector of the next key image frame relative to the current key frame.

[0140] Based on the current displacement vectors corresponding to multiple target reference points, the RANSAC (Random Sample Consensus) algorithm is used to filter out feature point pairs for calculating the transformation matrix, thereby aligning the target reference points in the common viewing area of ​​adjacent key image frames and obtaining the current second displacement vector of the next key image frame relative to the current key image frame.

[0141] Step S650: Displace the next key image frame based on the current second displacement vector to obtain the current second panoramic image;

[0142] Step S660: Determine the current initial second panoramic image as the current mapped image;

[0143] Step S670: Determine the next key image frame as the current key frame image;

[0144] Step S680, repeat the following steps: determine the first relative displacement vector of the target reference point in the current key image frame and the next key image frame, until the next key image frame is determined as the current key image frame, until there is no next key image frame.

[0145] Before moving each key image frame, the current mapped image is the first panoramic image. When moving to the next key image frame, the current mapped image changes. The current mapped displacement vector of the next key image frame is obtained based on the current mapped image. In this implementation, only the target reference point corresponding to the common viewing area is selected. The target reference point is calculated through non-full-view target reference points, which can eliminate interference points in non-common viewing areas, such as repeated texture interference. This effectively reduces the mismatch of target reference points, better adapts to weak texture and repeated texture scenes, and improves the problems of misalignment and distortion in target panoramic image stitching.

[0146] In one implementation, after shifting the next target image based on the current displacement vector to obtain the current initial second panoramic image, the method further includes:

[0147] Based on preset weights, the common viewing areas between adjacent key image frames are fused to obtain a second panoramic image; the preset weights represent the common viewing areas corresponding to any one of the key image frames in adjacent key image frames.

[0148] The common viewing areas of the key image frames in the second panoramic image largely overlap, but errors may exist. To address this, the common viewing areas between adjacent key image frames are fused using preset weights of 1 and 0. This ensures that only the common viewing area corresponding to any one of the adjacent key image frames is displayed, preventing content clutter in the common viewing area. Specifically, the stitching line is set based on the percentage of the common viewing area, generating a mask. A Laplacian pyramid is constructed for adjacent key image frames and the mask, using the mask as weights of 1 and 0. The weights of each layer of the Laplacian pyramid from both images are multiplied and summed to obtain the second panoramic image. By shifting each key image frame and then fusing the shifted adjacent key image frames, compared to cropping the common viewing area and then stitching adjacent key image frames together, the vertical misalignment of the target panoramic image is reduced, improving the stitching effect.

[0149] In this embodiment, after the key image frames are stitched together to form the first panoramic image, the key image frames are moved. After moving one key image frame, the two adjacent key image frames can be fused based on a mask. Alternatively, after all key image frames have been moved, the adjacent key image frames can be fused. The fusion order of multiple adjacent key image frames can be set by the user. The displacement of each key image frame and the image fusion can be performed in parallel, which can improve the stitching efficiency of the panoramic image.

[0150] In one specific embodiment, a scenario for simulating panoramic image stitching of a spherical robot was designed. For some extreme scenarios, corresponding to factors such as weak texture, no texture, repeated texture, noise interference, foreground interference and parallax, five scenarios were designed, corresponding to 14 test cases, and experimental data were collected according to the proposed test plan.

[0151] Considering the performance requirements such as time consumption in the application scenario, the optical flow estimation module in the experiment adopted a sparse optical flow estimation method, combined with a corner detection tracking method. The algorithm was debugged and iterated until reasonable values ​​for corner detection and optical flow estimation were finally set, especially for the quality parameters of the tracking corners. An end-to-end feature matching model was used to obtain the feature point pair matching results between image frames. Then, the RANSAC random sampling consensus algorithm was used to select feature point pairs for calculating the transformation matrix, completing the alignment of target reference points in the common viewing area.

[0152] In 14 test cases, experiments compared the proposed embodiment with the traditional method of extracting key image frames at equal intervals. Specifically, the threshold for selecting the common viewing percentage was above 50%. The method provided in this embodiment requires fewer frames compared to the method of extracting key image frames at equal intervals for target panoramic image stitching. The method in this embodiment considers the common viewing area of ​​the image when selecting key frames, which basically ensures successful stitching. It avoids the situation in the method of equal interval frame extraction where the frames participating in the stitching do not have a common viewing area, resulting in the inability to select the target reference point to complete the image registration. Therefore, the stitching success rate is higher.

[0153] For example, the target scene is a location near a window in an office, and a panoramic stitching is performed using a fast in-place rotation mode, with the original image frame series having the same number of frames. Figure 9 The image shown is a stitched image with equal-interval frames. By extracting keyframes at equal intervals, a total of 56 key image frames were identified. One frame needs to be extracted from every two frames to complete the panoramic image stitching. (Refer to...) Figure 10 The image stitching effect diagram of this embodiment shown has identified 19 key image frames. It can also complete the stitching of panoramic images and can reasonably and effectively select as few frames as possible to effectively reduce the stitching time.

[0154] Furthermore, a comparative test was conducted on the method of extracting target reference points from the shared viewing region and matching them with target reference points in the entire image, which can effectively reduce the false matching rate. Taking a case of natural outdoor light on the first floor, a camera height of 15cm, and an elevation angle of 15 degrees as an example, the stitching results were compared between feature matching of the entire image and feature matching of only the shared viewing region, as shown below. Figure 11 , Figure 12 As shown. It can be seen that, Figure 12 There are no obvious splicing misalignments, and Figure 11Mismatches caused vertical or horizontal misalignment in the splicing, with three noticeable instances (shown within boxes in the image). Figure 11 Taking the misalignment as an example, for instance... Figure 13 , Figure 14 The results shown are the full-image target reference point matching and the shared-view area full-image target reference point matching, respectively. The frames involved are frames 236 and 276, corresponding to keyframe indices 8 and 9. Figure 13 The points within the box represent mismatches caused by repeated textures, while Figure 14 During the shared-view region matching process, the shared-view region of the two images was cropped to avoid this situation.

[0155] Figure 15 This is a block diagram of an image processing apparatus according to an exemplary embodiment. (Refer to...) Figure 15 The apparatus is capable of performing all the above-described method steps, and the apparatus includes:

[0156] The image acquisition module 1510 is used to continuously acquire images of the target scene to obtain a sequence of original image frames.

[0157] The first determining module 1520 is used to determine the first image frame of the original image frame sequence as the current key image frame;

[0158] The second determining module 1530 is used to determine the current associated image set corresponding to the current key image frame; any image frame in the current associated image set is an image frame in the original image frame sequence, and the area of ​​the region co-viewed with the current key image frame is greater than the first threshold.

[0159] The third determining module 1540 is used to determine the image frame with the smallest area of ​​the shared viewing region with the current key image frame in the current associated image set as the current key image frame;

[0160] The repeat execution module 1550 is used to repeatedly execute the following steps: determine the current associated image set corresponding to the current key image frame; determine the image frame with the smallest area of ​​the shared viewing region with the current key image frame in the current associated image set as the current key image frame; until the current key image frame is the last image frame of the original image.

[0161] The image stitching module 1560 is used to perform image stitching processing based on each key image frame to obtain a panoramic image of the target scene.

[0162] The second determining module 1530 includes

[0163] The fourth determining module is used to determine the image frame in the original image frame sequence that is adjacent to the current key image frame and located after the current key image frame as the current target image frame;

[0164] The fifth determining module is used to determine the current area of ​​the shared viewing region between the current key image frame and the current target image frame;

[0165] The sixth determining module is used to determine the current target image frame as an associated image frame when the area of ​​the current region is greater than the first threshold.

[0166] The seventh determining module is used to determine the next image frame of the current target image frame as the current target image frame;

[0167] The first repetitive execution module is used to repeatedly execute the following steps: determining the area of ​​the shared viewing region between the current key image frame and the current target image frame; determining the next image frame of the current target image frame as the current target image frame; until the area of ​​the shared viewing region between the current key image frame and the current target image frame is less than or equal to a first threshold.

[0168] The eighth determination module is used to determine each associated image frame as the current associated image set corresponding to the current key image frame.

[0169] The image processing apparatus also includes:

[0170] The eighth determining module is used to determine the next image frame as the current key image frame when the area between the current key image frame and the next image frame is less than or equal to a first threshold.

[0171] The fifth determination module also includes:

[0172] The ninth determination module is used to perform optical flow estimation on multiple tracking corner points, and to determine the displacement of multiple tracking corner points in the current key image frame to multiple tracking corner points in the current target image frame; both the current key image frame and the current target image frame include multiple tracking corner points;

[0173] The tenth determination module is used to determine the average displacement based on the displacement of each of the multiple tracking corner points.

[0174] The eleventh determination module is used to determine the area of ​​the current region based on the average displacement.

[0175] The image stitching module also includes:

[0176] The first stitching module is used to stitch key image frames together to obtain the first panoramic image.

[0177] The displacement module is used to displacement the key image frames in the first panoramic image based on the area of ​​the shared viewing region of adjacent key image frames to obtain the second panoramic image.

[0178] The twelfth determining module is used to determine the cropping area of ​​the second panoramic image based on the number of target pixels in the second panoramic image; the target pixels are pixels with a pixel value of 0; the target pixels are located at the edge of the second panoramic image;

[0179] The cropping module is used to crop the area to be cropped to obtain the target panoramic image.

[0180] The twelfth determination module includes:

[0181] The thirteenth determining module is used to determine the number of target pixels corresponding to the target pixel row; the target pixel row is a pixel row located at the edge of the second panoramic image and the number of rows is a preset number; the second panoramic image includes multiple pixel rows;

[0182] The fourteenth module is used to determine the cropping region based on the number of target pixels. The number of pixel rows in the cropping region is less than or equal to the preset number of rows, and the number of target pixels in each pixel row of the cropping region is greater than the third threshold.

[0183] The displacement module includes:

[0184] The fifteenth module is used to determine the first panoramic image as the current mapped image; the common viewing area of ​​adjacent key image frames includes the target reference point;

[0185] The sixteenth module is used to determine the current first displacement vector of the target reference point in the current key image frame and the next key image frame;

[0186] The mapping module is used to map the current first displacement vector onto the current mapped image, so as to obtain the current mapped displacement vector of the current displacement vector in the current mapped image;

[0187] The displacement vector determination module is used to obtain the current second displacement vector of the next key image frame relative to the current key frame based on the current mapped displacement vector and the area of ​​the current shared region.

[0188] The first displacement module is used to displace the next key image frame based on the current second displacement vector to obtain the current second panoramic image.

[0189] The seventeenth module is used to determine the current initial second panoramic image as the current mapped image;

[0190] The eighteenth module is used to determine the next keyframe as the current keyframe image;

[0191] The second repetitive execution module is used to repeatedly execute the following steps: determining the current first relative displacement vector of the target reference point between the current key image frame and the next key image frame, until the next key image frame is determined as the current key image frame, until there is no next key image frame.

[0192] The image processing apparatus also includes:

[0193] The image fusion module is used to fuse the common viewing areas between adjacent key image frames based on preset weights to obtain a second panoramic image; the preset weights represent the common viewing areas corresponding to any one of the key image frames in the adjacent key image frames.

[0194] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0195] Figure 16 This is a block diagram illustrating an electronic device for image processing according to an exemplary embodiment. The electronic device may be a terminal, and its internal structure diagram may be as follows: Figure 16 As shown, the electronic device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements an image processing method. The display screen can be a liquid crystal display (LCD) or an e-ink display. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the device's casing, or an external keyboard, touchpad, or mouse.

[0196] Those skilled in the art will understand that Figure 16 The structure shown is merely a block diagram of a portion of the structure related to the present disclosure and does not constitute a limitation on the electronic device to which the present disclosure is applied. A specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0197] In an exemplary embodiment, an electronic device is also provided, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the image processing method as described in the embodiments of this disclosure.

[0198] In an exemplary embodiment, a computer-readable storage medium is also provided, wherein when the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the image processing method of the present disclosure.

[0199] In an exemplary embodiment, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform the image processing method of the present disclosure embodiments.

[0200] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0201] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0202] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that, The method includes: Continuous image acquisition is performed on the target scene to obtain the original image frame sequence; The first image frame in the original image frame sequence is determined as the current key image frame; The image frame in the original image frame sequence that is adjacent to the current key image frame and located after the current key image frame is determined as the current target image frame; both the current key image frame and the current target image frame include multiple tracking corner points; Optical flow estimation is performed on the multiple tracking corner points to determine the displacement of the multiple tracking corner points in the current key image frame to the multiple tracking corner points in the current target image frame; the average displacement is determined based on the displacement of each of the multiple tracking corner points; and the current area of ​​the co-view region of the current key image frame and the current target image frame is determined based on the average displacement. If the area of ​​the current region is greater than a first threshold, the current target image frame is determined as an associated image frame; the next image frame of the current target image frame is determined as the current target image frame. Repeat the following steps: determine the area of ​​the shared viewing region between the current key image frame and the current target image frame; until the next image frame of the current target image frame is determined as the current target image frame; until the area of ​​the shared viewing region between the current key image frame and the current target image frame is less than or equal to the first threshold. Each associated image frame is determined as the current associated image set corresponding to the current key image frame; The image frame with the smallest shared viewing area with the current key image frame in the current associated image set is determined as the current key image frame; Repeat the following steps: determine the current associated image set corresponding to the current key image frame; determine the image frame with the smallest area of ​​the shared viewing region with the current key image frame in the current associated image set as the current key image frame; until the current key image frame is the last image frame of the original image frame sequence; Image stitching is performed on each key image frame to obtain a panoramic image of the target scene.

2. The method according to claim 1, characterized in that, The method further includes: If the area of ​​the shared viewing region between the current key image frame and the next image frame is less than or equal to the first threshold, the next image frame is determined as the current key image frame.

3. The method according to claim 1, characterized in that, The image stitching process based on each key image frame to obtain the target panoramic image of the target scene includes: The key image frames are stitched together to obtain the first panoramic image; Based on the area of ​​the shared viewing region of adjacent key image frames, the key image frames in the first panoramic image are shifted to obtain the second panoramic image. Based on the number of target pixels in the second panoramic image, the cropping area of ​​the second panoramic image is determined; the target pixels are pixels with a pixel value of 0; the target pixels are located at the edge of the second panoramic image. The area to be cropped is cropped to obtain the target panoramic image.

4. The method according to claim 3, characterized in that, The second panoramic image comprises multiple rows of pixels; determining the cropping region of the second panoramic image based on the number of target pixels in the second panoramic image includes: Determine the number of target pixels corresponding to the target pixel row; the target pixel row is a pixel row located at the edge of the second panoramic image and has a preset number of rows; Based on the number of target pixels, the region to be cropped is determined. The number of pixel rows in the region to be cropped is less than or equal to the preset number of rows, and the number of target pixels in each pixel row of the region to be cropped is greater than a third threshold.

5. The method according to claim 3, characterized in that, The common viewing area of ​​adjacent key image frames includes the target reference point; The second panoramic image is obtained by shifting each key image frame in the first panoramic image based on the area of ​​the shared viewing region of adjacent key image frames, including: The first panoramic image is determined as the current mapped image; Determine the first displacement vector of the target reference point in the current key image frame and the next key image frame; Map the current first displacement vector to the current mapped image to obtain the current mapped displacement vector of the current first displacement vector in the current mapped image; Based on the current mapping displacement vector and the area of ​​the current shared region, the current second displacement vector of the next key image frame relative to the current key image frame is obtained; Based on the current second displacement vector, the next key image frame is displaced to obtain the current initial second panoramic image; The current initial second panoramic image is determined as the current mapped image; The next key image frame is determined as the current key frame image; Repeat the following steps: determine the target reference point in the current first relative displacement vector between the current key image frame and the next key image frame, until the next key image frame is determined as the current key image frame, until there is no next key image frame.

6. The method according to claim 5, characterized in that, After shifting the next key image frame based on the current second displacement vector to obtain the current second panoramic image, the method further includes: Based on preset weights, the common viewing areas between adjacent key image frames are fused to obtain the second panoramic image; the preset weights represent the common viewing areas corresponding to any one of the key image frames in the adjacent key image frames.

7. An image processing apparatus, characterized in that, The device includes: The image acquisition module is used to continuously acquire images of the target scene to obtain the original image frame sequence; The first determining module is used to determine the first image frame of the original image frame sequence as the current key image frame; The second determining module is used to determine the image frame in the original image frame sequence that is adjacent to the current key image frame and located after the current key image frame as the current target image frame; both the current key image frame and the current target image frame include multiple tracking corner points; optical flow estimation is performed on the multiple tracking corner points to determine the displacement amounts from the multiple tracking corner points in the current key image frame to the multiple tracking corner points in the current target image frame; an average displacement amount is determined based on the displacement amounts corresponding to each of the multiple tracking corner points; and the current common viewing area of ​​the current key image frame and the current target image frame is determined based on the average displacement amount. The area of ​​the region is determined; if the current area is greater than a first threshold, the current target image frame is determined as an associated image frame; the next image frame of the current target image frame is determined as the current target image frame; the steps are repeated: determining the area of ​​the shared viewing region of the current key image frame and the current target image frame; until the next image frame of the current target image frame is determined as the current target image frame; until the area of ​​the shared viewing region of the current key image frame and the current target image frame is less than or equal to the first threshold; each associated image frame is determined as the current associated image set corresponding to the current key image frame; The third determining module is used to determine the image frame with the smallest area of ​​the shared viewing region with the current key image frame in the current associated image set as the current key image frame; The repetitive execution module is used to repeatedly execute the following steps: determining the current associated image set corresponding to the current key image frame; determining the image frame with the smallest co-view area with the current key image frame in the current associated image set as the current key image frame; until the current key image frame is the last image frame of the original image. The image stitching module is used to perform image stitching processing based on each key image frame to obtain a target panoramic image of the target scene.

8. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the image processing method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is able to perform the image processing method as described in any one of claims 1 to 6.

Images