Image stitching method and system
By adjusting the target tracking frequency and memory application strategy according to the frame interval in camera image stitching with high frame rates, the problem of high frame rates of camera stitching delay is solved, and faster and more efficient image stitching is achieved.
Patent Information
- Application Number
- CN202210277462.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-21
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2042-03-21
AI Technical Summary
When using a camera with a higher frame rate to shoot videos and stitch images, it is easy to cause a long delay to obtain a complete stitching image, which is difficult to meet the needs of real-time processing.
By judging the interval between the video image and the last target tracking, if the interval does not exceed the preset frame number, the target tracking will not be performed. If the interval exceeds the target tracking, the subsequent frame count video image will be processed using the previous tracking result to reduce the The target tracks the number of times and stitches after obtaining all images to be stitched to save memory.
It improves the speed and efficiency of image stitching, reduces the number of target tracking and memory application times, and improves the accuracy and processing efficiency of stitching.
Smart Images

Figure CN114723607B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of image processing, and in particular to an image stitching method and system. Background Art
[0002] Image stitching technology is a technology that stitches together several overlapping video images into a seamless panorama or high-resolution video image. It is widely used in video image recording of moving objects.
[0003] In the related art, when recording information of a moving object, a camera is used to capture a video of the moving object entering a shooting area, and the captured video is tracked frame by frame for image stitching, that is, each frame is detected based on the feature position selected in the previous frame, and then the cropping width on each video image is determined for image stitching, and finally, after the moving object leaves the shooting area, a complete stitched image of the moving object is obtained.
[0004] With respect to the above-mentioned related technologies, the inventors have found that when a camera with a high frame rate is used to shoot a video and perform image stitching, it is easy to cause a long delay to obtain a complete stitched image. Summary of the invention
[0005] In order to reduce the delay when shooting with a camera with a higher frame rate and performing image stitching, the present application provides an image stitching method and system.
[0006] In a first aspect, an image stitching method provided in the present application adopts the following technical solution.
[0007] An image stitching method, comprising:
[0008] Acquire a frame of video image of a video containing a moving object;
[0009] Based on the frame number interval between the currently acquired video image and the video image for which target tracking was performed last time, it is determined whether target tracking needs to be performed on the currently acquired video image; if not, a new image to be spliced is obtained based on the image to be spliced obtained by the last target tracking; if yes, target tracking is performed on the currently acquired video image to obtain a new image to be spliced; and,
[0010] The obtained images to be stitched are stitched together; after all the images to be stitched together are stitched together, a complete stitched image of the moving object is obtained.
[0011] By adopting the above technical solution, when image stitching is performed, target tracking takes the longest time; in the present application, if a certain frame of video image has been subjected to target tracking, then the video images of a preset number of frames after this frame of video image can be processed using the previous tracking result; after an interval of a preset number of frames, the video image is reselected for target tracking, and the re-tracking result is continued to be used in the processing of the video images of the preset number of frames thereafter, so it is not necessary to perform target tracking on each frame of video image, which reduces the number of target tracking times and thus improves the stitching speed.
[0012] Optionally, stitching the obtained images to be stitched includes:
[0013] Saving the obtained image to be stitched;
[0014] After all the images to be stitched are obtained, the sum of the memory occupied by all the images to be stitched is calculated to obtain the total occupied memory;
[0015] Applying for memory space based on the total occupied memory; the applied memory space is not less than the total occupied memory;
[0016] The images to be stitched are copied to the memory space in sequence according to the order of frames to perform image stitching.
[0017] By adopting the above technical solution, after each image to be stitched is obtained, only the video image data of the image to be stitched is saved, and stitching is not performed directly. Image stitching is performed after all the images to be stitched are obtained, so there is no need to apply for memory for each stitching, which saves memory and improves stitching efficiency.
[0018] Optionally, performing target tracking on the currently acquired video image to obtain a new image to be stitched includes:
[0019] Set the tracking area of the moving object in the video image and record the position coordinates of the tracking area;
[0020] Using a tracking algorithm to track the target in the tracking area, obtaining the pixel displacement of the moving object in the current video image and the previous video image; and
[0021] A displacement video image corresponding to the pixel displacement of the moving object in the current video image is obtained, and the displacement video image is used as the image to be spliced.
[0022] Optionally, obtaining a new image to be stitched based on the image to be stitched obtained by the previous target tracking includes:
[0023] Obtain the last pixel displacement based on the result of the last target tracking; and,
[0024] A new image to be spliced is obtained based on the last image to be spliced and the last pixel displacement.
[0025] By adopting the above technical solution, if the previous frame of the current video image has performed target tracking, the previous pixel displacement is obtained according to the result of the previous target tracking to obtain the current image to be spliced; if the previous frame of the current video image has not performed target tracking, the image is still captured according to the pixel displacement obtained by the previous target tracking to obtain the current image to be spliced. Therefore, it is not necessary to perform target tracking on each frame of the video image, which reduces the number of target tracking times and thus improves the splicing speed.
[0026] Optionally, before setting the tracking area of the moving object in the video image, the method further includes: performing image enhancement processing on the video image, wherein the image enhancement processing includes adaptive equalization.
[0027] By adopting the above technical solutions, adaptive equalization makes the details of moving objects more prominent and improves the accuracy of the target tracking algorithm.
[0028] Optionally, the video containing the moving object is obtained by capturing video images through a 1.44 mm fisheye lens.
[0029] By adopting the above technical solution, the 1.44mm fisheye lens can shoot a wide range of scenes at a close distance, thereby being able to shoot large moving objects.
[0030] Optionally, after acquiring a frame of video image of a video containing a moving object, the method further includes: performing distortion correction on the video image.
[0031] By adopting the above technical solution, distortion correction improves the accuracy of the tracking algorithm, making the stitched image smoother and the imaging effect better.
[0032] In a second aspect, the present application provides an image stitching system that adopts the following technical solution.
[0033] An image stitching system, comprising:
[0034] An acquisition module, used for acquiring a frame of video image of a video containing a moving object;
[0035] A judgment module, based on the frame number interval between the currently acquired video image and the video image last used for target tracking, judges whether it is necessary to perform target tracking on the currently acquired video image;
[0036] A first to-be-stitched image acquisition module, which acquires a new to-be-stitched image based on the to-be-stitched image acquired by the last target tracking when there is no need to perform target tracking on the currently acquired video image;
[0037] A second image acquisition module for obtaining images to be stitched, which performs target tracking on the currently acquired video image to obtain a new image to be stitched when target tracking is required for the currently acquired video image; and
[0038] The stitching module is used to stitch the obtained images to be stitched; after all the images to be stitched are stitched, a complete stitched image of the moving object is obtained.
[0039] In a third aspect, the present application discloses a computer device, including a memory and a server, wherein the memory stores a computer program that is loaded by the server and executes any of the above methods.
[0040] In a fourth aspect, the present application discloses a computer-readable storage medium storing a computer program that can be loaded by a server and execute any of the above methods. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] Figure 1 It is a flow chart of one implementation of an image stitching method in an embodiment of the present application;
[0042] Figure 2 yes Figure 1 A flowchart of one implementation of step 105;
[0043] Figure 3 This is a flow chart of an embodiment of the present application for performing target tracking to obtain a new image to be stitched;
[0044] Figure 4 It is a system block diagram of one implementation of an image stitching system in an embodiment of the present application;
[0045] In the figure, 401 is an acquisition module; 402 is a judgment module; 403 is a first image acquisition module to be stitched; 404 is a second image acquisition module to be stitched; 405 is a stitching module. DETAILED DESCRIPTION
[0046] In order to make the purpose, technical solutions and advantages of this application more clear, the following Figure 1-4 It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0047] Compared with cameras with low frame rates, cameras with high frame rates display more frames per second, which can produce smoother and more realistic animations. When shooting moving objects, cameras with high frame rates are less likely to have afterimages in each frame of video images than cameras with low frame rates. Therefore, when stitching images of moving objects, cameras with high frame rates are generally used to shoot. In the prior art, when performing image stitching, the more video images there are, the better the imaging effect of the stitched complete image. When performing image stitching, the inventor found that when using a camera with a higher frame rate to shoot video and perform image stitching, it is easy to have a long delay in obtaining a complete stitched image; for example, upgrading the original 30-frame camera to a 120-frame camera, there is a long delay in obtaining a complete stitched image, which is difficult to meet the needs of real-time processing.
[0048] The present application embodiment discloses an image stitching method. Figure 1 As an implementation of an image stitching method, the method includes the following steps:
[0049] Step 101: Acquire a frame of video image containing a moving object.
[0050] Specifically, the video containing the moving object is obtained by shooting the moving object entering the shooting area of the camera. Each frame of the video image is obtained in sequence according to the arrangement order of the frames. The video image can be obtained by the processor directly obtaining the video containing the moving object, and then obtaining several frames of video images from the video; or a third-party processor can first obtain the above video, divide the above video into frames, and then send the video images to the processor in sequence.
[0051] Step 102: based on the frame interval between the currently acquired video image and the last video image for target tracking, determine whether target tracking needs to be performed on the currently acquired video image; if not, execute step 103; if yes, execute step 104.
[0052] Specifically, whether the frame interval between the currently acquired video image and the last video image for target tracking is greater than the preset frame number is used to determine whether the target tracking needs to be performed on the currently acquired video image. The preset frame number can be adjusted according to the frame rate of the camera; it only needs to meet the requirement that the position of the moving object does not change significantly within the preset frame number. Usually, for a 120-frame camera, the preset frame number does not exceed 5 frames.
[0053] Step 103: obtaining a new image to be stitched based on the image to be stitched obtained by the last target tracking.
[0054] Specifically, if the frame number interval between the current video image and the video image for which target tracking was performed last time does not exceed the preset frame number, target tracking does not need to be performed on the current video image to obtain its corresponding image to be spliced.
[0055] Step 104: performing target tracking on the currently acquired video image to obtain a new image to be stitched.
[0056] Specifically, if the frame number interval between the current video image and the video image for which target tracking was performed last time is greater than a preset frame number, target tracking is performed on the video image to obtain an image to be spliced.
[0057] Step 105 , stitching the obtained images to be stitched; after all the images to be stitched are stitched, a complete stitched image of the moving object is obtained.
[0058] Specifically, all the images to be stitched are stitched in sequence, and after all the images to be stitched are stitched, a complete stitched image of the moving object is obtained.
[0059] Compared with cameras with lower frame rates, cameras with higher frame rates produce clearer images and more accurate stitched images. For example, a 60-frame camera and a 120-frame camera, each frame of video image shot by a 120-frame camera is less likely to have residual images. Assuming that a moving object moves at a constant speed, the distance moved by the moving object in the video image shot by a 60-frame camera is recorded as A, and the distance moved by the moving object in the video image shot by a 120-frame camera is recorded as B, where A is twice B. Using a camera with a higher frame rate, the width of video image tracking is more accurate, and the stitched video image is more accurate. When performing image stitching, target tracking takes the longest time in the entire algorithm. In this application, if a frame of video image is tracked, the video images of the preset number of frames behind the frame of video image can be processed with the previous tracking results. After the preset number of frames, the video image is reselected for target tracking, and the re-tracking results continue to be used in the processing of the video images of the preset number of frames behind. Therefore, it is not necessary to track the target for each frame, which reduces the number of target tracking and improves the speed of stitching. The technical solution of the present application is used with a high frame rate camera (frame rate greater than 60), which can not only make the complete stitching image more accurate, but also improve the processing efficiency.
[0060] In one specific implementation manner, step 103, obtaining a new image to be stitched based on the image to be stitched obtained by the last target tracking, includes: obtaining a previous pixel displacement based on the result of the last target tracking; and obtaining a new image to be stitched based on the last image to be stitched and the last pixel displacement.
[0061] Specifically, if the previous frame of the current video image has been subjected to target tracking, the previous pixel displacement is obtained based on the result of the previous target tracking, and the size of the pixel displacement is recorded as N. The image to be spliced obtained last time is used as a reference, and the center line of the image to be spliced obtained last time is used as the starting line to intercept the image with a pixel displacement of N to obtain the current image to be spliced. If the previous frame of the current video image has not been subjected to target tracking, the image is still intercepted based on the pixel displacement obtained from the previous target tracking, and the center line of the image to be spliced obtained last time is used as the starting line to intercept the image with a pixel displacement of N to obtain the current image to be spliced.
[0062] The following is an example. Assuming that the preset number of frames is two, the first frame of video image is used for target tracking to obtain the image to be spliced and the pixel displacement corresponding to the first frame of video image. At this time, the pixel displacement size is M. When processing the second frame of video image, there is no need to track the target again. The center line of the image to be spliced corresponding to the first frame of video image is directly used as the starting line, and the image with a pixel displacement of M is intercepted to obtain the image to be spliced of the second frame. When processing the third frame of video image, the center line of the image to be spliced corresponding to the first frame of video image is used as the starting line, and the image with a pixel displacement of M is intercepted to obtain the image to be spliced of the third frame. When processing the fourth frame of video, since the frame interval with the video image (i.e., the first frame of video image) for the last target tracking is greater than the preset number of frames (two frames), it is necessary to track the target of the fourth frame of video image.
[0063] Reference Figure 2 , stitching the obtained images to be stitched includes the following steps:
[0064] Step 201: Save the obtained image to be stitched.
[0065] Specifically, after the image to be spliced corresponding to one of the frames of the video image is obtained, the image to be spliced is saved in the splicing queue according to the arrangement order of the frames.
[0066] Step 202: After all the images to be stitched are obtained, the sum of the memory occupied by all the images to be stitched is calculated to obtain the total occupied memory.
[0067] Specifically, if the processor cannot obtain a new video image, and the last frame of the video image obtained has obtained the corresponding image to be spliced, then the processor determines that all the images to be spliced have been obtained. At this time, the processor calculates the sum of the memory occupied by all the images to be spliced to obtain the total occupied memory.
[0068] Step 203: Apply for memory space based on the total occupied memory; the applied memory space is not less than the total occupied memory.
[0069] Step 204 : copy the images to be stitched into the memory space in sequence according to the order of the frames, and perform image stitching.
[0070] Specifically, in the prior art, when image stitching is performed, the images to be stitched are obtained and then stitching is performed. Each time stitching requires a memory application. For example, for 50 images to be stitched, each image to be stitched occupies n memory. The first memory application is n, the second memory application is 2n, and the 50th memory application is 50n. The total memory size occupied is 1275n. This processing method, on the one hand, occupies a large amount of memory, and on the other hand, is time-consuming. In the present application, after obtaining each image to be stitched, only the video image data of the image to be stitched is saved, and stitching is not performed directly. After all the images to be stitched are obtained, image stitching is performed. For the same 50 video images to be stitched, the present application only occupies a total memory size of 50n, so there is no need to apply for memory every time stitching. Since applying for memory takes time, the stitching efficiency can also be greatly improved.
[0071] Reference Figure 3 , the target tracking of the currently acquired video image is performed to obtain a new image to be stitched, including:
[0072] Step 301: set a tracking area for a moving object in a video image, and record the position coordinates of the tracking area;
[0073] Step 302: Use a tracking algorithm to track the target in the tracking area to obtain the pixel displacement of the moving object in the current video image and the previous video image; and
[0074] Step 303: Obtain a displacement video image corresponding to the pixel displacement of the moving object in the current video image, and use the displacement video image as the image to be stitched.
[0075] As another implementation of the image stitching method, before setting the tracking area of the moving object in the video image, it also includes: performing image enhancement processing on the video image, and the image enhancement processing includes adaptive equalization.
[0076] Specifically, the imaging effect of the camera is greatly affected by light. After the acquired video image is adaptively equalized, the brightness of the picture can be automatically balanced. The over-exposure or under-exposure of the image can also be repaired to make the details of the moving object more prominent and improve the accuracy of the target tracking algorithm.
[0077] As another implementation of an image stitching method, a video containing a moving object is obtained by capturing video images through a 1.44 mm fisheye lens.
[0078] Specifically, the fisheye lens is a special lens among the ultra-wide-angle lenses. The fisheye lens has a wide viewing angle range, which can generally reach 220° or 230°. The 1.44mm fisheye lens creates conditions for close-up photography of a wide range of scenes.
[0079] When a fisheye lens is used for shooting, after obtaining a frame of video image containing a moving object, the method further includes: performing distortion correction on the video image.
[0080] Specifically, some fisheye lenses have built-in distortion correction function, so this step can be omitted; if the selected fisheye lens does not have distortion correction function, the camera internal parameter matrix and distortion correction parameters can be used to correct the distortion of the acquired video image. Distortion correction can improve the accuracy of the tracking algorithm and make the stitched image smoother and more normal in proportion.
[0081] The present application also provides an image stitching system, comprising:
[0082] An acquisition module 401 is used to acquire a frame of video image of a video containing a moving object;
[0083] A judgment module 402 judges whether it is necessary to perform target tracking on the currently acquired video image based on a frame interval between the currently acquired video image and the video image last used for target tracking;
[0084] A first image to be stitched acquisition module 403, which acquires a new image to be stitched based on the image to be stitched obtained by the last target tracking when there is no need to perform target tracking on the currently acquired video image;
[0085] The second image to be stitched acquisition module 404 performs target tracking on the currently acquired video image to obtain a new image to be stitched when target tracking is required for the currently acquired video image; and
[0086] The stitching module 405 is used to stitch the obtained images to be stitched. After all the images to be stitched are stitched, a complete stitched image of the moving object is obtained.
[0087] The application embodiment also discloses a computer device.
[0088] Specifically, the device includes a memory and a server, and the memory stores a computer program that can be loaded by the server and execute any one of the above-mentioned image stitching methods.
[0089] The embodiment of the present application also discloses a computer-readable storage medium.
[0090] Specifically, the computer-readable storage medium stores a computer program that can be loaded by the server and execute any one of the above-mentioned image stitching methods. The computer-readable storage medium includes, for example: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.
[0091] The above are all preferred embodiments of the present application, and are not intended to limit the protection scope of the present application. Any feature disclosed in this specification (including the abstract and drawings), unless otherwise stated, can be replaced by other equivalent or alternative features with similar purposes. That is, unless otherwise stated, each feature is only an example of a series of equivalent or similar features.
Claims
1. An image stitching method, characterized in that: include: Acquire a frame of video image of a video containing a moving object; Determining whether it is necessary to perform target tracking on the currently acquired video image according to whether the frame number interval between the currently acquired video image and the last video image for target tracking is greater than a preset frame number; The preset number of frames is adjusted according to the frame rate of the camera; if not, a new image to be stitched is obtained based on the image to be stitched obtained by the last target tracking, and it is not necessary to perform target tracking on the frame image, so as to reduce the number of target tracking times and improve the stitching speed; in the case of, Then, target tracking is performed on the currently acquired video image to obtain a new image to be stitched; and, Stitching the obtained images to be stitched; after stitching all the images to be stitched, a complete stitched image of the moving object is obtained; A new image to be stitched is obtained based on the image to be stitched obtained by the last target tracking, including: The last pixel displacement is obtained based on the result of the last target tracking; as well as, Obtaining a new image to be stitched based on the last image to be stitched and the last pixel displacement; Specifically, if the previous frame of the current video image performed target tracking, the previous pixel displacement is obtained based on the result of the previous target tracking, and the size of the pixel displacement is recorded as N. The image to be spliced obtained last time is used as a reference, and the center line of the image to be spliced obtained last time is used as the starting line to intercept the image with a pixel displacement of N to obtain the current image to be spliced; if the previous frame of the current video image did not perform target tracking, the image is still captured based on the pixel displacement obtained from the previous target tracking, and the center line of the image to be spliced obtained last time is used as the starting line to intercept the image with a pixel displacement of N to obtain the current image to be spliced.
2. The image stitching method according to claim 1, characterized in that: Stitching the obtained images to be stitched includes: Saving the obtained image to be stitched; After all the images to be stitched are obtained, the sum of the memory occupied by all the images to be stitched is calculated to obtain the total occupied memory; Applying for memory space based on the total occupied memory; the applied memory space is not less than the total occupied memory; The images to be stitched are copied to the memory space in sequence according to the order of frames to perform image stitching.
3. The image stitching method according to claim 1, characterized in that: The step of performing target tracking on the currently acquired video image to obtain a new image to be stitched includes: Set the tracking area of the moving object in the video image and record the position coordinates of the tracking area; Using a tracking algorithm to track the target in the tracking area, obtaining the pixel displacement of the moving object in the current video image and the previous video image; and A displacement video image corresponding to the pixel displacement of the moving object in the current video image is obtained, and the displacement video image is used as the image to be spliced.
4. The image stitching method according to claim 3, characterized in that: Before setting the tracking area of the moving object in the video image, the method further includes: performing image enhancement processing on the video image, wherein the image enhancement processing includes adaptive equalization.
5. The image stitching method according to claim 1, characterized in that: The video containing the moving object is obtained by capturing video images through a 1.44 mm fisheye lens.
6. The image stitching method according to claim 5, characterized in that: After acquiring a frame of video image of a video containing a moving object, the method further includes: performing distortion correction on the video image.
7. An image stitching system, characterized in that: include: An acquisition module (401) is used to acquire a frame of video image of a video containing a moving object; A judgment module (402) is configured to judge whether it is necessary to perform target tracking on the currently acquired video image according to whether the frame number interval between the currently acquired video image and the video image for which target tracking was performed last time is greater than a preset frame number; The preset number of frames is adjusted according to the frame rate of the camera; A first image to be spliced acquisition module (403) is used to acquire a new image to be spliced based on the image to be spliced acquired by the last target tracking when there is no need to perform target tracking on the currently acquired video image, so as to reduce the number of target tracking times and improve the splicing speed; a second image acquisition module (404) for performing target tracking on the currently acquired video image to obtain a new image to be stitched when it is necessary to perform target tracking on the currently acquired video image; and a stitching module (405) for stitching the acquired images to be stitched; after stitching all the images to be stitched, a complete stitched image of the moving object is obtained; A new image to be stitched is obtained based on the image to be stitched obtained by the last target tracking, including: The last pixel displacement is obtained based on the result of the last target tracking; as well as, Obtaining a new image to be stitched based on the last image to be stitched and the last pixel displacement; Specifically, if the previous frame of the current video image performed target tracking, the previous pixel displacement is obtained based on the result of the previous target tracking, and the size of the pixel displacement is recorded as N. The image to be spliced obtained last time is used as a reference, and the center line of the image to be spliced obtained last time is used as the starting line to intercept the image with a pixel displacement of N to obtain the current image to be spliced; if the previous frame of the current video image did not perform target tracking, the image is still captured based on the pixel displacement obtained from the previous target tracking, and the center line of the image to be spliced obtained last time is used as the starting line to intercept the image with a pixel displacement of N to obtain the current image to be spliced.
8. A computer device, characterized in that: The method comprises a memory and a server, wherein the memory stores a computer program according to any one of the methods of claims 1 to 6 which is loaded and executed by the server.
9. A computer-readable storage medium, characterized in that: A computer program is stored which can be loaded by a server and execute the method according to any one of claims 1 to 6.
