Multi-processor parallel image stitching method and system for panoramic display

Abstract

The application relates to the technical field of panoramic image splicing, and discloses a multi-processor parallel image splicing method and system for panoramic display, which comprises the following steps: acquiring an original image sequence, and generating a pre-analysis image sequence based on the original image sequence; dividing the original image sequence into continuous display task blocks based on the pre-analysis image sequence, and calculating the predicted load of each display task block; acquiring the load state index of each processor, and respectively distributing each display task block to a processor based on the predicted load and the load state index; acquiring the local splicing image of each display task block and performing fusion to generate a panoramic image; and performing constraint correction on the local splicing images of adjacent display task blocks in the panoramic image. The application can improve the continuity and display quality of the panoramic image while improving the multi-processor parallel splicing efficiency.

Description

Technical Field

[0001] This application relates to the technical field of panoramic image stitching, specifically to a multiprocessor parallel image stitching method and system for panoramic display. Background Technology

[0002] Existing panoramic image stitching technologies typically possess mature basic processing capabilities such as feature extraction, image matching, geometric transformation, and fusion output, enabling them to complete relatively stable stitching tasks under conditions of small static data volumes or sufficient processing resources. However, as the scale of input image sequences increases and processing platforms evolve from single-processing units to multi-processing units, existing technologies still face some challenges in application.

[0003] Existing solutions, when dealing with continuous image sequences, often focus more on the accuracy of a single stitch, while neglecting task organization, load balancing, and overall throughput. When there are many images, high resolution, and complex overlap between adjacent images, the computational load between different processing units often varies significantly. This can easily lead to some processing units being overloaded while others remain idle, thus affecting overall processing efficiency and making it difficult to meet the needs of real-time display or high-concurrency processing scenarios. On the other hand, while existing technologies can complete local stitching during the synthesis of multi-segment image results, they are still insufficient in ensuring the continuity of the overall display and the consistency of boundaries. Especially when continuous scenes are processed in segments or when multiple processing units perform local stitching separately, problems such as local misalignment, obvious seams, and unnatural boundary transitions can easily occur in the final panoramic image.

[0004] The information disclosed in this background section is intended only to enhance the understanding of the overall background of this application and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0005] The technical problem to be solved by this application is to overcome the defects of the prior art and provide a multiprocessor parallel image stitching method and system for panoramic display, which improves the efficiency of multiprocessor parallel stitching while enhancing the continuity and display quality of panoramic images.

[0006] To solve the above-mentioned technical problems, this application provides the following technical solution: On the one hand, this application provides a multiprocessor parallel image stitching method for panoramic display, including the following steps: Obtain the original image sequence and generate a pre-analyzed image sequence based on the original image sequence; Based on the pre-analyzed image sequence, the original image sequence is divided into consecutive display task blocks, and the expected load of each display task block is calculated; Obtain the load status index of each processor, and allocate each display task block to a processor based on the expected load and the load status index; The local stitched images of each display task block are acquired and merged to generate a panoramic image; Constraint correction is performed on the local stitched images of adjacent display task blocks in the panoramic image.

[0007] As a preferred embodiment of the multiprocessor parallel image stitching method for panoramic display described in this application, the original image sequence includes original images sorted from early to late according to the acquisition time; The pre-analyzed image sequence includes pre-analyzed images that correspond one-to-one with the original images in the original image sequence; the generation of the pre-analyzed image sequence based on the original image sequence includes: performing downsampling processing on each original image in the original image sequence to obtain the pre-analyzed image corresponding to each original image; arranging the pre-analyzed images into a pre-analyzed image sequence according to the order of the corresponding original image acquisition time from early to late.

[0008] As a preferred embodiment of the multiprocessor parallel image stitching method for panoramic display described in this application, the method of dividing the original image sequence into consecutive display task blocks includes: Starting from the second pre-analyzed image in the pre-analyzed image sequence, it is determined in turn whether each pre-analyzed image has a stable adjacency relationship with its reference image. If it does, the original image corresponding to the pre-analyzed image is marked as a stable original image. The reference image of any pre-analyzed image is the previous pre-analyzed image that is adjacent to it in the pre-analyzed image sequence. Mark the first original image in the original image sequence as a task starting image; traverse each original image in the order recorded in the original image sequence. For any original image, if it is at least M original images away from the nearest task starting image, or if it has not been marked as a stable original image, then mark it as a task starting image; M is a positive integer. The labeled task start image divides the original image sequence into m consecutive display task blocks; each display task block is a sequence segment in the original image sequence, and its starting point is a task start image; m is the number of task start images.

[0009] As a preferred embodiment of the multiprocessor parallel image stitching method for panoramic display described in this application, the method for determining whether any pre-analyzed image has a stable adjacency relationship with its reference image includes: Label any pre-analyzed image as the target image; Feature point detection and matching are performed in the target image and the reference image respectively; the successfully matched feature points in the target image and the reference image are marked as valid feature points respectively. Calculate the overlap area ratio and overlap distribution ratio between the reference image and the target image based on effective feature points; If the overlap area ratio is greater than a preset overlap area threshold and the overlap distribution ratio is greater than a preset overlap distribution threshold, then the target image and its reference image have a stable adjacency relationship.

[0010] As a preferred embodiment of the multiprocessor parallel image stitching method for panoramic display described in this application, the method for calculating the overlap area ratio includes: calculating the matching coverage ratio of the target image and the matching coverage ratio of the reference image respectively; taking the average of the matching coverage ratio of the target image and the matching coverage ratio of the reference image as the overlap area ratio; wherein, the method for calculating the matching coverage ratio of the target image includes: calculating the area of ​​the minimum bounding rectangle of all effective feature points in the target image, and dividing it by the total area of ​​the target image as the matching coverage ratio of the target image; The method for calculating the overlap distribution ratio includes: calculating the matching distribution ratio of the target image and the matching distribution ratio of the reference image respectively; taking the average of the matching distribution ratio of the target image and the matching distribution ratio of the reference image as the overlap distribution ratio; wherein, the method for calculating the matching distribution ratio of the target image includes: dividing the target image into uniform grids; if there is at least one valid feature point in any grid, then the corresponding grid is marked as an occupied grid; calculating the ratio of the number of occupied grids to the total number of grids as the matching distribution ratio of the target image.

[0011] As a preferred embodiment of the multiprocessor parallel image stitching method for panoramic display described in this application, wherein: the local stitched image of any display task block is an image generated by the processor based on the fusion of all original images in the display task block; the adjacent display task blocks include any two display task blocks that are adjacent in the original image sequence; The constraint correction of the local stitched image of adjacent display task blocks in the panoramic image includes filtering constraint correction objects, specifically including: Obtain the initial sampling time for each display task block; the initial sampling time for any display task block is the acquisition time of the corresponding task's starting image. The display task block with the earliest initial sampling time among adjacent display task blocks is marked as the first display task block; the display task block with the latest initial sampling time among adjacent display task blocks is marked as the second display task block. The original image with the latest sampling time in the first display task block is marked as the first boundary image; the task start image in the second display task block is marked as the second boundary image. If the second boundary image is a stable original image, then the constraint correction object includes the local stitched image of the first display task block and the local stitched image of the second boundary image; otherwise, there is no constraint correction object.

[0012] As a preferred embodiment of the multiprocessor parallel image stitching method for panoramic display described in this application, the constraint correction of the local stitched image of adjacent display task blocks in the panoramic image further includes correction of the constraint correction object, specifically including: The pre-analyzed image corresponding to the first boundary image is labeled as the first analysis image; the pre-analyzed image corresponding to the second boundary image is labeled as the second analysis image; When the first analysis image is used as a reference image for the second analysis image, obtain the effective feature points in the first analysis image and the second analysis image; Based on the effective feature points in the first and second analysis images, calculate the boundary correction amount; Based on the boundary correction amount, coordinate corrections are performed on the local stitched images of the first display task block and the local stitched images of the second boundary image in the panoramic image.

[0013] As a preferred embodiment of the multiprocessor parallel image stitching method for panoramic display described in this application, the boundary correction amount includes a horizontal axis correction amount and a vertical axis correction amount; the method for calculating the boundary correction amount includes: marking any valid feature point in the first analysis image as a first feature point; marking a valid feature point in the second analysis image that successfully matches the first feature point as a second feature point; Obtain the x-coordinate of the first feature point mapped to the panoramic image, and denote it as the first x-coordinate; obtain the x-coordinate of the second feature point mapped to the panoramic image, and denote it as the second x-coordinate; Calculate the difference between the first and second horizontal coordinates, and use it as the horizontal axis offset of the first feature point; Calculate the mean of the horizontal axis offset of all valid feature points in the first analysis image, and multiply it by 0.5 to obtain the horizontal axis correction amount; Obtain the ordinate of the first feature point after mapping it to the panoramic image, and denote it as the first ordinate; obtain the ordinate of the second feature point after mapping it to the panoramic image, and denote it as the second ordinate; Calculate the difference between the first and second ordinates, and use it as the ordinate offset of the first feature point; Calculate the mean of the vertical axis offset of all valid feature points in the first analysis image, and multiply it by 0.5 to obtain the vertical axis correction amount.

[0014] As a preferred embodiment of the multiprocessor parallel image stitching method for panoramic display described in this application, wherein: based on the boundary correction amount, coordinate correction is performed on the local stitched image of the first display task block and the local stitched image of the second boundary image in the panoramic image, specifically including: The region in the panoramic image corresponding to the first boundary image is marked as the first correction region; the region in the panoramic image corresponding to the second boundary image is marked as the second correction region; For any pixel in the first correction region, subtract the horizontal axis correction amount from its horizontal coordinate and subtract the vertical axis correction amount from its vertical coordinate. For any pixel in the second correction region, add the horizontal axis correction amount to its horizontal coordinate and add the vertical axis correction amount to its vertical coordinate.

[0015] Secondly, this application provides a multiprocessor parallel image stitching system for panoramic display, including a data acquisition module, a task division module, a task allocation module, an image fusion module, and a stitching correction module; wherein: The data acquisition module is used to acquire the raw image sequence and the load status indicators of each processor; The task partitioning module generates a pre-analyzed image sequence based on the original image sequence, and then divides the original image sequence into continuous display task blocks based on the pre-analyzed image sequence; The task partitioning module is also used to calculate the expected load of each display task block; the task allocation module allocates each display task block to a processor based on the expected load and load status indicators, and collects the local stitched image of each display task block. The image fusion module is used to fuse the local stitched images of each display task block to generate a panoramic image; The stitching correction module is used to perform constraint correction on the local stitched images of adjacent display task blocks in the panoramic image.

[0016] Compared with the prior art, the beneficial effects achieved by this application are as follows: This application divides the original image sequence into consecutive display task blocks and allocates tasks based on the load status of each processor. This makes the multi-processor parallel processing more balanced, reducing situations where some processors are overloaded while others are idle, thereby improving overall throughput and execution efficiency. The display task blocks are not simply divided into a fixed number, but rather based on the stable adjacency relationship between images and task length constraints. Therefore, it can ensure a relatively stable stitching relationship between images within a block, while avoiding excessively long local tasks and heavy processing burdens, which is conducive to the stable implementation of subsequent parallel stitching.

[0017] By constraining and correcting the local stitched images of adjacent display task blocks in the panoramic image, the corresponding scene structures on both sides of the boundary can be further aligned in spatial position, thereby reducing problems such as boundary misalignment and visible seams, and improving the continuity and display effect of the panoramic image. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein: Figure 1 A flowchart of the multiprocessor parallel image stitching method for panoramic display provided in this application; Figure 2 This is a schematic diagram of the structure of the multiprocessor parallel image stitching system for panoramic display provided in this application. Detailed Implementation

[0019] The technical solution of this application will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments and specific features in the embodiments are detailed descriptions of the technical solution of this application, rather than limitations thereof. In the absence of conflict, the embodiments and technical features in the embodiments can be combined with each other.

[0020] Example 1 This embodiment describes a multiprocessor parallel image stitching method for panoramic display, referring to... Figure 1 The method includes the following steps: Obtain the original image sequence and generate a pre-analyzed image sequence based on the original image sequence; The original image sequence includes original images sorted from earliest to latest according to the acquisition time; In a panoramic display scenario, the original image sequence is a set of original images to be stitched together. Different original images are records of the same continuous scene from different perspectives or different spatial positions, and there is some overlapping scene content between adjacent original images so that the original images can be stitched together into a complete panoramic image.

[0021] The pre-analyzed image sequence includes pre-analyzed images that correspond one-to-one with the original images in the original image sequence; the generation of the pre-analyzed image sequence based on the original image sequence includes: performing downsampling processing on each original image in the original image sequence to obtain the pre-analyzed image corresponding to each original image; arranging the pre-analyzed images into a pre-analyzed image sequence according to the order of the corresponding original image acquisition time from early to late.

[0022] In this scheme, the pre-analysis image is a low-resolution image obtained by downsampling the corresponding original image. For example, if the original image has a resolution of 4K, a pre-analysis image with a resolution of 2K or lower can be obtained through downsampling. Downsampling refers to reducing the pixel resolution of the original image according to a preset ratio, reducing the number of pixels while maintaining the overall structure and main texture distribution features of the original scene. Downsampling can be implemented using existing image scaling methods, such as any of the following: nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, etc. After forming the pre-analysis image through downsampling, the computational overhead of subsequent feature extraction, image matching, and other analyses can be reduced, thereby improving the efficiency of the scheme.

[0023] Based on the pre-analyzed image sequence, the original image sequence is divided into consecutive display task blocks, and the expected load of each display task block is calculated; The method for dividing the original image sequence into consecutive display task blocks includes: Starting from the second pre-analyzed image in the pre-analyzed image sequence, it is determined in turn whether each pre-analyzed image has a stable adjacency relationship with its reference image. If it does, the original image corresponding to the pre-analyzed image is marked as a stable original image. The reference image of any pre-analyzed image is the previous pre-analyzed image that is adjacent to it in the pre-analyzed image sequence. Mark the first original image in the original image sequence as a task starting image; traverse each original image in the order recorded in the original image sequence. For any original image, if it is at least M original images away from the nearest task starting image, or if it has not been marked as a stable original image, then mark it as a task starting image; M is a positive integer. The labeled task start image divides the original image sequence into m consecutive display task blocks; each display task block is a sequence segment in the original image sequence, and its starting point is a task start image; m is the number of task start images.

[0024] Dividing display task blocks in the above manner ensures a stable stitching relationship within each block and constrains the block length, preventing excessive local workloads and facilitating subsequent multiprocessor parallel processing. Optionally, M can be set to 5, meaning each display task block contains at most 6 consecutive original images. This allows the processor to balance parallel processing efficiency with the computational overhead of constraint correction between adjacent display task blocks when processing each block.

[0025] Methods for determining whether a pre-analyzed image has a stable adjacency relationship with its reference image include: Label any pre-analyzed image as the target image; Feature point detection and matching are performed in the target image and the reference image respectively; the successfully matched feature points in the target image and the reference image are marked as valid feature points respectively. Optionally, feature extraction algorithms such as SIFT, SURF, or ORB are used to obtain feature points in the target image and the reference image; then, K-nearest neighbor matching or other methods can be used to match the feature points in the target image and the reference image to obtain successfully matched feature point pairs; any feature point pair contains one valid feature point from the target image and one from the reference image.

[0026] Calculate the overlap area ratio and overlap distribution ratio between the reference image and the target image based on effective feature points; If the overlap area ratio is greater than a preset overlap area threshold and the overlap distribution ratio is greater than a preset overlap distribution threshold, then the target image and its reference image have a stable adjacency relationship.

[0027] The method for calculating the overlap area ratio includes: calculating the matching coverage ratio of the target image and the matching coverage ratio of the reference image respectively; taking the average of the matching coverage ratio of the target image and the matching coverage ratio of the reference image as the overlap area ratio; wherein, the method for calculating the matching coverage ratio of the target image includes: calculating the area of ​​the minimum bounding rectangle of all effective feature points in the target image, and dividing it by the total area of ​​the target image as the matching coverage ratio of the target image. Similarly, the method for calculating the matching coverage ratio of the reference image includes: calculating the area of ​​the smallest bounding rectangle of all valid feature points in the reference image, and dividing it by the total area of ​​the reference image to obtain its matching coverage ratio; The method for calculating the overlap distribution ratio includes: calculating the matching distribution ratio of the target image and the matching distribution ratio of the reference image respectively; taking the average of the matching distribution ratio of the target image and the matching distribution ratio of the reference image as the overlap distribution ratio; wherein, the method for calculating the matching distribution ratio of the target image includes: dividing the target image into uniform grids; if there is at least one valid feature point in any grid, then the corresponding grid is marked as an occupied grid; calculating the ratio of the number of occupied grids to the total number of grids as the matching distribution ratio of the target image.

[0028] Similarly, the method for calculating the matching distribution ratio of the reference image includes: dividing the reference image into a uniform grid; if there is at least one valid feature point in any grid, then the corresponding grid is marked as an occupied grid; calculating the ratio of the number of occupied grids to the total number of grids, as the matching distribution ratio of the reference image. Optionally, for a target image or its reference image with a resolution of 4K (i.e., 3840×2160 pixels), it is uniformly divided into 16×16 rectangular grids to balance the accuracy and simplicity of the matching distribution ratio calculation.

[0029] Those skilled in the art can set specific values ​​for the overlap area threshold and overlap distribution threshold based on actual needs. For example, the overlap area threshold can be set to 0.2, and the overlap distribution threshold can be set to 0.3. The overlap area threshold determines whether there is a sufficiently large effective common area between two images to support subsequent stitching. If the target image and its reference image have a stable adjacency relationship, then the two images have an average of at least 20% effective common area, which is sufficient to provide a relatively stable common area for feature matching and geometric transformation. The overlap distribution threshold determines whether the effective feature points have a sufficiently dispersed distribution, rather than being concentrated only in small local areas. If the target image and its reference image have a stable adjacency relationship, it indicates that at least 30% of the grid contains effective feature points, which is sufficient to show that the matching relationship between the target image and its reference image covers multiple locations and can robustly support the determination of a stable adjacency relationship.

[0030] Methods for calculating the expected load of any display task block include: Calculate the load of each original image separately; calculate the sum of the expected loads of all original images in the display task block as the expected load of the display task block; wherein, the load of any original image is the product of its pixel area and the matching coverage ratio of the corresponding pre-analyzed image.

[0031] For any original image, the higher the resolution and the larger the pixel area, the greater the computational load of subsequent processing; the greater the matching coverage ratio of the corresponding pre-analyzed image, the larger the overlapping area between the original image and the adjacent original images, and the more associated calculations are required for subsequent feature detection and matching, geometric transformation and image fusion, and the greater the workload.

[0032] Obtain the load status index of each processor, and allocate each display task block to a processor based on the expected load and the load status index; The load status metrics for any processor include remaining utilization and task queue length; where the remaining utilization of any processor is 1 minus its real-time utilization; and the task queue length of any processor is the number of display task blocks that the processor is currently processing. Based on the estimated load and load status indicators, each display task block is assigned to a processor, specifically including: S10: Construct a display task queue; the display task queue includes each display task block, and the display task blocks are arranged in descending order of expected load. S20: Remove the first display task block from the display task queue and assign it to the target processor; The target processor is the processor with the highest remaining utilization rate; if there is more than one processor with the highest remaining utilization rate, then the target processor is the processor with the shortest task queue length among the processors with the highest remaining utilization rate. S30: Update the load status metrics for each processor; S40: Repeat S20 to S30 until the task queue is empty.

[0033] The local stitched images of each display task block are acquired and merged to generate a panoramic image; The local stitched image of any display task block is an image generated by the processor based on the fusion of all original images in the display task block; in this embodiment, any local stitched image is a local segment of a panoramic image. The processing of any display task block by any processor may include a feature extraction stage, an image matching stage, a geometric transformation stage, and a fusion output stage. In the feature extraction stage, the processor extracts feature points that can characterize local textures, corners, or edge structures from each original image within the display task block, for example, using SIFT, SURF, or ORB feature extraction algorithms. In the image matching stage, the processor uses nearest neighbor search or K-nearest neighbor matching methods to match feature points between any two adjacent original images, outputting a set of effective feature points between any two adjacent original images. In the geometric transformation stage, the processor calculates the spatial transformation relationship between any two adjacent original images based on the set of effective feature points between any two adjacent original images, and maps all original images to a unified local coordinate system. For example, affine transformation models, perspective transformation models, and homography matrix estimation methods can be used to calculate the geometric transformation parameters of each original image within the display task block, and further perform coordinate transformation on each original image, transforming the coordinates of pixels in the original images to the local coordinate system of the locally stitched image. The coordinate system of the first original image in the display task block can be selected as the local coordinate system. In the fusion output stage, the processor performs brightness, color, and boundary transition processing on the overlapping areas of different original images in the local coordinate system to reduce the stitching seams and form a visually continuous locally stitched image. Fusion algorithms that can be used include weighted average fusion methods and feathering fusion methods.

[0034] After generating the local stitched image for each display task block, the different local stitched images undergo the same feature extraction, image matching, geometric transformation, and fusion output stages to output a panoramic image. The geometric transformation stage maps the local coordinate systems of the different local stitched images to the global coordinate system for further coordinate alignment, thereby achieving panoramic image fusion. For example, the local coordinate system corresponding to the local stitched image of the first display task block can be selected as the global coordinate system.

[0035] Constraint correction is performed on the local stitched images of adjacent display task blocks in the panoramic image.

[0036] The adjacent display task blocks include any two adjacent display task blocks in the original image sequence; The constraint correction of the local stitched image of adjacent display task blocks in the panoramic image includes filtering constraint correction objects, specifically including: Obtain the initial sampling time for each display task block; the initial sampling time for any display task block is the acquisition time of the corresponding task's starting image. The display task block with the earliest initial sampling time among adjacent display task blocks is marked as the first display task block; the display task block with the latest initial sampling time among adjacent display task blocks is marked as the second display task block. The original image with the latest sampling time in the first display task block is marked as the first boundary image; the task start image in the second display task block is marked as the second boundary image. If the second boundary image is a stable original image, then the constraint correction object includes the local stitched image of the first display task block and the local stitched image of the second boundary image; otherwise, there is no constraint correction object.

[0037] In this embodiment, for any group of adjacent display task blocks, their first boundary image and second boundary image are adjacent in the original image sequence. When the second boundary image is a stable original image, it indicates that there is a stable adjacency relationship between the two; otherwise, the two can be regarded as natural boundaries of different areas of the scene, and there is no need to perform constraint correction to prevent the introduction of unnecessary correction errors.

[0038] The constraint correction of the local stitched image of adjacent display task blocks in the panoramic image also includes the correction of the constraint correction object, specifically including: The pre-analyzed image corresponding to the first boundary image is labeled as the first analysis image; the pre-analyzed image corresponding to the second boundary image is labeled as the second analysis image; When the first analysis image is used as a reference image for the second analysis image, obtain the effective feature points in the first analysis image and the second analysis image; Based on the effective feature points in the first and second analysis images, calculate the boundary correction amount; Based on the boundary correction amount, coordinate corrections are performed on the local stitched images of the first display task block and the local stitched images of the second boundary image in the panoramic image.

[0039] The boundary correction amount includes a horizontal axis correction amount and a vertical axis correction amount; the method for calculating the boundary correction amount includes: marking any valid feature point in the first analysis image as a first feature point; marking a valid feature point in the second analysis image that successfully matches the first feature point as a second feature point; Obtain the x-coordinate of the first feature point mapped to the panoramic image, and denote it as the first x-coordinate; obtain the x-coordinate of the second feature point mapped to the panoramic image, and denote it as the second x-coordinate; Calculate the difference between the first and second horizontal coordinates, and use it as the horizontal axis offset of the first feature point; Calculate the mean of the horizontal axis offset of all valid feature points in the first analysis image, and multiply it by 0.5 to obtain the horizontal axis correction amount; Obtain the ordinate of the first feature point after mapping it to the panoramic image, and denote it as the first ordinate; obtain the ordinate of the second feature point after mapping it to the panoramic image, and denote it as the second ordinate; Calculate the difference between the first and second ordinates, and use it as the ordinate offset of the first feature point; Calculate the mean of the vertical axis offset of all valid feature points in the first analysis image, and multiply it by 0.5 to obtain the vertical axis correction amount; As described above in this embodiment, after different processors process different display task blocks and generate corresponding local stitched images, they further map all local stitched images to the same global coordinate system so that the local stitched images can ultimately participate in the stitching generation of the same panoramic image. Since the pre-analysis image is formed by downsampling the original image, the coordinates of the effective feature points in the pre-analysis image can be restored according to the downsampling ratio and mapped to their respective original images to obtain the coordinates of the effective feature points mapped to the original images. When the processor processes the display task block, the pixel coordinates in the original image undergo the coordinate transformation described above and are further mapped to coordinates in the local coordinate system. The coordinates in the local coordinate system are further mapped to coordinates in the global coordinate system, thereby obtaining the first abscissa and first ordinate of the first feature point and the second abscissa and second ordinate of the second feature point.

[0040] Based on the boundary correction amount, coordinate corrections are performed on the local stitched images of the first display task block and the second boundary image in the panoramic image, respectively, specifically including: The region in the panoramic image corresponding to the first boundary image is marked as the first correction region; the region in the panoramic image corresponding to the second boundary image is marked as the second correction region; For any pixel in the first correction region, subtract the horizontal axis correction amount from its horizontal coordinate and subtract the vertical axis correction amount from its vertical coordinate. For any pixel in the second correction region, add the horizontal axis correction amount to its horizontal coordinate and add the vertical axis correction amount to its vertical coordinate.

[0041] In this embodiment, the panoramic image is generated by fusing the original image with overlapping regions through coordinate mapping. Therefore, any original image is mapped to a region in the panoramic image; for example, the first boundary image is mapped to the first correction region in the panoramic image. Since each display task block is processed independently by different processors, the geometric transformations and fusion results within each block are formed in their respective local coordinate systems. At the connection points of the local stitched images corresponding to adjacent display task blocks, misalignment and visible seams can easily occur, such as inconsistent positions of the same scene structure on both sides of the boundary. This solution eliminates obvious misalignment by constraining and correcting the local stitched images of adjacent display task blocks in the panoramic image, ensuring that the corresponding scene structures between the local stitched images of adjacent display task blocks are spatially aligned.

[0042] Example 2 This embodiment is the second embodiment of this application; it is based on the same inventive concept as Embodiment 1, and refers to... Figure 2 This embodiment introduces a multiprocessor parallel image stitching system for panoramic display, including a data acquisition module, a task division module, a task allocation module, an image fusion module, and a stitching correction module; wherein: The data acquisition module is used to acquire the original image sequence and the load status indicators of each processor; wherein, the original image sequence includes original images sorted from early to late according to the acquisition time, different original images are recorded from different perspectives or different spatial positions for the same continuous scene, and there is some overlap between adjacent original images; the load status indicators include the remaining utilization rate of the processor and the task queue length.

[0043] The task partitioning module generates a pre-analyzed image sequence based on the original image sequence, and divides the original image sequence into continuous display task blocks based on the pre-analyzed image sequence; wherein, the pre-analyzed image sequence includes pre-analyzed images that correspond one-to-one with the original images in the original image sequence, and the pre-analyzed images are obtained by downsampling the corresponding original images.

[0044] The task partitioning module is also used to calculate the expected load of each display task block; the task allocation module allocates each display task block to a processor based on the expected load and load status indicators, and collects the local stitched image of each display task block; the task allocation module first constructs a display task queue sorted from largest to smallest according to the expected load, and then prioritizes the allocation of the display task blocks in the queue to the target processor with higher remaining utilization and shorter task queue length, and dynamically updates the load status indicators of each processor during the allocation process.

[0045] The image fusion module is used to fuse the local stitched images of each display task block to generate a panoramic image; it also maps different local stitched images to a global coordinate system and further fuses them to obtain a panoramic image.

[0046] The stitching correction module is used to constrain and correct the local stitched images of adjacent display task blocks in the panoramic image. It corrects the pixel coordinates within the panoramic image region corresponding to the original image with stable adjacency to align the corresponding scene structures between adjacent local stitched images in spatial position and reduce boundary misalignment.

[0047] The specific functional implementation of each of the above modules refers to the relevant content in the multiprocessor parallel image stitching method for panoramic display described in Embodiment 1, and will not be repeated here.

[0048] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0049] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of protection of this application, and these forms are all within the protection scope of this application.

Claims

1. A multiprocessor parallel image stitching method for panoramic display, characterized in that: Includes the following steps: Obtain the original image sequence and generate a pre-analyzed image sequence based on the original image sequence; Based on the pre-analyzed image sequence, the original image sequence is divided into consecutive display task blocks, and the expected load of each display task block is calculated; Obtain the load status index of each processor, and allocate each display task block to a processor based on the expected load and the load status index; The local stitched images of each display task block are acquired and merged to generate a panoramic image; Constraint correction is performed on the local stitched images of adjacent display task blocks in the panoramic image.

2. The multiprocessor parallel image stitching method for panoramic display as described in claim 1, characterized in that: The original image sequence includes original images sorted from earliest to latest according to the acquisition time; The pre-analyzed image sequence includes pre-analyzed images that correspond one-to-one with the original images in the original image sequence; the generation of the pre-analyzed image sequence based on the original image sequence includes: performing downsampling processing on each original image in the original image sequence to obtain the pre-analyzed image corresponding to each original image; The pre-analyzed images are arranged into a pre-analyzed image sequence according to the original image acquisition time from morning to evening.

3. The multiprocessor parallel image stitching method for panoramic display as described in claim 2, characterized in that: The method for dividing the original image sequence into consecutive display task blocks includes: Starting from the second pre-analyzed image in the pre-analyzed image sequence, it is determined in turn whether each pre-analyzed image has a stable adjacency relationship with its reference image. If it does, the original image corresponding to the pre-analyzed image is marked as a stable original image. The reference image of any pre-analyzed image is the previous pre-analyzed image that is adjacent to it in the pre-analyzed image sequence. Mark the first original image in the original image sequence as a task starting image; traverse each original image in the order recorded in the original image sequence. For any original image, if it is at least M original images away from the nearest task starting image, or if it has not been marked as a stable original image, then mark it as a task starting image; M is a positive integer. The labeled task start image divides the original image sequence into m consecutive display task blocks; each display task block is a sequence segment in the original image sequence, and its starting point is a task start image; m is the number of task start images.

4. The multiprocessor parallel image stitching method for panoramic display as described in claim 3, characterized in that: Methods for determining whether a pre-analyzed image has a stable adjacency relationship with its reference image include: Label any pre-analyzed image as the target image; Feature point detection and matching are performed in the target image and the reference image respectively; the successfully matched feature points in the target image and the reference image are marked as valid feature points respectively. Calculate the overlap area ratio and overlap distribution ratio between the reference image and the target image based on effective feature points; If the overlap area ratio is greater than a preset overlap area threshold and the overlap distribution ratio is greater than a preset overlap distribution threshold, then the target image and its reference image have a stable adjacency relationship.

5. The multiprocessor parallel image stitching method for panoramic display as described in claim 4, characterized in that: The method for calculating the overlap area ratio includes: calculating the matching coverage ratio of the target image and the matching coverage ratio of the reference image respectively; taking the average of the matching coverage ratio of the target image and the matching coverage ratio of the reference image as the overlap area ratio; wherein, the method for calculating the matching coverage ratio of the target image includes: calculating the area of ​​the minimum bounding rectangle of all effective feature points in the target image, and dividing it by the total area of ​​the target image as the matching coverage ratio of the target image. The method for calculating the overlap distribution ratio includes: calculating the matching distribution ratio of the target image and the matching distribution ratio of the reference image respectively; taking the average of the matching distribution ratio of the target image and the matching distribution ratio of the reference image as the overlap distribution ratio; wherein, the method for calculating the matching distribution ratio of the target image includes: dividing the target image into uniform grids; if there is at least one valid feature point in any grid, then the corresponding grid is marked as an occupied grid; calculating the ratio of the number of occupied grids to the total number of grids as the matching distribution ratio of the target image.

6. The multiprocessor parallel image stitching method for panoramic display as described in claim 5, characterized in that: The local stitched image of any display task block is an image generated by the processor based on the fusion of all the original images in the display task block; the adjacent display task blocks include any two display task blocks that are adjacent in the original image sequence; The constraint correction of the local stitched image of adjacent display task blocks in the panoramic image includes filtering constraint correction objects, specifically including: Obtain the initial sampling time for each display task block; the initial sampling time for any display task block is the acquisition time of the corresponding task's starting image. The display task block with the earliest initial sampling time among adjacent display task blocks is marked as the first display task block; the display task block with the latest initial sampling time among adjacent display task blocks is marked as the second display task block. The original image with the latest sampling time in the first display task block is marked as the first boundary image; the task start image in the second display task block is marked as the second boundary image. If the second boundary image is a stable original image, then the constraint correction object includes the local stitched image of the first display task block and the local stitched image of the second boundary image; otherwise, there is no constraint correction object.

7. The multiprocessor parallel image stitching method for panoramic display as described in claim 6, characterized in that: The constraint correction of the local stitched image of adjacent display task blocks in the panoramic image also includes the correction of the constraint correction object, specifically including: The pre-analyzed image corresponding to the first boundary image is labeled as the first analysis image; the pre-analyzed image corresponding to the second boundary image is labeled as the second analysis image; When the first analysis image is used as a reference image for the second analysis image, obtain the effective feature points in the first analysis image and the second analysis image; Based on the effective feature points in the first and second analysis images, calculate the boundary correction amount; Based on the boundary correction amount, coordinate corrections are performed on the local stitched images of the first display task block and the local stitched images of the second boundary image in the panoramic image.

8. The multiprocessor parallel image stitching method for panoramic display as described in claim 7, characterized in that: The boundary correction amount includes horizontal axis correction amount and vertical axis correction amount; The method for calculating the boundary correction includes: marking any valid feature point in the first analysis image as a first feature point; and marking a valid feature point in the second analysis image that successfully matches the first feature point as a second feature point. Obtain the x-coordinate of the first feature point mapped to the panoramic image, and denote it as the first x-coordinate; obtain the x-coordinate of the second feature point mapped to the panoramic image, and denote it as the second x-coordinate; Calculate the difference between the first and second horizontal coordinates, and use it as the horizontal axis offset of the first feature point; Calculate the mean of the horizontal axis offset of all valid feature points in the first analysis image, and multiply it by 0.5 to obtain the horizontal axis correction amount; Obtain the ordinate of the first feature point after mapping it to the panoramic image, and denote it as the first ordinate; obtain the ordinate of the second feature point after mapping it to the panoramic image, and denote it as the second ordinate; Calculate the difference between the first and second ordinates, and use it as the ordinate offset of the first feature point; Calculate the mean of the vertical axis offset of all valid feature points in the first analysis image, and multiply it by 0.5 to obtain the vertical axis correction amount.

9. The multiprocessor parallel image stitching method for panoramic display as described in claim 8, characterized in that: Based on the boundary correction amount, coordinate corrections are performed on the local stitched images of the first display task block and the local stitched images of the second boundary image in the panoramic image, specifically including: The region in the panoramic image corresponding to the first boundary image is marked as the first correction region; the region in the panoramic image corresponding to the second boundary image is marked as the second correction region; For any pixel in the first correction region, subtract the horizontal axis correction amount from its horizontal coordinate and subtract the vertical axis correction amount from its vertical coordinate. For any pixel in the second correction region, add the horizontal axis correction amount to its horizontal coordinate and add the vertical axis correction amount to its vertical coordinate.

10. A multiprocessor parallel image stitching system for panoramic display, used to implement the multiprocessor parallel image stitching method for panoramic display as described in any one of claims 1-9, characterized in that: It includes a data acquisition module, a task division module, a task allocation module, an image fusion module, and a stitching correction module; among which: The data acquisition module is used to acquire the raw image sequence and the load status indicators of each processor; The task partitioning module generates a pre-analyzed image sequence based on the original image sequence, and then divides the original image sequence into continuous display task blocks based on the pre-analyzed image sequence; The task partitioning module is also used to calculate the expected load of each display task block; the task allocation module allocates each display task block to a processor based on the expected load and load status indicators, and collects the local stitched image of each display task block. The image fusion module is used to fuse the local stitched images of each display task block to generate a panoramic image; The stitching correction module is used to perform constraint correction on the local stitched images of adjacent display task blocks in the panoramic image.

Images