An image processing system and method for a suspended spherical screen panoramic vision

Abstract

The present invention belongs to the technical field of spherical screen image processing, and particularly relates to an image processing system and method for suspended spherical screen panoramic vision. The system obtains the attributes and position information of the spherical screen and multiple projection devices through a spherical screen modeling module, constructs a spherical screen-projection integrated three-dimensional model, and accurately divides the projection areas corresponding to the devices. The image processing module collects image sequences at different angles of the panoramic vision scene. The image stitching module uses feature point detection, the YOLO algorithm, and a convolutional neural network to identify overlapping areas and discriminate dynamic areas, and obtains dynamic pixel stitching lines through a dynamic programming search algorithm. The image correction module constructs a projection mapping function to correct the geometric and optical distortions of the images, and optimizes the stitching effect through a stitching correction function. Finally, the multi-view output module uses multiple projection devices to project the corrected image sequences onto the corresponding areas of the spherical screen, realizing real-time dynamic stitching and monitoring, and ensuring projection synchronization and image quality.

Description

Technical Field

[0001] The present invention belongs to the technical field of spherical screen image processing, and particularly relates to an image processing system and method for a suspended spherical screen panoramic vision. Background Art

[0002] As a new type of immersive display technology, a suspended spherical screen panoramic vision system projects 360-degree panoramic images on a hemispherical or global spherical screen; however, due to the particularity of spherical projection, if the overlapping area is not properly processed during projection, obvious stitching seams are likely to occur, affecting the visual effect and the color calibration between different projectors is inconsistent, resulting in uneven image colors on the entire spherical screen. In addition to the above-mentioned distortions, there are also optical distortion distortions such as barrel distortion and pillow distortion, all of which will cause distortion problems in the final output image.

[0003] For example, the Chinese patent application with the publication number CN108776951A discloses an image processing method for an LED spherical screen display. According to the actual size of the LED spherical screen display and the size of the source image supported by the playback system, a target image that can have the same visual effect after being played on the spherical screen display as on a flat screen display is obtained. The target image is unfolded and projected on a stereoscopic hemisphere, and the corresponding latitude pixels on the target image are cyclically processed through mathematical formulas such as the ellipse formula and trigonometric functions to obtain a flat image. Then, the flat image is compared with the source image, and finally an applicable image that meets the requirements of the LED spherical screen display playback is obtained. The applicable image is then played on the LED spherical screen display through the playback system.

[0004] For example, the Chinese patent application with the publication number CN115760555A discloses a method and device for synthesizing spherical screen images. Four orthogonal camera channels are created; a three-dimensional scene is rendered into four camera channel images; based on the mapping relationship between the plane coordinates of a single-channel camera image and the local spherical screen image coordinates, each camera channel image is converted into a local spherical screen image; a mask template corresponding to each camera channel image is generated; all the local spherical screen images and mask templates are synthesized to obtain a spherical screen image.

[0005] The above prior arts have the following problems: When the cameras in the prior art lack concentric circle alignment, obvious stitching seams, stitching misalignment or uneven colors are likely to occur, and when a dynamic object passes through the stitching area, stitching misalignment is likely to occur; therefore, the present invention provides an image processing system and method for a suspended spherical screen panoramic vision. Summary of the Invention

[0006] In view of the deficiencies of the prior art, the present invention proposes an image processing system and method for a suspended spherical screen panoramic vision. The system obtains the attributes and position information of the spherical screen and multiple projection devices through a spherical screen modeling module, constructs a spherical screen-projection integrated three-dimensional model, and accurately divides the projection area corresponding to the device; the image processing module collects image sequences at different angles of the panoramic vision scene; the image stitching module uses feature point detection, YOLO algorithm and convolutional neural network to identify the overlapping area and distinguish the dynamic area, and obtains the dynamic pixel stitching line through the dynamic programming search algorithm; the image correction module constructs a projection mapping function to correct the geometric and optical distortion of the image, and optimizes the stitching effect through the stitching correction function; finally, the multi-view output module uses multiple projection devices to project the corrected image sequence onto the corresponding area of the spherical screen, realizing real-time dynamic stitching and monitoring, ensuring projection synchronization and image quality; the present invention provides an accurate image processing solution for the suspended spherical screen panoramic vision, and solves the problem of easy misalignment in dynamic display.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] An image processing system for a suspended spherical screen panoramic vision, comprising: a spherical screen modeling module, an image processing module and an image stitching module; the spherical screen modeling module includes a three-dimensional model construction unit and a projection area division unit; obtains the attribute information of the spherical screen and the attribute information and viewing angle information of multiple projection devices, and inputs them into the three-dimensional model construction unit to obtain a spherical screen-projection integrated three-dimensional model, and obtains the corresponding projection area and numbers it on the spherical screen three-dimensional model according to the projection device viewing area information;

[0009] Collects image sequences at different angles of the same panoramic vision scene in different time dimensions through the image processing module and numbers and labels them; the image stitching module includes an overlapping area recognition unit, a dynamic area discrimination and segmentation unit, a feature extraction unit and a stitching seam search and optimization unit;

[0010] Inputs the labeled image sequence into the overlapping area recognition unit to perform overlapping area recognition and judgment, obtains overlapping area image pairs, and at the same time inputs the labeled image sequence into the dynamic area discrimination and segmentation unit to perform dynamic area judgment and dynamic area movement path prediction, obtains the stitching seam dynamic adjustment factor; and inputs the overlapping area image pairs and the corresponding YUV chrominance space pairs into the feature extraction unit to obtain the overlapping area feature difference value, inputs the overlapping area feature difference value and the stitching seam dynamic adjustment factor into the stitching seam search and optimization unit, and obtains the overlapping area dynamic pixel stitching line through the dynamic programming search algorithm.

[0011] Specifically, the image processing system further includes an image correction module and a multi-view output module;

[0012] The image correction module includes an image correction unit and a stitching correction unit; input the overlapping region image pairs at the same moment into the image correction unit, perform geometric and optical corrections through the configured projection mapping function, and obtain the corrected projection image sequence; at the same time, input the dynamic pixel stitching line of the overlapping region into the stitching correction unit, and obtain the dynamically pixel stitching line with secondary correction after correction through the configured stitching correction factor;

[0013] Project the corrected projection image sequence to the corresponding projection area through the multi-view output module and the labeled labels, and perform real-time dynamic stitching and monitoring evaluation on the projection image through the dynamically pixel stitching line with secondary correction, and feedback the evaluation result to the projection control unit and the stitching seam search and optimization unit to perform real-time adjustment on the projection device angle and the dynamically pixel stitching line parameters of the overlapping region.

[0014] Specifically, the steps for obtaining the dynamic adjustment factor of the stitching seam include:

[0015] A1. Build and train a dynamic detection and segmentation model based on the YOLO-tiny algorithm, input the image sequences of different angles of the same panoramic visual scene after annotation into the dynamic detection and segmentation model, and obtain the target region anchor boxes of the same discriminant target in the images of different angles in the image sequence under the same panoramic visual scene , where represents the region anchor box of the j-th discriminant target in the i-th image under the same panoramic visual scene, respectively represent the maximum and minimum values of the abscissa point and the ordinate point of the region anchor box corresponding to the discriminant target;

[0016] A2. Input the discriminant target region anchor boxes in the images of different angles obtained into the optical flow algorithm and combine the built-in discriminant formula to determine whether each discriminant target region anchor box is a moving target region anchor box, and segment the discriminant moving target region anchor box from the corresponding image to obtain the moving target region anchor box and the corresponding moving direction and the overlapping region image pair including the mask region;

[0017] A3. According to the obtained moving target region anchor box and the corresponding moving direction and the overlapping regions in the images of different angles under the same panoramic visual scene obtained by the overlapping region recognition unit, judge whether the moving target region anchor box intersects with the overlapping region or will intersect with the overlapping region at a future moment through the Kalman filtering algorithm;

[0018] A4. If it intersects, retain the corresponding moving target region anchor box to construct the dynamic adjustment factor of the stitching seam, and if it does not intersect, it is not used to construct the dynamic adjustment factor of the stitching seam.

[0019] Specifically, the built-in discriminant formula in A2 is:

[0020] ,

[0021] Among them, is the feature point extracted by the optical flow algorithm from the i-th image in the same panoramic visual scene, represents the set of feature points in the regional anchor box of the j-th discriminative target in the i-th image, represents the motion vector angle of the k-th feature point in the regional anchor box of the j-th discriminative target in the i-th image, represents the mean value of the motion vector angles of all feature points in the regional anchor box of the j-th discriminative target in the i-th image; represents the norm length of the k-th feature point in the regional anchor box of the j-th discriminative target in the i-th image and the k-th feature point in the regional anchor box of the j-th discriminative target in the (i + 1)-th image.

[0022] Specifically, the steps to obtain the dynamic pixel stitching line of the overlapping region include:

[0023] B1. For the overlapping region image pair obtained by the overlapping region recognition unit, through color space transformation, obtain the overlapping region YUV chrominance space pair;

[0024] B2. Input the overlapping region image pair and the corresponding overlapping region YUV chrominance space pair into the pyramid feature extraction model, and extract the overlapping region brightness difference value, chrominance difference value, and the horizontal gradient difference and vertical gradient difference values of the overlapping region content structure;

[0025] B3. Based on the overlapping region brightness difference value, chrominance difference value, and the horizontal gradient difference and vertical gradient difference values of the overlapping region content structure, construct a multi-index fusion energy function E, specifically: , where represents the chrominance difference value of each pair of corresponding pixels in the overlapping region, represents the difference value of the gray information entropy of each pair of corresponding pixels in the overlapping region, represents the horizontal gradient difference of each pair of overlapping region content structures, represents the vertical gradient difference value of each pair of overlapping regions, successively represent the weighting coefficients of the multi-index fusion energy function.

[0026] Specifically, the steps to obtain the dynamic pixel stitching line of the overlapping region further include:

[0027] B4. Based on the multi-index fusion energy function and the stitching seam dynamic adjustment factor, construct the search function D and the constraint conditions of the dynamic pixel stitching line of the overlapping region, specifically:

[0028] ,

[0029] Among them, represents the pixel points in the overlapping area, and (x, y) are the coordinates of the pixel points. represents the dynamic adjustment factor of the splicing seam, and & represents the sum symbol;

[0030] B5. Calculate the difference value of the gray information entropy of all row pixel points in the overlapping area based on the difference value of the gray information entropy extracted from the overlapping area, and take the point with the smallest difference value of the gray information entropy of all row pixel points in the overlapping area as the search starting point;

[0031] B6. Input the obtained search starting point, search function D, and constraint conditions into the dynamic programming search algorithm, and in each step of the search process in the overlapping area, set the energy values of 5 pixel points in the adjacent columns corresponding to the pixels one row apart from the previous optimal point as the consideration basis, obtain the point with the smallest energy value in the current search step, take it as the search result of this step, and then add this search result to the optimal point set of the dynamic pixel splicing line;

[0032] B7. Set the row where the search starting point is located as the starting row, and search in both the upward and downward directions. When the search reaches the last row in both the upward and downward directions, the search ends, and output the dynamic pixel splicing line of the overlapping area constructed from the obtained optimal point set.

[0033] Specifically, the steps for obtaining the corrected projection image sequence include:

[0034] C1. Let the coordinates of the pixel point of the jth discrimination target in the ith image of the current projection be (x, y), and set the plane projection surface of the projection area i corresponding to the ith image as and coincides with the corresponding projected plane image;

[0035] C2. Obtain the radius R and the center position C of the current projection area in the spherical screen, and establish a world coordinate system with the center of the sphere as the coordinate point. Let be the coordinate of a plane projection point on , and obtain the curved surface projection point of and the corresponding projection geometric distortion function according to the intersection point coordinates between the connection line between and the center position C of the current projection area;

[0036] C3. Repeat step C2 to obtain all the curved surface projection points corresponding to the current projection area and the corresponding projection geometric distortion functions, and solve for the optimal geometric projection angle and the corresponding center position through the particle swarm algorithm according to the sum of all the obtained projection geometric distortion functions, and perform real-time optimization on the image geometric projection;

[0037] C4. Input the projection image into the radial distortion and tangential distortion models to perform real-time optical correction on the optical distortion of the projection image;

[0038] C5. Repeat C3 and C4 to perform geometric correction and optical correction on all projection images at the same moment, and obtain the geometrically corrected projection surface image.

[0039] Specifically, the steps for obtaining the dynamically pixel-stitched line of the secondary correction include:

[0040] D1. Obtain the optimal geometric projection angle, the corresponding center position in C3, and the luminance projection parameters corresponding to the overlapping area after optical correction in C4. Input the obtained parameters and the dynamically pixel-stitched line of the corresponding overlapping area into the dynamic programming search algorithm for secondary search correction to obtain the dynamically pixel-stitched line of the secondary correction;

[0041] D2. Use the dynamically pixel-stitched line of the secondary correction to stitch the overlapping area image pair. At the same time, replace the pixel values corresponding to the stitched overlapping area with the pixel average values at the corresponding pixel positions in the overlapping area image pair;

[0042] D3. Repeat the processes of D1 and D2 to search for and stitch the overlapping area image pair to obtain a complete panoramic visual image projection;

[0043] D4. Fuse the segmented moving target area anchor boxes that have undergone geometric and optical corrections into the panoramic visual image projection through a fusion mapping function to obtain a dynamic panoramic visual image projection , and the fusion mapping function is specifically:

[0044] ,

[0045] Among them, represents the stitched complete panoramic visual image projection, represents the corresponding mask area, represents and the matching fusion coefficient of the corresponding boundaries.

[0046] An image processing method for a suspended spherical screen panoramic vision, the steps include:

[0047] S1. Obtain the spherical screen and projection device attribute information, construct a spherical screen-projection integrated three-dimensional model, and divide the projection area on the spherical screen three-dimensional model to correspond the projection area to the projection device one by one;

[0048] S2. Obtain the image sequences at different angles of the panoramic visual scene, obtain the overlapping areas between different images through the feature point recognition algorithm, construct the overlapping area image pair, and input the overlapping area image pair and the corresponding YUV chrominance space pair into the convolutional neural network to obtain the overlapping area feature difference value;

[0049] S3. Using the obtained overlapping region feature difference value and the splicing seam dynamic adjustment factor, through the dynamic programming search algorithm, obtain the corresponding dynamic pixel splicing line within the overlapping region;

[0050] S4. Construct a projection mapping function, use the projection mapping function to perform geometric correction and optical correction on the image sequence projected onto the corresponding projection region, and at the same time use the dynamically pixel splicing line after secondary correction corrected by the splicing correction factor to perform real-time dynamic splicing on the projected image sequence.

[0051] A computer-readable storage medium, on which computer instructions are stored, and when the computer instructions run, an image processing method for a suspended spherical screen panoramic vision is executed.

[0052] Compared with the prior art, the beneficial effects of the present invention are:

[0053] 1. Aiming at the deficiencies of the prior art, the present invention accurately obtains the attribute information of the spherical screen and the projection device through the three-dimensional model construction unit and the projection region division unit, constructs a high-precision spherical screen-projection integrated three-dimensional model, ensuring a more reasonable division of the projection region of the projection device on the spherical screen; secondly, the dynamic region discrimination and segmentation unit uses the YOLO algorithm and the optical flow algorithm to accurately identify and predict the movement path of the dynamic region, generating a splicing seam dynamic adjustment factor, enabling the system to adjust the splicing seam in real time and avoiding the splicing misalignment problem caused by dynamic objects.

[0054] 2. Aiming at the deficiencies of the prior art, the present invention converts the overlapping region image pair into a YUV chrominance space pair through the feature extraction unit and inputs it into the pyramid convolutional neural network to obtain the luminance, chrominance, and content structure difference features. Combining with the splicing seam dynamic adjustment factor, through the dynamic programming search algorithm, the optimal dynamic pixel splicing line is obtained, not only improving the splicing accuracy, but also ensuring the color uniformity and structural consistency of the splicing region.

[0055] 3. Aiming at the deficiencies of the prior art, the present invention performs geometric and optical correction on the projected image through the projection mapping function configured by the image correction unit and the splicing correction unit, ensuring the accuracy and clarity of the projected image on the spherical screen; at the same time, the multi-view output module performs real-time dynamic splicing on the projected image through the dynamically pixel splicing line after secondary correction and feeds back the evaluation result to the projection control unit and the splicing seam search and optimization unit, realizing the real-time adjustment of the projection device angle and the dynamically pixel splicing line parameters in the overlapping region; this closed-loop feedback mechanism ensures the long-term stability and high precision of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a module diagram of an image processing system for a suspended spherical screen panoramic vision according to Embodiment 1 of the present invention;

[0057] Figure 2 It is the flowchart of the working process of the corresponding unit of an image processing system for a suspended spherical screen panoramic vision in Embodiment 2 of the present invention;

[0058] Figure 3 It is the flowchart of an image processing method for a suspended spherical screen panoramic vision in Embodiment 3 of the present invention. Specific embodiments

[0059] Embodiment 1

[0060] Please refer to Figure 1 , an embodiment provided by the present invention: an image processing system for a suspended spherical screen panoramic vision, including: a spherical screen modeling module, an image processing module, an image stitching module, an image correction module, and a multi-view output module;

[0061] The spherical screen modeling module is used to obtain the spherical screen attribute information, the attribute information and position information of multiple projection devices to construct a spherical screen projection three-dimensional model; the spherical screen modeling module includes a three-dimensional model construction unit, a projection area division unit, and a projection regulation unit;

[0062] The three-dimensional model construction unit is used to obtain a spherical screen-projection integrated three-dimensional model through a three-dimensional model generation algorithm according to the spherical screen attribute information, the attribute information and viewing angle information of multiple projection devices;

[0063] Furthermore, in this embodiment, the spherical screen-projection integrated three-dimensional model includes a spherical screen three-dimensional model and a projection device three-dimensional model;

[0064] The projection area division unit is used to divide the spherical screen three-dimensional model in the spherical screen-projection integrated three-dimensional model into different projection areas according to the projection viewing angle area information of the projection device on the spherical screen, and label each projection area and correspond it to the corresponding projection device one by one;

[0065] Furthermore, in this embodiment, the projection area is obtained through a genetic algorithm according to the projection viewing angle area information of the projection device on the spherical screen and the spherical screen three-dimensional model information to obtain the corresponding projection area division;

[0066] The projection regulation unit is used to adjust the projection angle of the projection device according to the image projection information, and monitor whether the projection images of each projection device are synchronized at the same moment. If they are not synchronized, projection synchronization adjustment is performed;

[0067] Furthermore, in this embodiment, the projection device is configured with a position regulation model to regulate the angle and height of the projection device, and the position regulation model is constructed through the SAC algorithm.

[0068] The image processing module is used to collect image sequences of the same panoramic visual scene at different angles in different time dimensions, perform preprocessing of enhancement and denoising on the collected image sequences, and at the same time perform label numbering;

[0069] Through the three-dimensional model construction unit, this process can accurately establish a spherical screen-projection integrated three-dimensional model according to the actual parameters of the spherical screen and projection equipment, ensuring the accuracy and practicality of the model, and providing a solid foundation for subsequent projection area division. Secondly, the projection area division unit uses the genetic algorithm for intelligent division, which not only improves the division efficiency, but also ensures the best match between each projection area and the projection equipment, effectively avoiding the problems of overlap and blind areas, and enhancing the realism and immersion of the projection effect. Furthermore, the projection regulation unit uses the position regulation model constructed by the SAC algorithm to dynamically adjust the angles and heights of the projection equipment, ensuring the synchronization of the projection images among multiple devices, solving the problem of inconsistent projections caused by differences between devices, and greatly improving the stability and reliability of the system. Finally, the image processing module significantly improves the image quality by performing preprocessing of enhancement and denoising on the panoramic visual scene image sequences in different time dimensions, making the final image presented to the audience clearer.

[0070] The image stitching module is used to stitch the enhanced panoramic images; the image stitching module includes an overlapping area recognition unit, a dynamic area discrimination and segmentation unit, a feature extraction unit, and a stitching seam search and optimization unit;

[0071] The overlapping area recognition unit is used to identify the corresponding overlapping areas between different images through a feature point detection algorithm according to the planar images of the same panoramic visual scene at different angles, and construct overlapping area image pairs by taking two images with overlapping areas as a group;

[0072] The dynamic area discrimination and segmentation unit is used to identify and judge whether there is a dynamic area according to the images with overlapping areas through the YOLO algorithm, and predict the moving path of the dynamic area through the optical flow algorithm to obtain a stitching seam dynamic adjustment factor;

[0073] Furthermore, the steps for obtaining the stitching seam dynamic adjustment factor in this embodiment include:

[0074] A1. Build and train a dynamic detection and segmentation model based on the YOLO-tiny algorithm, and input the labeled image sequences of the same panoramic visual scene at different angles into the dynamic detection and segmentation model to obtain the target area anchor boxes of the same discriminant target in the images at different angles in the image sequence under the same panoramic visual scene , where represents the area anchor box of the jth discriminant target in the ith image under the same panoramic visual scene, respectively represent the maximum and minimum values of the abscissa and ordinate points of the regional anchor box corresponding to the discrimination target;

[0075] A2. Input the regional anchor boxes of the discrimination targets in the acquired images at different angles into the optical flow algorithm and combine the built-in discrimination formula to determine whether each regional anchor box of the discrimination target is a moving target regional anchor box, and segment the discrimination target regional anchor boxes determined to be moving from the corresponding images to obtain the moving target regional anchor boxes and their corresponding moving directions and the overlapping region image pairs including the mask regions;

[0076] Further, in this embodiment, the discrimination formula is:

[0077] ,

[0078] where is the feature point extracted by the optical flow algorithm from the i-th image under the same panoramic vision scene, represents the set of feature points in the regional anchor box of the j-th discrimination target in the i-th image, represents the motion vector angle of the k-th feature point in the regional anchor box of the j-th discrimination target in the i-th image, represents the mean value of the motion vector angles of all feature points in the regional anchor box of the j-th discrimination target in the i-th image; represents the norm length of the k-th feature point in the regional anchor box of the j-th discrimination target in the i-th image and the k-th feature point in the regional anchor box of the j-th discrimination target in the (i + 1)-th image;

[0079] Further, the meaning represented by the above formula in this embodiment is: If there are target points screened by the optical flow in the regional anchor box of the discrimination target, and the motion vector directions are similar or the same and the norm length is greater than one pixel distance, then it is determined that the target region is a moving target regional anchor box.

[0080] A3. According to the acquired moving target regional anchor boxes and their corresponding moving directions and the overlapping regions in the images at different angles in the same panoramic vision scene obtained by the overlapping region recognition unit, use the Kalman filter algorithm to determine whether the moving target regional anchor boxes intersect with the overlapping regions or will intersect with the overlapping regions at a future moment;

[0081] A4. If they intersect, retain the corresponding moving target regional anchor boxes to construct the splicing seam dynamic adjustment factor, and if they do not intersect, do not use them to construct the splicing seam dynamic adjustment factor.

[0082] This process first uses an advanced feature point detection algorithm through the overlapping area recognition unit, which can quickly and accurately find the overlapping parts between different images in a complex environment. Secondly, the dynamic area discrimination and segmentation unit introduces the YOLO algorithm to identify and separate dynamic objects, and combines the optical flow algorithm to predict their movement trajectories, thus avoiding the interference of these dynamic elements on the stitching result and maintaining the consistency of the static background. In addition, by analyzing the moving target area through the Kalman filtering algorithm, factors that may affect the stitching effect can be predicted in advance, and corresponding measures can be taken to avoid them. Finally, under the action of the stitching seam search and optimization unit, the system can automatically find the best stitching scheme to achieve the seamless connection effect.

[0083] The feature extraction unit is used to convert the overlapping area image pair into a YUV chrominance space pair. The overlapping area image pair and the corresponding YUV chrominance space pair are input into a convolutional neural network to obtain the luminance difference value, chrominance difference value, and overlapping area structure difference feature of each position point corresponding pixel in the overlapping area.

[0084] The stitching seam search and optimization unit is used to obtain the dynamic pixel stitching line in the overlapping area through the dynamic programming search algorithm according to the luminance difference value, chrominance difference value, and overlapping area structure difference feature of the corresponding position pixels in the overlapping area and the stitching seam dynamic adjustment factor.

[0085] Furthermore, the steps of obtaining the dynamic pixel stitching line in the overlapping area in this embodiment include:

[0086] B1. The overlapping area image pair obtained by the overlapping area recognition unit is subjected to color space transformation to obtain the overlapping area YUV chrominance space pair.

[0087] B2. The overlapping area image pair and the corresponding overlapping area YUV chrominance space pair are input into the pyramid feature extraction model to extract the overlapping area luminance difference value, chrominance difference value, and overlapping area content structure horizontal gradient difference and vertical gradient difference values.

[0088] B3. Based on the overlapping area luminance difference value, chrominance difference value, and overlapping area content structure horizontal gradient difference and vertical gradient difference values, a multi-index fusion energy function E is constructed as follows: , where represents the chrominance difference value of each pair of corresponding pixel points in the overlapping area, represents the gray information entropy difference value of each pair of corresponding pixel points in the overlapping area, represents the horizontal gradient difference of the overlapping area content structure of each pair, represents the vertical gradient difference value of each pair of overlapping areas, represent the weighting coefficients of the multi-index fusion energy function in turn;

[0089] B4. Construct a search function D and constraint conditions for the dynamic pixel stitching line in the overlapping region based on the multi-index fusion energy function and the stitching seam dynamic adjustment factor, specifically as follows:

[0090] ,

[0091] wherein, represents the pixel points in the overlapping region, and (x, y) are the coordinates of the pixel points, represents the stitching seam dynamic adjustment factor, and & represents the sum symbol;

[0092] Furthermore, in this embodiment, the search function D is to search for the pixel point corresponding to the earliest corresponding fusion energy value, and this pixel point cannot be in the moving target region anchor box corresponding to the stitching seam dynamic adjustment factor; and this pixel point can only be searched in the overlapping region.

[0093] B5. Calculate the point with the smallest gray information entropy difference value of all row pixel points in the overlapping region based on the gray information entropy difference value extracted from the overlapping region as the search starting point;

[0094] B6. Input the obtained search starting point, search function D, and constraint conditions into the dynamic programming search algorithm, and in each step of the search process in the overlapping region, set the energy values of 5 pixel points in the adjacent columns corresponding to the pixels one row apart from the previous optimal point as the consideration basis, obtain the point with the smallest energy value in the current search step, and use it as the search result of this step, and then add this search result to the optimal point set of the dynamic pixel stitching line;

[0095] B7. Set the row where the search starting point is located as the starting row, and search in both the up and down directions. When the search reaches the last row in both the up and down directions, the search ends, and output the dynamic pixel stitching line in the overlapping region constructed from the obtained optimal point set.

[0096] This process first converts the overlapping region image pair to a YUV chrominance space pair through a feature extraction unit, and uses a convolutional neural network to extract features of luminance difference, chrominance difference, and structural difference. This step not only enhances the ability to capture image details but also provides rich and accurate data support for the subsequent optimization of the stitching seam. Secondly, based on these difference features and the stitching seam dynamic adjustment factor, the stitching seam search and optimization unit constructs a multi-index fusion energy function E, comprehensively considering multiple dimensions such as chrominance, grayscale information entropy, horizontal and vertical gradients, ensuring that the selection of the stitching line takes into account both visual continuity and the impact of dynamic objects. In particular, through the dynamic programming search algorithm, starting from the point with the smallest difference in grayscale information entropy, the optimal stitching line is searched row by row, effectively avoiding the stitching seam passing through the moving object area, reducing the stitching trace, and making the stitching result smoother and more natural. This method not only solves the hard boundary problem existing in traditional stitching techniques but also improves the adaptability to dynamic scenes, greatly enhancing the overall quality of the panoramic image and the user experience.

[0097] The image correction module is used to construct a projection mapping function between the planar image and the curved spherical screen through a set algorithm, and correct the geometric distortion and optical distortion of the projection image through the projection mapping function. The image correction module includes an image correction unit and a stitching correction unit;

[0098] The image correction unit is used to perform geometric correction and optical correction on the image projected onto the corresponding projection area through the configured projection mapping function to obtain a sequence of corrected projection images;

[0099] Further, the steps for obtaining the sequence of corrected projection images in this embodiment include:

[0100] C1. Let the coordinates of the pixel point of the j-th discriminant target in the i-th image of the current projection be (x, y), and set the planar projection plane of the i-th image corresponding to the projection area i as and coincides with the corresponding projected planar image;

[0101] C2. Obtain the radius R and the center position C of the current projection area in the spherical screen, and establish a world coordinate system with the center of the sphere as the coordinate point. Let be a planar projection point coordinate on According to the intersection point coordinate between the line connecting and the center position C of the current projection area, obtain the curved surface projection point of

[0102] C3. Repeat step C2 to obtain all the surface projection points corresponding to the current projection area and the corresponding projection geometric distortion functions. Then, based on the sum of all the obtained projection geometric distortion functions, use the particle swarm optimization algorithm to solve for the optimal geometric projection angle and the corresponding center position, and perform real-time optimization on the image geometric projection;

[0103] C4. Input the projected image into the radial distortion and tangential distortion models to perform real-time optical correction on the optical distortion of the projected image;

[0104] C5. Repeat C3 and C4 to perform geometric correction and optical correction on all the projected images at the same moment, and obtain the projected surface image after geometric correction.

[0105] The splicing correction unit is used to construct a splicing correction function according to the projection mapping functions of two images corresponding to the adjacent overlapping areas, obtain a splicing correction factor, and feedback the splicing correction factor to the splicing seam search and optimization unit to perform geometric and optical correction on the obtained dynamic pixel splicing line, and obtain the dynamically pixel splicing line after secondary correction;

[0106] Further, in this embodiment, the steps for obtaining the dynamically pixel splicing line after secondary correction include:

[0107] D1. Obtain the optimal geometric projection angle and the corresponding center position in C3 and the brightness projection parameters corresponding to the overlapping area after optical correction in C4. Input the obtained parameters and the dynamic pixel splicing line corresponding to the overlapping area into the dynamic programming search algorithm for secondary search correction to obtain the dynamically pixel splicing line after secondary correction;

[0108] D2. Use the dynamically pixel splicing line after secondary correction to splice the overlapping area image pair. At the same time, replace the pixel point values corresponding to the spliced overlapping area with the pixel average values at the corresponding pixel positions in the overlapping area image pair;

[0109] D3. Repeat the processes of D1 and D2 to search for and splice the overlapping area image pair to obtain a complete panoramic visual image projection;

[0110] D4. Fuse the segmented moving target area anchor boxes that have undergone geometric and optical correction into the panoramic visual image projection through the fusion mapping function to obtain a dynamic panoramic visual image projection , and the fusion mapping function is specifically:

[0111] ,

[0112] where represents the spliced complete panoramic visual image projection, represents the mask area corresponding to , represents and The matching fusion coefficient corresponding to the boundary.

[0113] Furthermore, in this embodiment, the boundary matching fusion coefficient is obtained through the corresponding boundary pixel feature points and the matching algorithm.

[0114] This process optimizes the geometric projection angle and the center position of the circle by the particle swarm algorithm through the image correction unit, ensuring the geometric accuracy of the projected image on the spherical screen and eliminating the geometric distortion caused by the curved surface projection; at the same time, the application of the radial distortion and tangential distortion models effectively corrects the image deformation caused by the optical elements, ensuring the clarity and realism of the image; and the pixel values corresponding to the overlapping areas after stitching are replaced with the pixel average values at the corresponding pixel positions in the overlapping area images, solving the problem of ghosting in the projection overlapping areas and making the transition of the stitching area more natural.

[0115] The stitching correction unit further optimizes the stitching effect. By constructing a stitching correction function and using the dynamic programming search algorithm to perform secondary correction on the dynamic pixel stitching line, it ensures that the stitching line after projection will not deform and the smoothness and consistency of the stitching between different projection areas; especially when processing dynamic targets, the corrected moving targets are re-fused into the panoramic vision image through the fusion mapping function, not only maintaining the coherence of the dynamic content but also enhancing the naturalness of the overall image; in addition, the introduction of the boundary matching fusion coefficient makes the transition between different regions more natural, reduces the stitching marks, and improves the user's immersive experience.

[0116] The multi-view output module is used to project the image sequences corrected by the image correction unit at different angles of the same panoramic vision scene corresponding to different time dimensions onto the corresponding projection areas on the spherical screen through multiple projection devices, perform real-time dynamic stitching on the projected images using the obtained dynamically corrected dynamic pixel stitching line, monitor and obtain the image projection information in real time, and feedback the obtained projection information to the projection control unit and the stitching seam search and optimization unit to adjust the angles of the projection devices and the parameters of the dynamic pixel stitching line in the overlapping areas in real time.

[0117] Embodiment 2

[0118] Please refer to Figure 2 , another embodiment provided by the present invention: the working process of the corresponding unit of an image processing system for a suspended spherical screen panoramic vision, including:

[0119] First, a spherical screen-projection integrated three-dimensional model is constructed by the three-dimensional model construction unit, and on the spherical screen three-dimensional model in the constructed spherical screen-projection integrated three-dimensional model, the corresponding projection areas are obtained and numbered through the projection area division unit;

[0120] Second, obtain image sequences of the same panoramic visual scene at different angles in different time dimensions after enhanced denoising through the image processing module, label and annotate them, and input the image sequences of the same panoramic visual scene at different angles into the overlapping area recognition unit to obtain overlapping area image pairs. At the same time, input the image sequences of the same panoramic visual scene at different angles into the dynamic area discrimination and segmentation unit, and obtain the dynamic area and the dynamic adjustment factor of the splicing seam through the YOLO algorithm combined with the optical flow algorithm;

[0121] Third, input the overlapping area image pairs into the feature extraction unit to obtain the brightness difference value, chromaticity difference value, and overlapping area structure difference features of the corresponding pixels at each position point within the overlapping area. Input the obtained brightness difference value, chromaticity difference value, overlapping area structure difference features of the corresponding pixels at each position point within the overlapping area, and the dynamic adjustment factor of the splicing seam into the splicing seam search and optimization unit to obtain the corresponding dynamic pixel splicing line within the overlapping area;

[0122] Fourth, after correcting the image sequences of the same panoramic visual scene at different angles at the same moment through the projection mapping function built in the image correction unit, project them into the corresponding projection areas divided on the spherical screen through the multi-view output module. At the same time, use the splicing correction factor obtained by the splicing correction unit to correct the dynamic pixel splicing line between the overlapping area image pairs, and use the corrected secondary corrected dynamic pixel splicing line to perform dynamic splicing on the images projected into the projection area in real time to obtain a complete full-visual projection image. Real-time evaluate and detect the obtained projection deviation angle, and feedback the obtained projection deviation angle to the projection control unit to perform real-time synchronous adjustment on the projection angle of the projection device. At the same time, feedback the projection evaluation result to the splicing seam search and optimization unit to perform real-time optimization on the dynamic pixel splicing line parameters.

[0123] Embodiment 3

[0124] Please refer to Figure 3 , another embodiment provided by the present invention: An image processing method for a suspended spherical screen panoramic vision, the steps include:

[0125] S1. Obtain the attribute information of the spherical screen and the projection device, construct a spherical screen-projection integrated three-dimensional model, divide the projection area on the spherical screen three-dimensional model, and correspond the projection area to the projection device one by one;

[0126] S2. Obtain image sequences of different angles of the panoramic visual scene, obtain the overlapping areas between different images through the feature point recognition algorithm, construct overlapping area image pairs, and input the overlapping area image pairs and the corresponding YUV chromaticity space pairs into the convolutional neural network to obtain the overlapping area feature difference values;

[0127] Further, in this embodiment, the overlapping area feature difference values include the brightness difference value, the chromaticity difference value, and the overlapping area structure difference features;

[0128] S3. Using the obtained overlapping region feature difference value and the splicing seam dynamic adjustment factor, through the dynamic programming search algorithm, obtain the corresponding dynamic pixel splicing line within the overlapping region;

[0129] S4. Construct a projection mapping function, use the projection mapping function to perform geometric correction and optical correction on the image sequence projected onto the corresponding projection region, and at the same time use the dynamically pixel splicing line after secondary correction corrected by the splicing correction factor to perform real-time dynamic splicing on the projected image sequence;

[0130] S5. Real-time monitor and evaluate the splicing effect of the image sequence, obtain the projection deviation angle, and feedback the obtained projection deviation angle to the configured projection device control subsystem to adjust the projection angle of the projection device in real time. At the same time, feedback the projection evaluation result to the dynamic programming search algorithm to optimize the parameters of the dynamic pixel splicing line.

[0131] The embodiments of the present invention have been described above in conjunction with the accompanying drawings. However, the present invention is not limited to the above specific embodiments. The above specific embodiments are merely illustrative and not restrictive. Under the inspiration of the present invention, those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments without departing from the purpose of the present invention and the scope protected by the claims. These all fall within the protection scope of the present invention.

[0132] If the technical solution of the present disclosure involves personal information, before the product applying the technical solution of the present disclosure processes personal information, it has clearly informed the personal information processing rules and obtained the personal's autonomous consent. If the technical solution of the present disclosure involves sensitive personal information, before the product applying the technical solution of the present disclosure processes sensitive personal information, it has obtained the personal's separate consent and at the same time meets the requirement of "express consent". For example, at a personal information collection device such as a camera, a clear and prominent sign is set to inform that the personal information collection range has been entered and personal information will be collected. If an individual voluntarily enters the collection range, it is regarded as consenting to the collection of their personal information; or on the personal information processing device, when the personal information processing rules are informed by obvious signs / information, personal authorization is obtained through pop-up information or asking the individual to upload their personal information by themselves, etc.; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

Claims

1. An image processing system for a suspended spherical screen panoramic vision, characterized in that, Including: A spherical screen modeling module, an image processing module, and an image stitching module; The spherical screen modeling module includes a three-dimensional model construction unit and a projection area division unit; Obtain the spherical screen attribute information, the attribute information and viewing angle information of multiple projection devices, and input them into the three-dimensional model construction unit to obtain a spherical screen-projection integrated three-dimensional model. Based on the viewing angle area information of the projection devices on the spherical screen three-dimensional model, obtain the corresponding projection areas and label them; Collect image sequences of different angles of the same panoramic visual scene in different time dimensions through the image processing module and perform label annotation; the image stitching module includes an overlapping area recognition unit, a dynamic area discrimination and segmentation unit, a feature extraction unit, and a stitching seam search and optimization unit; Input the labeled image sequences into the overlapping area recognition unit to perform overlapping area recognition and judgment to obtain overlapping area image pairs. At the same time, input the labeled image sequences into the dynamic area discrimination and segmentation unit to perform dynamic area judgment and dynamic area movement path prediction to obtain a stitching seam dynamic adjustment factor; and input the overlapping area image pairs and the corresponding YUV chrominance space pairs into the feature extraction unit to obtain the overlapping area feature difference value. Input the overlapping area feature difference value and the stitching seam dynamic adjustment factor into the stitching seam search and optimization unit, and through the dynamic programming search algorithm, obtain the overlapping area dynamic pixel stitching line; The steps for obtaining the stitching seam dynamic adjustment factor include: A1. A dynamic detection and segmentation model is constructed and trained based on the YOLO-tiny algorithm. An image sequence of the same panoramic visual scene at different angles after annotation is input into the dynamic detection and segmentation model to obtain the target region anchor boxes of the same discriminative target in the images of different angles in the image sequence of the same panoramic visual scene , where represents the region anchor box of the j-th discriminative target in the i-th image of the same panoramic visual scene, successively represent the maximum and minimum values of the abscissa point and the ordinate point of the region anchor box corresponding to the discriminative target; A2. Input the discriminant target area anchor boxes in the images of different angles obtained into the optical flow algorithm and combine the built-in discriminant formula to determine whether each discriminant target area anchor box is a moving target area anchor box. Segment the discriminant moving target area anchor boxes from the corresponding images to obtain the moving target area anchor boxes and their corresponding moving directions and overlapping area image pairs including the mask areas; A3. According to the obtained moving target area anchor boxes and their corresponding moving directions and the overlapping areas in the images of different angles of the same panoramic visual scene obtained by the overlapping area recognition unit, use the Kalman filtering algorithm to determine whether the moving target area anchor boxes intersect with the overlapping areas or will intersect with the overlapping areas at future times; A4. If they intersect, retain the corresponding moving target area anchor boxes to construct the stitching seam dynamic adjustment factor. If they do not intersect, they are not used to construct the stitching seam dynamic adjustment factor.

2. The image processing system for a suspended spherical screen panoramic vision according to claim 1, wherein The image processing system further includes an image correction module and a multi-view output module; The image correction module includes an image correction unit and a stitching correction unit; input the overlapping area image pairs at the same time into the image correction unit, and perform geometric and optical corrections through the configured projection mapping function to obtain the corrected projection image sequences; at the same time, input the overlapping area dynamic pixel stitching line into the stitching correction unit, and obtain the secondarily corrected dynamic pixel stitching line after correction through the configured stitching correction factor; Project the corrected projection image sequence onto the corresponding projection area through the multi-view output module and the labeled labels, and perform real-time dynamic stitching and monitoring evaluation on the projection images through the dynamically pixel stitching line of the secondary correction, and feedback the evaluation results to the projection control unit and the stitching seam search and optimization unit to perform real-time adjustment on the projection device angle and the parameters of the dynamically pixel stitching line in the overlapping area.

3. The image processing system for a suspended spherical screen panoramic vision according to claim 2, characterized in that, The discrimination formula built in A2 is as follows: ， Among them, is the feature point extracted by the optical flow algorithm from the i-th image in the same panoramic visual scene, represents the set of feature points in the regional anchor box of the j-th discriminative target in the i-th image, represents the motion vector angle of the k-th feature point in the regional anchor box of the j-th discriminative target in the i-th image, represents the mean value of the motion vector angles of all feature points in the regional anchor box of the j-th discriminative target in the i-th image; represents the norm length between the k-th feature point in the regional anchor box of the j-th discriminative target in the i-th image and the k-th feature point in the regional anchor box of the j-th discriminative target in the (i + 1)-th image.

4. The image processing system for a suspended spherical screen panoramic vision according to claim 3, wherein, The steps for obtaining the dynamically pixel stitching line in the overlapping area include: B1. For the overlapping area image pair obtained by the overlapping area recognition unit, obtain the overlapping area YUV chrominance space pair through color space transformation. B2. Input the overlapping area image pair and the corresponding overlapping area YUV chrominance space pair into the pyramid feature extraction model to extract the brightness difference value, chrominance difference value, horizontal gradient difference and vertical gradient difference value of the overlapping area content structure. B3. Based on the luminance difference value, chromaticity difference value of the overlapping region, and the horizontal gradient difference and vertical gradient difference values of the content structure of the overlapping region, a multi-index fusion energy function E is constructed as follows: , where represents the chromaticity difference value of the corresponding pixel points of each pair of overlapping regions, represents the gray information entropy difference value of the corresponding pixel points of each pair of overlapping regions, represents the horizontal gradient difference of the content structure of each pair of overlapping regions, represents the vertical gradient difference value of each pair of overlapping regions, successively represent the weighting coefficients of the multi-index fusion energy function.

5. The image processing system for a suspended spherical screen panoramic vision according to claim 4, characterized in that The steps for obtaining the dynamically pixel stitching line in the overlapping area further include: B4. Based on the multi-index fusion energy function and the stitching seam dynamic adjustment factor, construct the search function D and the constraint conditions for the dynamically pixel stitching line in the overlapping area, specifically: ， Among them, represents the pixel points within the overlapping area, and (x, y) are the coordinates of the pixel points. represents the dynamic adjustment factor of the splicing seam, and & represents the sum symbol; B5. Based on the gray information entropy difference value extracted from the overlapping area, calculate the point with the smallest gray information entropy difference value of all row pixel points in the overlapping area as the search starting point. B6. Input the obtained search starting point, search function D and constraint conditions into the dynamic programming search algorithm, and in each step of the search process in the overlapping area, set the energy values of 5 pixel points in the adjacent columns corresponding to one row interval from the previous optimal point as the consideration basis, obtain the point with the smallest energy value in the current search step, take it as the search result of this step, and then add the search result to the optimal point set of the dynamically pixel stitching line. B7. Set the row where the search starting point is located as the starting row, and search in both the up and down directions. When the search reaches the last row in both directions, the search ends, and output the dynamically pixel stitching line in the overlapping area constructed by the obtained optimal point set.

6. The image processing system for a suspended spherical screen panoramic vision according to claim 5, characterized in that, The steps for obtaining the corrected projection image sequence include: C1. Let the coordinates of the pixel point of the j-th discrimination target in the i-th image of the current projection be (x, y), and set the plane projection surface of the i-th image corresponding to the projection area i as and coincide with the corresponding projected plane image; C2. Obtain the radius R and the center position C corresponding to the current projection area in the spherical screen, and establish a world coordinate system with the center of the sphere as the coordinate point. Let be a coordinate of a plane projection point on . Obtain the intersection point coordinate between the connection line between and the center position C and the current projection area, and obtain the surface projection point of and the corresponding projection geometric distortion function; C3. Repeat step C2 to obtain all the curved surface projection points corresponding to the current projection area and the corresponding projection geometric distortion functions, and according to the sum of all the obtained projection geometric distortion functions, solve for the optimal geometric projection angle and the corresponding center position through the particle swarm algorithm to perform real-time optimization on the image geometric projection. C4. Input the projection image into the radial distortion and tangential distortion models to perform real-time optical correction on the optical distortion of the projection image. C5. Repeat C3 and C4 to perform geometric correction and optical correction on all projection images at the same moment to obtain the geometrically corrected projection curved surface image.

7. The image processing system for a suspended spherical screen panoramic vision according to claim 6, wherein The steps for obtaining the dynamically pixel stitching line of the secondary correction include: D1. Obtain the optimal geometric projection angle and the corresponding center position in C3 and the brightness projection parameters corresponding to the overlapping area after optical correction in C4, input the obtained parameters and the corresponding dynamically pixel stitching line in the overlapping area into the dynamic programming search algorithm for secondary search correction to obtain the dynamically pixel stitching line of the secondary correction. D2. Use the dynamic pixel stitching line with secondary correction to stitch the overlapping region image pairs, and at the same time, replace the pixel values corresponding to the stitched overlapping region with the pixel average values at the corresponding pixel positions in the overlapping region image pairs; D3. Repeat the processes of D1 and D2, search for overlapping region image pairs for stitching, and obtain a complete panoramic visual image projection; D4. The segmented moving target region anchor boxes that have undergone geometric and optical correction are fused into the panoramic visual image projection through a fusion mapping function to obtain a dynamic panoramic visual image projection. , and the fusion mapping function is specifically: ， Among them, represents the spliced complete panoramic visual image projection, represents the corresponding mask area, represents and the matching fusion coefficient of the corresponding boundary.

8. An image processing method for a suspended spherical screen panoramic vision, which is implemented based on the image processing system for a suspended spherical screen panoramic vision described in any one of claims 1-7, and is characterized in that the steps Including: S1. Obtain the property information of the spherical screen and the projection device, construct a spherical screen-projection integrated three-dimensional model, divide the projection area on the spherical screen three-dimensional model, and correspond the projection area to the projection device one by one; S2. Obtain the image sequences at different angles of the panoramic visual scene, obtain the overlapping regions between different images through the feature point recognition algorithm, construct overlapping region image pairs, and input the overlapping region image pairs and the corresponding YUV chrominance space pairs into the convolutional neural network to obtain the overlapping region feature difference values; S3. Use the obtained overlapping region feature difference values and the stitching seam dynamic adjustment factor, and through the dynamic programming search algorithm, obtain the corresponding dynamic pixel stitching line within the overlapping region; S4. Construct a projection mapping function, use the projection mapping function to perform geometric correction and optical correction on the image sequences projected onto the corresponding projection areas, and at the same time, perform real-time dynamic stitching on the projected image sequences using the dynamically pixel stitching line with secondary correction corrected by the stitching correction factor.

9. A computer-readable storage medium, characterized in that, It stores computer instructions, and when the computer instructions run, it executes the image processing method of a suspended spherical screen panoramic vision described in claim 8.

Images