Suspension type spherical screen panoramic vision image processing system and method

Abstract

The invention belongs to the technical field of spherical screen image processing, and particularly relates to an image processing system and method for suspension type spherical screen panoramic vision, and the system obtains the attribute and position information of a spherical screen and a plurality of projection devices through a spherical screen modeling module, constructs a spherical screen-projection integrated three-dimensional model, and precisely divides projection areas to correspond to the devices; the image processing module collects image sequences of different angles of the panoramic vision scene; the image splicing module uses feature point detection, a YOLO algorithm and a convolutional neural network to identify an overlapping region and discriminate a dynamic region, and obtains a dynamic pixel splicing line through a dynamic planning search algorithm; the image correction module constructs a projection mapping function, corrects geometric and optical distortion of the image, and optimizes a splicing effect through a splicing correction function; and finally, the multi-view output module projects the corrected image sequence to a corresponding area of the dome screen by using a plurality of projection devices, so that real-time dynamic splicing and monitoring are realized, and projection synchronization and image quality are ensured.

Description

Technical Field

[0001] The present invention belongs to the technical field of ball screen image processing, and in particular relates to an image processing system and method for a suspended ball screen panoramic vision. Background Art

[0002] As a new type of immersive display technology, the suspended dome panoramic vision system projects 360-degree panoramic images on a hemispherical or spherical screen. However, due to the particularity of spherical projection, if the overlapping areas are not handled properly during the projection process, obvious seams are likely to be produced, affecting the visual effect. In addition, the color calibration between different projectors is inconsistent, resulting in uneven image color on the entire dome. In addition to the above-mentioned distortion, there are also optical distortions, such as barrel distortion and pincushion distortion, which will cause distortion in the final output image.

[0003] For example, a Chinese patent application with publication number CN108776951A discloses an image processing method for an LED ball screen display. According to the actual size of the LED ball screen display and the size of the source image supported by the playback system, a target image is obtained that has the same visual effect as that of a flat screen after being played on the ball screen display. The target image is expanded and projected on a three-dimensional hemisphere, and the corresponding latitude pixels on the target image are cyclically processed through mathematical formulas such as ellipse formulas and trigonometric functions to obtain a flat image. The flat image is then compared with the source image to finally obtain an image suitable for playback on the LED ball screen, and the suitable image is played on the LED ball screen through the playback system.

[0004] For example, a Chinese patent application with publication number CN115760555A discloses a method and device for synthesizing a spherical screen image, which creates four orthogonal camera channels; renders a three-dimensional scene into four camera channel images; converts each camera channel image into a local spherical screen image based on a mapping relationship between the plane coordinates of a single-channel camera image and the coordinates of a local spherical screen image; generates a mask template corresponding to each camera channel image; and synthesizes all local spherical screen images and the mask template to obtain a spherical screen image.

[0005] The above-mentioned prior art has the following problems: when the cameras lack concentric circle alignment, the prior art is prone to produce obvious stitching seams, stitching misalignment or color unevenness, and when a dynamic object passes through the stitching area, it is easy to cause stitching misalignment; for this reason, the present invention provides an image processing system and method for a suspended spherical screen panoramic vision. Summary of the invention

[0006] In view of the shortcomings of the prior art, the present invention proposes an image processing system and method for a suspended spherical screen panoramic vision. The system obtains the properties and position information of the spherical screen and multiple projection devices through a spherical screen modeling module, constructs a spherical screen-projection integrated three-dimensional model, and accurately divides the projection area and the corresponding equipment; the image processing module collects image sequences at different angles of the panoramic vision scene; the image stitching module uses feature point detection, YOLO algorithm and convolutional neural network to identify overlapping areas and distinguish dynamic areas, and obtains dynamic pixel stitching lines through a dynamic planning search algorithm; the image correction module constructs a projection mapping function to correct the geometric and optical distortion of the image, and optimizes the stitching effect through a stitching correction function; finally, the multi-view output module uses multiple projection devices to project the corrected image sequence to the corresponding area of ​​the spherical screen, realizes real-time dynamic stitching and monitoring, and ensures projection synchronization and image quality; the present invention provides an accurate image processing solution for the suspended spherical screen panoramic vision, and solves the problem of easy misalignment of dynamic display.

[0007] To achieve the above object, the present invention provides the following technical solutions: An image processing system for a suspended spherical screen panoramic vision comprises: a spherical screen modeling module, an image processing module and an image stitching module; the spherical screen modeling module comprises a three-dimensional model building unit and a projection area division unit; the spherical screen attribute information and the attribute information and viewing angle information of a plurality of projection devices are obtained, and the information is input into the three-dimensional model building unit to obtain a spherical screen-projection integrated three-dimensional model, and the corresponding projection area is obtained and numbered on the spherical screen three-dimensional model according to the viewing angle area information of the projection device; The image processing module collects image sequences of different angles of the same panoramic visual scene in different time dimensions and labels them; the image stitching module includes an overlapping area recognition unit, a dynamic area discrimination and segmentation unit, a feature extraction unit and a stitching seam search and optimization unit; The labeled image sequence is input into the overlapping area recognition unit to perform overlapping area recognition and judgment to obtain overlapping area image pairs. At the same time, the labeled image sequence is input into the dynamic area discrimination and segmentation unit to perform dynamic area judgment and dynamic area moving path prediction to obtain the dynamic adjustment factor of the splicing seam. The overlapping area image pairs and the corresponding YUV chromaticity space pairs are input into the feature extraction unit to obtain overlapping area feature difference values. The overlapping area feature difference values ​​and the dynamic adjustment factor of the splicing seam are input into the splicing seam search optimization unit to obtain the dynamic pixel splicing line of the overlapping area through a dynamic programming search algorithm.

[0008] Specifically, the image processing system also includes an image correction module and a multi-view output module; The image correction module includes an image correction unit and a stitching correction unit; the image pairs of overlapping areas at the same time are input into the image correction unit, and geometric and optical corrections are performed through the configured projection mapping function to obtain a corrected projection image sequence; at the same time, the dynamic pixel stitching line of the overlapping area is input into the stitching correction unit, and the dynamic pixel stitching line of secondary correction is obtained after correction through the configured stitching correction factor; The corrected projection image sequence is projected to the corresponding projection area through the multi-view output module and the annotated numbers, and the projection image is dynamically spliced ​​and monitored and evaluated in real time through the secondary corrected dynamic pixel splicing line, and the evaluation results are fed back to the projection control unit and the splicing seam search optimization unit to adjust the projection device angle and the overlapping area dynamic pixel splicing line parameters in real time.

[0009] Specifically, the steps of obtaining the dynamic adjustment factor of the splicing seam include: A1. Build and train a dynamic detection segmentation model based on the YOLO-tiny algorithm, input the labeled image sequences of the same panoramic visual scene at different angles into the dynamic detection segmentation model, and obtain the target area anchor frames of the same discriminated target in the image sequence of the same panoramic visual scene at different angles. ,in represents the regional anchor box of the jth discriminant target in the i-th image under the same panoramic visual scene, Represents the maximum and minimum values ​​of the horizontal and vertical coordinate points of the regional anchor box corresponding to the discriminant target; A2. Input the target region anchor frames in the acquired images at different angles into the optical flow algorithm and combine them with the built-in discrimination formula to determine whether each target region anchor frame is a moving target region anchor frame, and segment the moving target region anchor frame from the corresponding image to obtain the moving target region anchor frame and the corresponding moving direction and the overlapping region image pair including the mask region; A3, according to the obtained moving target region anchor frame and the corresponding moving direction and the overlapping region in the images of different angles under the same panoramic visual scene obtained by the overlapping region identification unit, through the Kalman filter algorithm, determine whether the moving target region anchor frame intersects with the overlapping region or will intersect with the overlapping region in the future; A4. If they intersect, the corresponding moving target area anchor frame is retained to construct the dynamic adjustment factor of the splicing seam. If they do not intersect, it is not used to construct the dynamic adjustment factor of the splicing seam.

[0010] Specifically, the built-in discriminant formula in A2 is: , in, is the feature point extracted by the optical flow algorithm from the i-th image in the same panoramic visual scene, Represents the set of feature points in the anchor box of the region that discriminates the jth target in the i-th image, Represents the motion vector angle of the kth feature point in the regional anchor frame of the jth discriminant target in the i-th image, Represents the mean value of the motion vector angles of all feature points in the anchor box of the j-th discriminant target in the i-th image; Represents the modulus of the kth feature point in the region anchor frame of the jth discriminant target in the i-th image and the kth feature point in the region anchor frame of the jth discriminant target in the i+1-th image.

[0011] Specifically, the steps of obtaining a dynamic pixel stitching line in the overlapping area include: B1, obtaining an overlapping area YUV chromaticity space pair by color space transformation of the overlapping area image pair obtained by the overlapping area recognition unit; B2, inputting the overlapping area image pair and the corresponding overlapping area YUV chromaticity space pair into the pyramid feature extraction model, extracting the overlapping area brightness difference value, chromaticity difference value and overlapping area content structure horizontal gradient difference and vertical gradient difference value; B3. Based on the brightness difference value, chromaticity difference value of the overlapping area and the horizontal gradient difference and vertical gradient difference value of the content structure of the overlapping area, a multi-index fusion energy function E is constructed, which is: ,in, Represents the chromaticity difference value of each pair of corresponding pixels in the overlapping area, Represents the grayscale information entropy difference value of each pair of corresponding pixels in the overlapping area, Indicates the horizontal gradient difference of the content structure of each pair of overlapping regions, Represents the vertical gradient difference value of each pair of overlapping areas, They represent the weighted coefficients of the multi-index fusion energy function in turn.

[0012] Specifically, the step of obtaining the dynamic pixel stitching line of the overlapping area also includes: B4. Based on the multi-index fusion energy function and the dynamic adjustment factor of the stitching seam, the search function D and the restriction conditions of the dynamic pixel stitching line in the overlapping area are constructed, specifically: , in, Represents the pixel point in the overlapping area, (x, y) is the coordinate of the pixel point, represents the dynamic adjustment factor of the joint seam, & represents the sum symbol; B5. Based on the grayscale information entropy difference value extracted from the overlapping area, the point with the smallest grayscale information entropy difference value of all rows of pixels in the overlapping area is calculated as the search starting point; B6, input the obtained search starting point, search function D and restriction conditions into the dynamic programming search algorithm, and in each step of the search process of the overlapping area, set the energy values ​​of the five pixel points in the adjacent column that are separated by one row from the corresponding pixel point of the previous optimal point as the consideration basis, obtain the point with the minimum energy value in the current search step, take it as the search result of this step, and then add the search result to the optimal point set of the dynamic pixel stitching line; B7. Set the row where the search starting point is located as the starting row, and search in both the upper and lower directions. The search ends when the last row is searched in both the upper and lower directions. Output the dynamic pixel stitching line of the overlapping area constructed by the optimal point set obtained by the search.

[0013] Specifically, the steps of acquiring the corrected projection image sequence include: C1. Let the coordinates of the pixel point of the jth discrimination target of the i-th image of the current projection be (x, y), and set the plane projection surface of the i-th image corresponding to the projection area i to be ,and coincide with the corresponding projected plane image; C2. Get the radius R and the center position C of the current projection area in the spherical screen, and establish a world coordinate system with the center of the sphere as the coordinate point. for The coordinates of a plane projection point on The intersection coordinates of the line connecting the center position C and the current projection area are obtained. The surface projection points and the corresponding projection geometric distortion function; C3, repeat step C2 to obtain all the surface projection points and the corresponding projection geometric distortion functions corresponding to the current projection area, and according to the sum of all the obtained projection geometric distortion functions, use the particle swarm algorithm to solve the optimal geometric projection angle and the corresponding circle center position, and optimize the image geometric projection in real time; C4, inputting the projection image into the radial distortion and tangential distortion models to perform real-time optical correction on the optical distortion of the projection image; C5. Repeat C3 and C4 to perform geometric correction and optical correction on all projection images at the same time to obtain a projection curved surface image after geometric correction.

[0014] Specifically, the steps of obtaining the secondary corrected dynamic pixel stitching line include: D1, obtaining the optimal geometric projection angle and the corresponding circle center position in C3 and the brightness projection parameters corresponding to the overlapping area after optical correction in C4, inputting the obtained parameters and the dynamic pixel stitching line of the corresponding overlapping area into the dynamic programming search algorithm for secondary search correction, and obtaining the secondary corrected dynamic pixel stitching line; D2, stitching the overlapping area image pair using the secondary corrected dynamic pixel stitching line, and replacing the pixel point values ​​corresponding to the stitched overlapping area with the pixel average value of the corresponding pixel position in the overlapping area image pair; D3, repeat the process of D1 and D2, search for overlapping area images to stitch, and obtain a complete panoramic visual image projection; D4. The anchor frame of the segmented moving target area that has been geometrically and optically corrected is fused to the panoramic visual image projection through a fusion mapping function to obtain a dynamic panoramic visual image projection. , the fusion mapping function is specifically: , in, Represents the complete panoramic visual image projection stitched together, Representation and The corresponding mask area, express and The matching fusion coefficient corresponding to the boundary.

[0015] A method for processing an image of a suspended spherical screen panoramic vision, comprising the following steps: S1. Acquire the property information of the ball screen and the projection device, construct a ball screen-projection integrated three-dimensional model, divide the projection area on the ball screen three-dimensional model, and correspond the projection area to the projection device one by one; S2, obtaining image sequences of different angles of the panoramic visual scene, obtaining the overlapping areas between different images through a feature point recognition algorithm, constructing overlapping area image pairs, inputting the overlapping area image pairs and the corresponding YUV color space pairs into a convolutional neural network, and obtaining the overlapping area feature difference values; S3, using the obtained overlapping area feature difference value and the dynamic adjustment factor of the stitching seam, through a dynamic programming search algorithm, to obtain the corresponding dynamic pixel stitching line in the overlapping area; S4. Construct a projection mapping function, use the projection mapping function to perform geometric correction and optical correction on the image sequence projected to the corresponding projection area, and use the dynamic pixel stitching line corrected by the stitching correction factor to perform real-time dynamic stitching on the projected image sequence.

[0016] A computer-readable storage medium stores computer instructions, which, when executed, execute an image processing method for a suspended spherical screen panoramic vision.

[0017] Compared with the prior art, the present invention has the following beneficial effects: 1. In view of the deficiencies in the prior art, the present invention uses a three-dimensional model construction unit and a projection area division unit to accurately obtain the attribute information of the ball screen and the projection device, and construct a high-precision ball screen-projection integrated three-dimensional model, thereby ensuring that the projection area division of the projection device on the ball screen is more reasonable; secondly, the dynamic area discrimination and segmentation unit uses the YOLO algorithm and the streamer algorithm to accurately identify and predict the moving path of the dynamic area, and generate a dynamic adjustment factor for the splicing seam, so that the system can adjust the splicing seam in real time, avoiding the splicing misalignment problem caused by dynamic objects.

[0018] 2. In view of the shortcomings of the prior art, the present invention converts the overlapping area image pairs into YUV chromaticity space pairs through a feature extraction unit, and inputs them into a pyramid convolutional neural network to obtain the brightness, chromaticity and content structure difference features, and combines the dynamic adjustment factor of the stitching seam to obtain the optimal dynamic pixel stitching line through a dynamic programming search algorithm, which not only improves the stitching accuracy, but also ensures the color uniformity and structural consistency of the stitching area.

[0019] 3. In view of the deficiencies of the prior art, the present invention performs geometric and optical correction on the projected image through the projection mapping function configured by the image correction unit and the stitching correction unit, thereby ensuring the accuracy and clarity of the projected image on the spherical screen; at the same time, the multi-view output module performs real-time dynamic stitching of the projected image through the dynamic pixel stitching line of secondary correction, and feeds back the evaluation results to the projection control unit and the stitching seam search optimization unit, thereby realizing real-time adjustment of the projection device angle and the dynamic pixel stitching line parameters of the overlapping area; this closed-loop feedback mechanism ensures the long-term stability and high precision of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a module diagram of an image processing system for a suspended spherical screen panoramic vision according to Embodiment 1 of the present invention; Figure 2 This is a flowchart of the corresponding units of an image processing system for a suspended spherical screen panoramic vision according to Embodiment 2 of the present invention; Figure 3 This is a flow chart of an image processing method for a suspended spherical screen panoramic vision according to embodiment 3 of the present invention. DETAILED DESCRIPTION

[0021] Example 1 See also Figure 1 , an embodiment provided by the present invention: an image processing system for a suspended spherical screen panoramic vision, comprising: a spherical screen modeling module, an image processing module, an image stitching module, an image correction module and a multi-view output module; The spherical screen modeling module is used to obtain the spherical screen attribute information and the attribute information and position information of multiple projection devices to construct a spherical screen projection three-dimensional model; the spherical screen modeling module includes a three-dimensional model construction unit, a projection area division unit and a projection control unit; The three-dimensional model construction unit is used to obtain a three-dimensional model of a spherical screen-projection integration through a three-dimensional model generation algorithm according to the spherical screen attribute information and the attribute information and viewing angle information of the plurality of projection devices; Furthermore, in this embodiment, the ball screen-projection integrated three-dimensional model includes a ball screen three-dimensional model and a projection device three-dimensional model; The projection area division unit is used to divide the spherical screen three-dimensional model in the spherical screen-projection integrated three-dimensional model into different projection areas according to the projection viewing angle area information of the projection device on the spherical screen, and to correspond each projection area number to the corresponding projection device one by one; Furthermore, in this embodiment, the projection area is divided into corresponding projection areas according to the projection viewing area information of the projection device on the spherical screen and the 3D model information of the spherical screen through a genetic algorithm; The projection control unit is used to adjust the projection angle of the projection device according to the image projection information, and monitor whether the images projected by each projection device at the same time are synchronized. If not, the projection synchronization adjustment is performed; Furthermore, in this embodiment, the projection device is configured with a position control model to control the angle and height of the projection device, and the position control model is constructed by the SAC algorithm.

[0022] The image processing module is used to collect image sequences of different angles of the same panoramic visual scene in different time dimensions, and perform enhancement and denoising preprocessing on the collected image sequences, and label them at the same time; This process can accurately establish a spherical screen-projection integrated 3D model according to the actual parameters of the spherical screen and projection equipment through the 3D model construction unit, ensuring the accuracy and practicality of the model, and providing a solid foundation for the subsequent projection area division. Secondly, the projection area division unit uses genetic algorithms for intelligent division, which not only improves the division efficiency, but also ensures the best match between each projection area and the projection equipment, effectively avoids the problems of overlap and blind spots, and enhances the realism and immersion of the projection effect. Furthermore, the projection control unit uses the position control model constructed by the SAC algorithm to dynamically adjust the angle and height of the projection equipment, ensure the synchronization of the projected images between multiple devices, solve the problem of inconsistent projection caused by differences between devices, and greatly improve the stability and reliability of the system. Finally, the image processing module significantly improves the image quality by enhancing and denoising the panoramic visual scene image sequence under different time dimensions, making the final picture presented to the audience clearer.

[0023] The image stitching module is used to stitch the panoramic image after enhancement processing; the image stitching module includes an overlapping area recognition unit, a dynamic area discrimination and segmentation unit, a feature extraction unit and a stitching seam search and optimization unit; The overlapping region identification unit is used to identify the corresponding overlapping regions between different images based on the planar images of the same panoramic visual scene at different angles through a feature point detection algorithm, and to construct overlapping region image pairs by grouping the overlapping region images in pairs; The dynamic area discrimination and segmentation unit is used to identify and determine whether there is a dynamic area based on the picture with overlapping areas through the YOLO algorithm, and predict the moving path of the dynamic area through the streamer algorithm to obtain the dynamic adjustment factor of the splicing seam; Furthermore, in this embodiment, the step of obtaining the dynamic adjustment factor of the joint seam includes: A1. Build and train a dynamic detection segmentation model based on the YOLO-tiny algorithm, input the labeled image sequences of the same panoramic visual scene at different angles into the dynamic detection segmentation model, and obtain the target area anchor frames of the same discriminated target in the image sequence of the same panoramic visual scene at different angles. ,in represents the regional anchor box of the jth discriminant target in the i-th image under the same panoramic visual scene, Represents the maximum and minimum values ​​of the horizontal and vertical coordinate points of the regional anchor box corresponding to the discriminant target; A2. Input the target region anchor frames in the acquired images at different angles into the optical flow algorithm and combine them with the built-in discrimination formula to determine whether each target region anchor frame is a moving target region anchor frame, and segment the moving target region anchor frame from the corresponding image to obtain the moving target region anchor frame and the corresponding moving direction and the overlapping region image pair including the mask region; Furthermore, the discriminant formula in this embodiment is: , in, is the feature point extracted by the optical flow algorithm from the i-th image in the same panoramic visual scene, Represents the set of feature points in the anchor box of the region that discriminates the jth target in the i-th image, Represents the motion vector angle of the kth feature point in the regional anchor frame of the jth discriminant target in the i-th image, Represents the mean value of the motion vector angles of all feature points in the anchor box of the j-th discriminant target in the i-th image; Represents the modulus of the k-th feature point in the regional anchor frame of the j-th discriminant target in the i-th image and the k-th feature point in the regional anchor frame of the j-th discriminant target in the i+1-th image; Furthermore, the meaning of the above formula in this embodiment is: if there is a target point selected by the optical flow in the target area anchor frame, and the motion vector direction of the target point is Similar or identical and modulus length If the distance is greater than one pixel, the target area is determined to be a moving target area anchor frame.

[0024] A3, according to the obtained moving target region anchor frame and the corresponding moving direction and the overlapping region in the images of different angles under the same panoramic visual scene obtained by the overlapping region identification unit, through the Kalman filter algorithm, determine whether the moving target region anchor frame intersects with the overlapping region or will intersect with the overlapping region in the future; A4. If they intersect, the corresponding moving target area anchor frame is retained to construct the dynamic adjustment factor of the splicing seam. If they do not intersect, it is not used to construct the dynamic adjustment factor of the splicing seam.

[0025] The process first uses an advanced feature point detection algorithm through the overlapping area recognition unit, which can quickly and accurately find the overlapping parts between different images in a complex environment; secondly, the dynamic area discrimination and segmentation unit introduces the YOLO algorithm to identify and separate dynamic objects, and combines the optical flow algorithm to predict their movement trajectories, thereby avoiding these dynamic elements from interfering with the stitching results and maintaining the consistency of the static background; in addition, the Kalman filter algorithm is used to analyze the moving target area, which can predict factors that may affect the stitching effect in advance, and then take corresponding measures to avoid them; finally, under the action of the stitching seam search optimization unit, the system can automatically find the best stitching solution to achieve a seamless connection effect.

[0026] The feature extraction unit is used to convert the overlapping area image pair into a YUV chromaticity space pair, and the overlapping area image pair and the corresponding YUV chromaticity space pair are input into a convolutional neural network to obtain the brightness difference value, chromaticity difference value and overlapping area structural difference feature of the pixel corresponding to each position point in the overlapping area; The seam search optimization unit is used to obtain the corresponding dynamic pixel stitching line in the overlapping area through a dynamic programming search algorithm according to the brightness difference value, chromaticity difference value of the pixels at the corresponding position in the overlapping area, the structural difference characteristics of the overlapping area and the dynamic adjustment factor of the seam.

[0027] Furthermore, in this embodiment, the step of obtaining a dynamic pixel stitching line in the overlapping area includes: B1, obtaining an overlapping area YUV chromaticity space pair by color space transformation of the overlapping area image pair obtained by the overlapping area recognition unit; B2, inputting the overlapping area image pair and the corresponding overlapping area YUV chromaticity space pair into the pyramid feature extraction model, extracting the overlapping area brightness difference value, chromaticity difference value and overlapping area content structure horizontal gradient difference and vertical gradient difference value; B3. Based on the brightness difference value, chromaticity difference value of the overlapping area and the horizontal gradient difference and vertical gradient difference value of the content structure of the overlapping area, a multi-index fusion energy function E is constructed, which is: ,in, Represents the chromaticity difference value of each pair of corresponding pixels in the overlapping area, Represents the grayscale information entropy difference value of each pair of corresponding pixels in the overlapping area, Indicates the horizontal gradient difference of the content structure of each pair of overlapping regions, Represents the vertical gradient difference value of each pair of overlapping areas, They represent the weighted coefficients of the multi-index fusion energy function in turn; B4. Based on the multi-index fusion energy function and the dynamic adjustment factor of the stitching seam, the search function D and the restriction conditions of the dynamic pixel stitching line in the overlapping area are constructed, specifically: , in, Represents the pixel point in the overlapping area, (x, y) is the coordinate of the pixel point, represents the dynamic adjustment factor of the joint seam, & represents the sum symbol; Furthermore, in this embodiment, the search function D searches for the pixel point that first corresponds to the corresponding fusion energy value, and the pixel point cannot be in the moving target area anchor frame corresponding to the dynamic adjustment factor of the splicing seam; and the pixel point can only be searched in the overlapping area.

[0028] B5. Based on the grayscale information entropy difference value extracted from the overlapping area, the point with the smallest grayscale information entropy difference value of all rows of pixels in the overlapping area is calculated as the search starting point; B6, input the obtained search starting point, search function D and restriction conditions into the dynamic programming search algorithm, and in each step of the search process of the overlapping area, set the energy values ​​of the five pixel points in the adjacent column that are separated by one row from the corresponding pixel point of the previous optimal point as the consideration basis, obtain the point with the minimum energy value in the current search step, take it as the search result of this step, and then add the search result to the optimal point set of the dynamic pixel stitching line; B7. Set the row where the search starting point is located as the starting row, and search in both the upper and lower directions. The search ends when the last row is searched in both the upper and lower directions. Output the dynamic pixel stitching line of the overlapping area constructed by the optimal point set obtained by the search.

[0029] The process first converts the overlapping area image pairs into YUV chromaticity space pairs through the feature extraction unit, and uses the convolutional neural network to extract the brightness difference, chromaticity difference and structural difference features. This step not only enhances the ability to capture image details, but also provides rich and accurate data support for the subsequent optimization of stitching seams. Secondly, the stitching seam search optimization unit is based on these difference features and the dynamic adjustment factor of the stitching seam. By constructing a multi-index fusion energy function E, it comprehensively considers multiple dimensions such as chromaticity, grayscale information entropy, horizontal and vertical gradients, ensuring that the selection of the stitching line takes into account both visual continuity and the influence of dynamic targets. In particular, through the dynamic programming search algorithm, starting from the point with the smallest grayscale information entropy difference, the optimal stitching line is searched line by line, which effectively avoids the stitching seam passing through the moving target area, reduces the stitching traces, and makes the stitching result smoother and more natural. This method not only solves the hard boundary problem existing in traditional stitching technology, but also improves the adaptability to dynamic scenes, greatly improving the overall quality of panoramic images and user experience.

[0030] The image correction module is used to construct a projection mapping function of a plane image and a curved spherical screen through a set algorithm, and correct the geometric distortion and optical distortion of the projection image through the projection mapping function; the image correction module includes an image correction unit and a splicing correction unit; The image correction unit is used to perform geometric correction and optical correction on the image projected onto the corresponding projection area through a configured projection mapping function to obtain a corrected projection image sequence; Furthermore, in this embodiment, the step of acquiring the corrected projection image sequence includes: C1. Let the coordinates of the pixel point of the jth discrimination target of the i-th image of the current projection be (x, y), and set the plane projection surface of the i-th image corresponding to the projection area i to be ,and coincide with the corresponding projected plane image; C2. Get the radius R and the center position C of the current projection area in the spherical screen, and establish a world coordinate system with the center of the sphere as the coordinate point. for The coordinates of a plane projection point on The intersection coordinates of the line connecting the center position C and the current projection area are obtained. The surface projection points and the corresponding projection geometric distortion function; C3, repeat step C2 to obtain all the surface projection points and the corresponding projection geometric distortion functions corresponding to the current projection area, and according to the sum of all the obtained projection geometric distortion functions, use the particle swarm algorithm to solve the optimal geometric projection angle and the corresponding circle center position, and optimize the image geometric projection in real time; C4, inputting the projection image into the radial distortion and tangential distortion models to perform real-time optical correction on the optical distortion of the projection image; C5. Repeat C3 and C4 to perform geometric correction and optical correction on all projection images at the same time to obtain a projection curved surface image after geometric correction.

[0031] The stitching correction unit is used to construct a stitching correction function according to the projection mapping function of the two images corresponding to the adjacent overlapping areas, obtain a stitching correction factor, and feed the stitching correction factor back to the stitching seam search optimization unit to perform geometric and optical correction on the obtained dynamic pixel stitching line to obtain a secondary corrected dynamic pixel stitching line; Furthermore, in this embodiment, the steps of obtaining the dynamic pixel stitching line after secondary correction include: D1, obtaining the optimal geometric projection angle and the corresponding circle center position in C3 and the brightness projection parameters corresponding to the overlapping area after optical correction in C4, inputting the obtained parameters and the dynamic pixel stitching line of the corresponding overlapping area into the dynamic programming search algorithm for secondary search correction, and obtaining the secondary corrected dynamic pixel stitching line; D2, stitching the overlapping area image pair using the secondary corrected dynamic pixel stitching line, and replacing the pixel point values ​​corresponding to the stitched overlapping area with the pixel average value of the corresponding pixel position in the overlapping area image pair; D3, repeat the process of D1 and D2, search for overlapping area images to stitch, and obtain a complete panoramic visual image projection; D4. The anchor frame of the segmented moving target area that has been geometrically and optically corrected is fused to the panoramic visual image projection through a fusion mapping function to obtain a dynamic panoramic visual image projection. , the fusion mapping function is specifically: , in, represents the stitching of the complete panoramic visual image projection, Representation and The corresponding mask area, express and The matching fusion coefficient corresponding to the boundary.

[0032] Furthermore, in this embodiment, the boundary matching fusion coefficient is obtained by corresponding boundary pixel feature points and a matching algorithm.

[0033] In this process, the image correction unit optimizes the geometric projection angle and the center position through the particle swarm algorithm, ensuring the geometric accuracy of the projected image on the spherical screen and eliminating the geometric distortion caused by the curved surface projection. At the same time, the application of radial distortion and tangential distortion models effectively corrects the image deformation caused by optical elements and ensures the clarity and realism of the image. The pixel value corresponding to the overlapping area after stitching is replaced by the pixel average value of the corresponding pixel position in the overlapping area image pair, which solves the problem of ghosting in the overlapping projection area and makes the transition of the stitching area more natural.

[0034] The stitching correction unit further optimizes the stitching effect by constructing a stitching correction function and using a dynamic programming search algorithm to perform secondary correction on the dynamic pixel stitching line, ensuring that the stitching line will not be deformed after projection and that the stitching between different projection areas is smooth and consistent; especially when processing dynamic targets, the corrected moving targets are reintegrated into the panoramic visual image through a fusion mapping function, which not only maintains the continuity of the dynamic content, but also improves the naturalness of the overall image; in addition, the introduction of the boundary matching fusion coefficient makes the transition between different areas more natural, reduces stitching traces, and enhances the user's immersive experience.

[0035] The multi-view output module is used to project image sequences corrected by the image correction unit at different angles of the same panoramic visual scene in different time dimensions to the corresponding projection area on the spherical screen through multiple projection devices, and use the acquired secondary corrected dynamic pixel stitching line to perform real-time dynamic stitching of the projected images, and monitor and acquire image projection information in real time, and feed back the acquired projection information to the projection control unit and the stitching seam search optimization unit to adjust the projection device angle and the dynamic pixel stitching line parameters of the overlapping area in real time.

[0036] Example 2

[0037] See also Figure 2 Another embodiment provided by the present invention is a corresponding unit workflow of an image processing system for a suspended spherical screen panoramic vision, comprising: First, a spherical screen-projection integrated three-dimensional model is constructed by a three-dimensional model construction unit, and on the spherical screen three-dimensional model in the constructed spherical screen-projection integrated three-dimensional model, a corresponding projection area is obtained and numbered by a projection area division unit; Second, the image processing module is used to obtain image sequences of different angles of the same panoramic visual scene in different time dimensions after enhancement and denoising, and the image sequences of different angles of the same panoramic visual scene are input into the overlapping area recognition unit to obtain overlapping area image pairs. At the same time, the image sequences of different angles of the same panoramic visual scene are input into the dynamic area discrimination and segmentation unit, and the dynamic adjustment factors of the dynamic area and the splicing seam are obtained by combining the YOLO algorithm with the streamer algorithm. Third, the overlapping area image pair is input into the feature extraction unit to obtain the brightness difference value, chromaticity difference value and overlapping area structural difference feature of the pixel corresponding to each position point in the overlapping area, and the brightness difference value, chromaticity difference value, overlapping area structural difference feature and the dynamic adjustment factor of the splicing seam corresponding to each position point in the overlapping area are input into the splicing seam search optimization unit to obtain the corresponding dynamic pixel splicing line in the overlapping area; Fourth, the image sequence of different angles of the same panoramic visual scene at the same time is corrected by the built-in projection mapping function of the image correction unit, and then projected into the corresponding projection area divided on the spherical screen through the multi-view output module. At the same time, the dynamic pixel stitching line between the image pairs in the overlapping area is corrected by the stitching correction factor obtained by the stitching correction unit, and the image projected into the projection area in real time is dynamically stitched using the corrected secondary correction dynamic pixel stitching line to obtain a complete full-vision projection image, and the projection deviation angle is obtained by real-time evaluation and detection, and the obtained projection deviation angle is fed back to the projection control unit to perform real-time synchronous adjustment of the projection angle of the projection device, and the projection evaluation result is fed back to the stitching seam search optimization unit to optimize the dynamic pixel stitching line parameters in real time.

[0038] Example 3

[0039] See also Figure 3 Another embodiment provided by the present invention is a method for processing an image of a suspended spherical screen panoramic vision, the steps comprising: S1. Acquire the property information of the ball screen and the projection device, construct a ball screen-projection integrated three-dimensional model, divide the projection area on the ball screen three-dimensional model, and correspond the projection area to the projection device one by one; S2, obtaining image sequences of different angles of the panoramic visual scene, obtaining the overlapping areas between different images through a feature point recognition algorithm, constructing overlapping area image pairs, inputting the overlapping area image pairs and the corresponding YUV color space pairs into a convolutional neural network, and obtaining the overlapping area feature difference values; Furthermore, in this embodiment, the overlapping region feature difference value includes a brightness difference value, a chromaticity difference value, and an overlapping region structure difference feature; S3, using the obtained overlapping area feature difference value and the dynamic adjustment factor of the stitching seam, through a dynamic programming search algorithm, to obtain the corresponding dynamic pixel stitching line in the overlapping area; S4, constructing a projection mapping function, using the projection mapping function to perform geometric correction and optical correction on the image sequence projected onto the corresponding projection area, and simultaneously using the dynamic pixel stitching line of secondary correction corrected by the stitching correction factor to perform real-time dynamic stitching on the projected image sequence; S5. Monitor and evaluate the image sequence stitching effect in real time, obtain the projection deviation angle, and feed back the obtained projection deviation angle to the configured projection device control subsystem to adjust the projection angle of the projection device in real time. At the same time, feed back the projection evaluation result to the dynamic programming search algorithm to optimize the parameters of the dynamic pixel stitching line.

[0040] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in the field may also change, modify, replace and modify the above-mentioned embodiments without departing from the scope of protection of the purpose of the present invention and the claims, and all of these are within the protection of the present invention.

[0041] If the disclosed technical solution involves personal information, the product using the disclosed technical solution has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the disclosed technical solution involves sensitive personal information, the product using the disclosed technical solution has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to the collection of his or her personal information; or on the device that processes personal information, when the personal information processing rules are notified by obvious signs / information, the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

Claims

1. A suspended spherical screen panoramic vision image processing system, characterized in that: include: Dome modeling module, image processing module and image stitching module; The ball screen modeling module includes a three-dimensional model building unit and a projection area division unit; Acquire the property information of the ball screen and the property information and viewing angle information of the plurality of projection devices, and input the property information and viewing angle information of the ball screen into the three-dimensional model construction unit to obtain the ball screen-projection integrated three-dimensional model, and obtain and label the corresponding projection area on the ball screen three-dimensional model according to the viewing angle area information of the projection device; The image processing module collects image sequences of different angles of the same panoramic visual scene in different time dimensions and labels them; the image stitching module includes an overlapping area recognition unit, a dynamic area discrimination and segmentation unit, a feature extraction unit and a stitching seam search and optimization unit; The labeled image sequence is input into the overlapping area recognition unit to perform overlapping area recognition and judgment to obtain overlapping area image pairs. At the same time, the labeled image sequence is input into the dynamic area discrimination and segmentation unit to perform dynamic area judgment and dynamic area moving path prediction to obtain the dynamic adjustment factor of the splicing seam. The overlapping area image pairs and the corresponding YUV chromaticity space pairs are input into the feature extraction unit to obtain overlapping area feature difference values. The overlapping area feature difference values ​​and the dynamic adjustment factor of the splicing seam are input into the splicing seam search optimization unit to obtain the dynamic pixel splicing line of the overlapping area through a dynamic programming search algorithm.

2. The image processing system for a suspended spherical screen panoramic vision according to claim 1, characterized in that: The image processing system also includes an image correction module and a multi-view output module; The image correction module includes an image correction unit and a stitching correction unit; the image pairs of overlapping areas at the same time are input into the image correction unit, and geometric and optical corrections are performed through the configured projection mapping function to obtain a corrected projection image sequence; at the same time, the dynamic pixel stitching line of the overlapping area is input into the stitching correction unit, and the dynamic pixel stitching line of secondary correction is obtained after correction through the configured stitching correction factor; The corrected projection image sequence is projected to the corresponding projection area through the multi-view output module and the annotated numbers, and the projection image is dynamically spliced ​​and monitored and evaluated in real time through the secondary corrected dynamic pixel splicing line, and the evaluation results are fed back to the projection control unit and the splicing seam search optimization unit to adjust the projection device angle and the overlapping area dynamic pixel splicing line parameters in real time.

3. The image processing system for a suspended spherical screen panoramic vision according to claim 2, characterized in that: The step of obtaining the dynamic adjustment factor of the splicing seam includes: A1. Build and train a dynamic detection segmentation model based on the YOLO-tiny algorithm, input the labeled image sequences of the same panoramic visual scene at different angles into the dynamic detection segmentation model, and obtain the target area anchor frames of the same discriminated target in the image sequence of the same panoramic visual scene at different angles. ,in represents the regional anchor box of the jth discriminant target in the i-th image under the same panoramic visual scene, Represents the maximum and minimum values ​​of the horizontal and vertical coordinate points of the regional anchor box corresponding to the discriminant target; A2. Input the target region anchor frames in the acquired images at different angles into the optical flow algorithm and combine them with the built-in discrimination formula to determine whether each target region anchor frame is a moving target region anchor frame, and segment the moving target region anchor frame from the corresponding image to obtain the moving target region anchor frame and the corresponding moving direction and the overlapping region image pair including the mask region; A3, according to the obtained moving target region anchor frame and the corresponding moving direction and the overlapping region in the images of different angles under the same panoramic visual scene obtained by the overlapping region identification unit, through the Kalman filter algorithm, determine whether the moving target region anchor frame intersects with the overlapping region or will intersect with the overlapping region in the future; A4. If they intersect, the corresponding moving target area anchor frame is retained to construct the dynamic adjustment factor of the splicing seam. If they do not intersect, it is not used to construct the dynamic adjustment factor of the splicing seam.

4. The image processing system for a suspended spherical screen panoramic vision according to claim 3, characterized in that: The discriminant formula built into A2 is: ， in, is the feature point extracted by the optical flow algorithm from the i-th image in the same panoramic visual scene, Represents the set of feature points in the anchor box of the region that discriminates the jth target in the i-th image, Represents the motion vector angle of the kth feature point in the regional anchor frame of the jth discriminant target in the i-th image, Represents the mean value of the motion vector angles of all feature points in the anchor box of the j-th discriminant target in the i-th image; Represents the modulus of the kth feature point in the region anchor frame of the jth discriminant target in the i-th image and the kth feature point in the region anchor frame of the jth discriminant target in the i+1-th image.

5. The image processing system for a suspended spherical screen panoramic vision according to claim 4, characterized in that: The step of obtaining a dynamic pixel stitching line in the overlapping area comprises: B1, obtaining an overlapping area YUV chromaticity space pair by color space transformation of the overlapping area image pair obtained by the overlapping area recognition unit; B2, input the overlapping area image pair and the corresponding overlapping area YUV chromaticity space pair into the pyramid feature extraction model, and extract the overlapping area brightness difference value, chromaticity difference value and overlapping area content structure horizontal gradient difference and vertical gradient difference value; B3. Based on the brightness difference value, chromaticity difference value of the overlapping area and the horizontal gradient difference and vertical gradient difference value of the content structure of the overlapping area, a multi-index fusion energy function E is constructed, which is: ,in, Represents the chromaticity difference value of each pair of corresponding pixels in the overlapping area, Represents the grayscale information entropy difference value of each pair of corresponding pixels in the overlapping area, Indicates the horizontal gradient difference of the content structure of each pair of overlapping regions, Represents the vertical gradient difference value of each pair of overlapping areas, They represent the weighted coefficients of the multi-index fusion energy function in turn.

6. The image processing system for a suspended spherical screen panoramic vision according to claim 5, characterized in that: The step of obtaining a dynamic pixel stitching line in the overlapping area also includes: B4. Based on the multi-index fusion energy function and the dynamic adjustment factor of the stitching seam, the search function D and the restriction conditions of the dynamic pixel stitching line in the overlapping area are constructed, specifically: ， in, Represents the pixel point in the overlapping area, (x, y) is the coordinate of the pixel point, represents the dynamic adjustment factor of the joint seam, & represents the sum symbol; B5. Based on the grayscale information entropy difference value extracted from the overlapping area, the point with the smallest grayscale information entropy difference value of all rows of pixels in the overlapping area is calculated as the search starting point; B6, input the obtained search starting point, search function D and restriction conditions into the dynamic programming search algorithm, and in each step of the search process of the overlapping area, set the energy values ​​of the five pixel points in the adjacent column that are separated by one row from the corresponding pixel point of the previous optimal point as the consideration basis, obtain the point with the minimum energy value in the current search step, use it as the search result of this step, and then add the search result to the optimal point set of the dynamic pixel stitching line; B7. Set the row where the search starting point is located as the starting row, and search in both the upper and lower directions. The search ends when the last row is searched in both the upper and lower directions. Output the dynamic pixel stitching line of the overlapping area constructed by the optimal point set obtained by the search.

7. The image processing system for a suspended spherical screen panoramic vision according to claim 6, characterized in that: The step of acquiring the corrected projection image sequence comprises: C1. Let the coordinates of the pixel point of the jth discrimination target of the i-th image of the current projection be (x, y), and set the plane projection surface of the i-th image corresponding to the projection area i to be ,and coincide with the corresponding projected plane image; C2. Get the radius R and the center position C of the current projection area in the spherical screen, and establish a world coordinate system with the center of the sphere as the coordinate point. for The coordinates of a plane projection point on The intersection coordinates of the line connecting the center position C and the current projection area are obtained. The surface projection points and the corresponding projection geometric distortion function; C3, repeat step C2 to obtain all the surface projection points and the corresponding projection geometric distortion functions corresponding to the current projection area, and according to the sum of all the obtained projection geometric distortion functions, use the particle swarm algorithm to solve the optimal geometric projection angle and the corresponding circle center position, and optimize the image geometric projection in real time; C4, inputting the projection image into the radial distortion and tangential distortion models to perform real-time optical correction on the optical distortion of the projection image; C5. Repeat C3 and C4 to perform geometric correction and optical correction on all projection images at the same time to obtain a projection curved surface image after geometric correction.

8. The image processing system for a suspended spherical screen panoramic vision according to claim 7, characterized in that: The step of obtaining the secondary corrected dynamic pixel stitching line comprises: D1, obtaining the optimal geometric projection angle and the corresponding circle center position in C3 and the brightness projection parameters corresponding to the overlapping area after optical correction in C4, inputting the obtained parameters and the dynamic pixel stitching line of the corresponding overlapping area into the dynamic programming search algorithm for secondary search correction, and obtaining the secondary corrected dynamic pixel stitching line; D2, stitching the overlapping area image pair using the secondary corrected dynamic pixel stitching line, and replacing the pixel point values ​​corresponding to the stitched overlapping area with the pixel average value of the corresponding pixel position in the overlapping area image pair; D3, repeat the process of D1 and D2, search for overlapping area images to stitch, and obtain a complete panoramic visual image projection; D4. The anchor frame of the segmented moving target area that has been geometrically and optically corrected is fused to the panoramic visual image projection through a fusion mapping function to obtain a dynamic panoramic visual image projection. , the fusion mapping function is specifically: ， in, Represents the complete panoramic visual image projection stitched together, Representation and The corresponding mask area, express and The matching fusion coefficient corresponding to the boundary.

9. A method for processing an image of a suspended spherical screen panoramic vision, which is implemented based on an image processing system for a suspended spherical screen panoramic vision according to any one of claims 1 to 8, characterized in that the steps include: S1. Acquire the property information of the ball screen and the projection device, construct a ball screen-projection integrated three-dimensional model, divide the projection area on the ball screen three-dimensional model, and correspond the projection area to the projection device one by one; S2, obtaining image sequences of different angles of the panoramic visual scene, obtaining the overlapping areas between different images through a feature point recognition algorithm, constructing overlapping area image pairs, inputting the overlapping area image pairs and the corresponding YUV color space pairs into a convolutional neural network, and obtaining the overlapping area feature difference values; S3, using the obtained overlapping area feature difference value and the dynamic adjustment factor of the stitching seam, through a dynamic programming search algorithm, to obtain the corresponding dynamic pixel stitching line in the overlapping area; S4. Construct a projection mapping function, use the projection mapping function to perform geometric correction and optical correction on the image sequence projected to the corresponding projection area, and use the dynamic pixel stitching line corrected by the stitching correction factor to perform real-time dynamic stitching on the projected image sequence.

10. A computer-readable storage medium, characterized in that: Computer instructions are stored thereon, and when the computer instructions are executed, the image processing method for the suspended spherical screen panoramic vision described in claim 9 is executed.

Images