Virtual scene dome line detection method and device, electronic equipment and storage medium

Abstract

The application particularly relates to a virtual view sky dome line detection method and device, electronic equipment and a storage medium, which comprises the following steps: obtaining a target image by performing down-sampling processing on an acquired virtual view original image and calculating a gradient amplitude graph; performing iterative calculation on the gradient amplitude graph to obtain an optimal initial boundary sequence, optimizing the optimal initial boundary sequence to obtain a sky dome line boundary sequence, and then performing polynomial fitting and global linear fitting on the sky dome line boundary sequence to obtain a final sky dome line; calculating a longitudinal coordinate mean value of the sky dome line boundary sequence and an absolute value mean value of a longitudinal coordinate difference value between adjacent column boundary points in the sky dome line boundary sequence; and in a case where it is determined according to the longitudinal coordinate mean value and the absolute value mean value that the virtual view original image has an effective sky region, generating a binary sky mask based on the final sky dome line, and performing up-sampling processing to obtain a final size of the virtual view original image.

Description

Technical Field

[0001] This application relates to the fields of digital image processing and computer vision technology, and in particular to a method, apparatus, electronic device and storage medium for detecting virtual visual dome lines. Background Technology

[0002] With the rapid development of technologies such as flight simulation training, airborne augmented reality, UAV autonomous navigation, and aerial electro-optical reconnaissance, high-precision matching and fusion of 3D virtual scenes and real aerial electro-optical images has become one of the core technologies in this field. The dome line, as the natural physical boundary between the sky and the ground / sea surface, is the most stable and globally representative spatial feature benchmark in an aerial scene. Its detection accuracy, continuity, and stability directly determine the registration and fusion effect between virtual scenes and real electro-optical images, and also have a decisive impact on the accuracy and reliability of downstream tasks such as attitude calculation, navigation and positioning, and scene matching for the flight platform.

[0003] In actual airborne application scenarios, the input aviation optoelectronic images and three-dimensional virtual scene images are easily interfered with by a variety of complex factors, which brings many difficulties to the accurate detection of the sky line: (1) Environmental factors such as sudden changes in illumination, cloud cover, haze scattering, and strong light reflection from the sea surface cause the grayscale, texture and gradient features of the sky and non-sky areas to be unstable, and the boundary between heaven and earth to be blurred; (2) Complex ground features such as buildings, mountains, and vegetation will produce strong edge textures, which can easily form false sky line features and seriously interfere with the extraction of the real sky boundary; (3) Airborne platforms mostly use domestic embedded computing units, which have limited computing power and storage resources, and put forward strict requirements on the computational complexity and real-time performance of the detection algorithm. Complex algorithms are difficult to deploy and implement on such platforms; (4) The sky line in the virtual scene presents a smooth and continuous straight line shape. The boundary output by the existing detection algorithm is mostly an irregular broken line, which cannot directly adapt to the matching and fusion requirements of the virtual scene. It is necessary to add post-processing optimization steps, which further increases the complexity of the algorithm.

[0004] In related technologies, skyline detection methods are mainly divided into three categories. The first category is traditional methods based on threshold segmentation, such as the Otsu's method and the region growing method. These methods have simple computational logic and good real-time performance, but they are extremely unreliable to changes in lighting, cloud interference, and complex terrain scenes, and are prone to problems such as boundary breaks and missegmentation. Their detection success rate in complex aviation scenes is extremely low, which cannot meet the needs of practical applications. The second category is edge detection methods, such as the Canny operator and the Sobel operator. These methods extract the skyline boundary through gradient features, but they are susceptible to salt-and-pepper noise and texture interference. The first type is highly sensitive to disturbances and easily affected by strong ground edges and cloud textures, resulting in a large number of false edges. It is difficult to accurately locate the real skyline and the continuity of the output boundary is poor, making it impossible to directly adapt to the linearization requirements of virtual scenes. The second type is detection methods based on deep learning. These methods learn the deep features of the skyline through convolutional neural networks, which have strong detection accuracy and anti-interference ability. However, the model has a large number of parameters, high computational complexity, and high inference latency. It cannot achieve real-time processing on domestic embedded platforms with limited computing power. In addition, the model is difficult to deploy and has poor portability, making it difficult to meet the engineering implementation requirements of airborne scenarios.

[0005] Therefore, all three types of celestial line detection methods mentioned above have shortcomings in adapting to the requirements of the aforementioned aviation virtual visual application scenarios, and these shortcomings urgently need to be addressed. Summary of the Invention

[0006] This application provides a virtual visual skyline detection method, device, electronic device, and storage medium to solve problems such as interference with registration accuracy due to differences in sky regions and difficulty in balancing the real-time performance and detection robustness of embedded platforms in the matching and fusion of aerial photoelectric images and three-dimensional virtual visual scenes.

[0007] The first aspect of this application provides a method for detecting the dome line in a virtual view, including the following steps: The original image of the virtual scene is acquired, and the original image of the virtual scene is downsampled to obtain the target image. The gradient magnitude map of the target image is then calculated. Based on a preset iterative gradient threshold range, the gradient magnitude map is iteratively calculated to obtain the optimal initial boundary sequence of the gradient magnitude map after iteration. The optimal initial boundary sequence is then optimized using a multi-level processing mechanism to obtain the dome line boundary sequence. Finally, the dome line boundary sequence is subjected to polynomial fitting and global linear fitting to obtain the final dome line. Calculate the mean of the longitudinal coordinates of the dome boundary sequence and the mean of the absolute values ​​of the differences in the longitudinal coordinates of adjacent column boundary points in the dome boundary sequence. If it is determined that there is a valid sky region in the original virtual scene image based on the mean of the longitudinal coordinates and the mean of the absolute values, generate a binarized sky mask based on the final dome and upsample the binarized sky mask to obtain the final size of the original virtual scene image.

[0008] According to one embodiment of this application, calculating the gradient magnitude map of the target image includes: The target image is converted to grayscale to obtain a target grayscale image; Calculate the first gradient component of the target grayscale image in the horizontal direction and the second gradient component of the target grayscale image in the vertical direction; The gradient magnitude of each pixel in the target grayscale image is calculated based on the first gradient component and the second gradient component to obtain the gradient magnitude map of the target image.

[0009] According to one embodiment of this application, the step of iteratively calculating the gradient magnitude map based on a preset iterative gradient threshold range to obtain the optimal initial boundary sequence of the iterated gradient magnitude map includes: Determine the iteration range, iteration step size, and total number of iterations for the gradient magnitude map; Based on the iteration range and the iteration step size, the gradient magnitude map is iteratively calculated column by column according to the total number of iterations, and the initial boundary points of the gradient magnitude map after each iteration are extracted column by column to obtain multiple initial boundary points of the target image; Based on the multiple initial boundary points, determine the optimal initial boundary sequence that satisfies the preset conditions.

[0010] According to one embodiment of this application, the step of optimizing the optimal initial boundary sequence using a multi-level processing mechanism to obtain the dome line boundary sequence includes: Determine the valid boundary points in the optimal initial boundary sequence; When the proportion of the effective boundary points in the multiple initial boundary points is greater than or equal to a first threshold, it is determined that there is an effective sky region in the optimal initial boundary sequence, and the longitudinal coordinates of the boundary points of the effective sky region are extracted. Based on the longitudinal coordinates, a set of effective point boundary points of the effective sky region is constructed. Calculate the median of all valid boundary points in the set of valid boundary points, and based on the median and the maximum permissible deviation threshold, mark the valid boundary points in the optimal initial boundary sequence whose vertical coordinates deviate from the median by more than the maximum permissible deviation threshold as outliers to obtain initial outlier values; The missing value column in the optimal initial boundary sequence is determined, and a first effective boundary point is found based on a first direction of the missing value column and a second effective boundary point is found based on a second direction of the missing value column. Based on the first effective boundary point and the second effective boundary point, the missing value column is filled using the missing value filling rule to obtain the complete initial boundary sequence. Based on the first direction of the initial outlier, non-outlier valid boundary points are found, and the initial outlier is corrected in position based on the non-outlier valid boundary points to obtain the dome line boundary sequence.

[0011] According to one embodiment of this application, the step of performing polynomial fitting and global linear fitting on the dome boundary sequence to obtain the final dome line includes: A polynomial fitting model is constructed, the polynomial coefficients of the polynomial fitting model are calculated using a preset least squares method, and the dome line boundary sequence is fitted with a polynomial based on the polynomial coefficients. Calculate the threshold value of the quantile outlier of the dome boundary sequence at the target location; Based on the fitting results and the quantile outlier threshold of the target location, the initial outlier values ​​are replaced with fitted values ​​to obtain the smoothed dome line fitting boundary. A polynomial linear fitting model is constructed, and a global linear fitting is performed on the fitting boundary of the dome line based on the polynomial linear fitting model to obtain the final dome line.

[0012] According to one embodiment of this application, the step of generating a binarized sky mask based on the final dome line and upsampling the binarized sky mask to obtain the final size of the original image of the virtual view includes: A single-channel mask matrix is ​​constructed based on the size of the target image, and all element values ​​within the single-channel mask matrix are initialized. Based on the final skyline, the single-channel mask matrix is ​​conditionally assigned pixel by pixel to generate a binary sky mask. The binarized sky mask is upsampled by a target factor to obtain the final size of the original virtual scene image.

[0013] According to the virtual scene skyline detection method of this application embodiment, the target image is obtained by downsampling the acquired virtual scene original image, and a gradient magnitude map is calculated; the gradient magnitude map is iteratively calculated to obtain the optimal initial boundary sequence, and the skyline boundary sequence is optimized to obtain the skyline boundary sequence; then, polynomial fitting and global linear fitting are performed on the skyline boundary sequence to obtain the final skyline; the mean of the vertical coordinates of the skyline boundary sequence and the mean of the absolute values ​​of the differences in the vertical coordinates of adjacent column boundary points in the skyline boundary sequence are calculated; if it is determined that there is a valid sky region in the virtual scene original image based on the mean of the vertical coordinates and the mean of the absolute values, a binarized sky mask is generated based on the final skyline, and upsampling processing is performed to obtain the final size of the virtual scene original image. This invention addresses the challenges in matching and fusing aerial photoelectric images with 3D virtual scenes, such as differences in sky regions interfering with registration accuracy and the difficulty in balancing real-time performance and detection robustness on embedded platforms. By iteratively extracting initial boundary points column by column through gradient thresholding, and combining a four-level robust optimization mechanism with a polynomial-linear hybrid fitting strategy, it achieves adaptive, highly robust detection and linearization optimization of celestial lines in virtual scene images.

[0014] A second aspect of this application provides a virtual visual dome detection device, comprising: The calculation module is used to acquire the original image of the virtual scene, perform downsampling processing on the original image of the virtual scene to obtain the target image, and calculate the gradient magnitude map of the target image; The fitting module is used to iteratively calculate the gradient magnitude map based on a preset iterative gradient threshold range to obtain the optimal initial boundary sequence of the gradient magnitude map after iteration, and to optimize the optimal initial boundary sequence using a multi-level processing mechanism to obtain the dome line boundary sequence. Finally, the dome line boundary sequence is subjected to polynomial fitting and global linear fitting to obtain the final dome line. The sampling processing module is used to calculate the mean of the longitudinal coordinates of the dome boundary sequence and the mean of the absolute values ​​of the differences in the longitudinal coordinates of adjacent column boundary points in the dome boundary sequence. If it is determined that there is a valid sky region in the original virtual scene image based on the mean of the longitudinal coordinates and the mean of the absolute values, a binarized sky mask is generated based on the final dome, and the binarized sky mask is upsampled to obtain the final size of the original virtual scene image.

[0015] According to one embodiment of this application, the computing module includes: A conversion unit is used to perform grayscale conversion on the target image to obtain a target grayscale image; The first calculation unit is used to calculate the first gradient component of the target grayscale image in the horizontal direction and the second gradient component of the target grayscale image in the vertical direction; The second calculation unit is used to calculate the gradient magnitude of each pixel in the target grayscale image based on the first gradient component and the second gradient component, so as to obtain the gradient magnitude map of the target image.

[0016] According to one embodiment of this application, the fitting module includes: The first determining unit is used to determine the iteration range, iteration step size, and total number of iterations of the gradient magnitude map; An extraction unit is used to iteratively calculate the gradient magnitude map column by column according to the total number of iterations based on the iteration range and the iteration step size, and extract the initial boundary points of the gradient magnitude map after each iteration to obtain multiple initial boundary points of the target image. The second determining unit is used to determine the optimal initial boundary sequence that satisfies preset conditions based on the plurality of initial boundary points.

[0017] According to one embodiment of this application, the fitting module includes: The third determining unit is used to determine the valid boundary points in the optimal initial boundary sequence; The first construction unit is used to determine that there is a valid sky region in the optimal initial boundary sequence when the proportion of the valid boundary points in the multiple initial boundary points is greater than or equal to a first threshold, and to extract the longitudinal coordinates of the boundary points of the valid sky region, and to construct a set of valid point boundary points of the valid sky region based on the longitudinal coordinates. An anomaly marking unit is used to calculate the median of all valid boundary points in the set of valid boundary points, and based on the median and the maximum permissible deviation threshold, mark the valid boundary points in the optimal initial boundary sequence whose vertical coordinates deviate from the median by more than the maximum permissible deviation threshold as anomalies to obtain initial anomaly values; A filling unit is used to determine the missing value column in the optimal initial boundary sequence, and to find a first valid boundary point based on a first direction of the missing value column and a second valid boundary point based on a second direction of the missing value column. Based on the first valid boundary point and the second valid boundary point, the missing value column is filled using a missing value filling rule to obtain a complete initial boundary sequence. The correction unit is used to find non-abnormal valid boundary points based on the first direction of the initial outlier, and to correct the position of the initial outlier based on the non-abnormal valid boundary points to obtain the dome line boundary sequence.

[0018] According to one embodiment of this application, the fitting module includes: The second construction unit is used to construct a polynomial fitting model, calculate the polynomial coefficients of the polynomial fitting model using a preset least squares method, and perform polynomial fitting on the dome line boundary sequence based on the polynomial coefficients. The third calculation unit is used to calculate the threshold value of the quantile outlier value of the dome boundary sequence at the target location; The fitting unit is used to replace the initial outlier with a fitted value based on the fitting result and the quantile outlier threshold of the target position, so as to obtain a smoothed celestial dome fitting boundary. The third building unit is used to build a polynomial linear fitting model and perform global linear fitting on the celestial dome fitting boundary based on the polynomial linear fitting model to obtain the final celestial dome.

[0019] According to one embodiment of this application, the sampling processing module includes: The fourth construction unit is used to construct a single-channel mask matrix based on the size of the target image and initialize all element values ​​within the single-channel mask matrix; The generation unit is used to perform pixel-by-pixel conditional assignment on the single-channel mask matrix according to the final skyline, and generate a binary sky mask. The sampling processing unit is used to perform upsampling processing on the binarized sky mask by a target multiple to obtain the final size of the original image of the virtual scene.

[0020] According to the virtual scene skyline detection device of this application embodiment, a target image is obtained by downsampling the acquired virtual scene original image, and a gradient magnitude map is calculated; the gradient magnitude map is iteratively calculated to obtain the optimal initial boundary sequence, and the skyline boundary sequence is optimized to obtain the skyline boundary sequence; then, polynomial fitting and global linear fitting are performed on the skyline boundary sequence to obtain the final skyline; the mean of the vertical coordinates of the skyline boundary sequence and the mean of the absolute values ​​of the differences in the vertical coordinates of adjacent column boundary points in the skyline boundary sequence are calculated; if it is determined that there is a valid sky region in the virtual scene original image based on the mean of the vertical coordinates and the mean of the absolute values, a binarized sky mask is generated based on the final skyline, and upsampling processing is performed to obtain the final size of the virtual scene original image. This invention addresses the challenges in matching and fusing aerial photoelectric images with 3D virtual scenes, such as differences in sky regions interfering with registration accuracy and the difficulty in balancing real-time performance and detection robustness on embedded platforms. By iteratively extracting initial boundary points column by column through gradient thresholding, and combining a four-level robust optimization mechanism with a polynomial-linear hybrid fitting strategy, it achieves adaptive, highly robust detection and linearization optimization of celestial lines in virtual scene images.

[0021] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the virtual view skyline detection method as described in the above embodiments.

[0022] A fourth aspect of this application provides a computer-readable storage medium storing computer instructions for causing the computer to perform the virtual visual skyline detection method as described in the above embodiments.

[0023] A fifth aspect of this application provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the virtual view skyline detection method as described in the above embodiments.

[0024] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0025] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is a flowchart of a virtual view skyline detection method according to an embodiment of this application; Figure 2 This is a flowchart illustrating the overall process of a virtual scene dome detection method based on gradient optimization and polynomial fitting according to an embodiment of this application. Figure 3 This is a schematic diagram of a gradient feature extraction process according to an embodiment of this application; Figure 4 This is a schematic diagram of a four-level processing mechanism for robust boundary optimization according to an embodiment of this application; Figure 5 This is a schematic diagram illustrating the effect of sky mask generation and result visualization according to an embodiment of this application; Figure 6 This is an example diagram of a virtual visual dome line detection device according to an embodiment of this application; Figure 7 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0026] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0027] The following describes a virtual scene celestial line detection method, apparatus, electronic device, and storage medium according to embodiments of this application with reference to the accompanying drawings. Addressing the issues mentioned in the background art regarding the interference of sky region differences on registration accuracy and the difficulty in balancing the real-time performance and detection robustness of embedded platforms in the matching and fusion of aerial photoelectric images and 3D virtual scenes, this application provides a virtual scene celestial line detection method. In this method, a target image is obtained by downsampling the acquired original virtual scene image, and a gradient magnitude map is calculated. The gradient magnitude map is iteratively calculated to obtain an optimal initial boundary sequence, which is then optimized to obtain a celestial line boundary sequence. Polynomial fitting and global linear fitting are then performed on the celestial line boundary sequence to obtain the final celestial line. The mean of the vertical coordinates of the celestial line boundary sequence and the mean absolute value of the difference in vertical coordinates between adjacent column boundary points in the celestial line boundary sequence are calculated. If a valid sky region exists in the original virtual scene image based on the mean of the vertical coordinates and the mean absolute value, a binarized sky mask is generated based on the final celestial line and upsampled to obtain the final size of the original virtual scene image. This invention addresses the challenges in matching and fusing aerial photoelectric images with 3D virtual scenes, such as differences in sky regions interfering with registration accuracy and the difficulty in balancing real-time performance and detection robustness on embedded platforms. By iteratively extracting initial boundary points column by column through gradient thresholding, and combining a four-level robust optimization mechanism with a polynomial-linear hybrid fitting strategy, it achieves adaptive, highly robust detection and linearization optimization of celestial lines in virtual scene images.

[0028] Specifically, Figure 1 This is a flowchart illustrating a virtual view skyline detection method provided in an embodiment of this application.

[0029] like Figure 1 As shown, the virtual view skyline detection method includes the following steps: In step S101, the original image of the virtual scene is acquired, and the original image of the virtual scene is downsampled to obtain the target image, and the gradient magnitude map of the target image is calculated.

[0030] According to one embodiment of this application, calculating the gradient magnitude map of a target image includes: performing grayscale conversion on the target image to obtain a target grayscale image; calculating a first gradient component of the target grayscale image in the horizontal direction and a second gradient component of the target grayscale image in the vertical direction; and calculating the gradient magnitude of each pixel in the target grayscale image based on the first gradient component and the second gradient component to obtain the gradient magnitude map of the target image.

[0031] Specifically, to address the technical problems of existing dome line detection methods in related technologies, such as the inability to simultaneously ensure detection robustness and real-time performance on embedded platforms, insufficient detection accuracy in complex aviation scenarios, and the inability to directly adapt detection results to the linearization requirements of virtual scenes, this application provides a dome line detection method with high detection accuracy, strong anti-interference capability, low computational complexity, real-time processing on domestic embedded platforms, and detection results directly adapted to the matching and fusion requirements of virtual scenes. For example, Figure 2 As shown, this application mainly includes eight steps: image preprocessing, gradient feature extraction, gradient threshold iteration and initial boundary extraction, robust boundary optimization, polynomial fitting correction, linearization optimization, sky region existence determination, sky mask generation and size restoration. The following describes each step in detail with reference to specific embodiments.

[0032] Specifically, firstly, the original image of the virtual scene is acquired; this original image is a color image with a resolution of 1920×1080. , The detection algorithm is implemented using C++ and the OpenCV library and can run on a domestic embedded platform (1.8GHz main frequency, 2GB memory); secondly, the input virtual scene original image... Preprocessing is performed, which involves applying a 2x Gaussian pyramid downsampling method. The main steps include: first, using a 5x5 normalized Gaussian kernel to check the original virtual scene image. After performing convolutional smoothing, the target image is downsampled by a factor of 2 through alternating rows and columns. ,in, , (For example, H=540, W=960) The target image is a color target image. While preserving the core gradient features of the image, the number of pixels is reduced to 1 / 4 of the original image of the virtual scene, thereby greatly reducing the subsequent computational complexity and laying the foundation for real-time processing on the embedded platform.

[0033] The mathematical expression for the 5×5 normalized Gaussian kernel is: ; The mathematical expression for convolution smoothing is: ; in, , These are the pixel coordinates of the original image. For the pixel value after convolution smoothing, the mathematical expression for 2x downsampling is: ; in, , These are the pixel coordinates of the downsampled target image. m This represents the offset of the Gaussian kernel along the row direction of the image. n This represents the offset of the Gaussian kernel along the column direction of the image.

[0034] Secondly, such as Figure 3 As shown in (a), gradient feature extraction of the target image mainly includes: grayscale processing, 3×3 Sobel operator, and gradient calculation, as follows: Figure 3 As shown in (b), the cv::cvtColor function is used to perform grayscale conversion on the downsampled color target image to eliminate the interference of color channels on gradient calculation and obtain a single-channel target grayscale image. The conversion formula is as follows: ; in, , , These are the pixel values ​​of the B, G, and R channels of the target image, respectively.

[0035] Again, such as Figure 3 As shown in (c), a 3×3 Sobel operator is used, and the first gradient component of the target grayscale image in the horizontal direction (x direction) is calculated by the cv::Sobel function. and the second gradient component of the target grayscale image in the vertical direction (y direction) The gradient maps in the x-direction were obtained respectively. Figure 3 (c) left side view) and y-direction gradient map ( Figure 3 (c) The gradient calculation uses the CV_64F64-bit floating-point data type to avoid gradient value overflow and ensure calculation accuracy.

[0036] Finally, as Figure 3 As shown in (d), using the cv::magnitude function, through the first gradient component Second gradient component Calculate the gradient magnitude of each pixel in the target grayscale image to obtain the single-channel gradient magnitude map of the target image. As the core input for subsequent boundary extraction, it enables accurate extraction of the gradient features of the sky and earth boundaries. The gradient magnitude calculation formula is as follows: .

[0037] Among them, the first gradient component in the horizontal direction The second gradient component in the vertical direction The calculation expression is: ; ; in, A horizontal Sobel kernel. Vertical Sobel kernel, horizontal Sobel kernel Sobel kernel in the vertical direction The mathematical expression is: , ; In step S102, based on the preset iterative gradient threshold range, the gradient magnitude map is iteratively calculated to obtain the optimal initial boundary sequence of the iterated gradient magnitude map. The optimal initial boundary sequence is then optimized using a multi-level processing mechanism to obtain the dome line boundary sequence. Finally, polynomial fitting and global linear fitting are performed on the dome line boundary sequence to obtain the final dome line.

[0038] According to one embodiment of this application, the gradient magnitude map is iteratively calculated based on a preset iterative gradient threshold range to obtain the optimal initial boundary sequence of the gradient magnitude map after iteration, including: determining the iteration range, iteration step size, and total number of iterations of the gradient magnitude map; iteratively calculating the gradient magnitude map column by column according to the iteration range and iteration step size and the total number of iterations, and extracting the initial boundary points of the gradient magnitude map after each iteration to obtain multiple initial boundary points of the target image; and determining the optimal initial boundary sequence that satisfies the preset conditions based on the multiple initial boundary points.

[0039] According to one embodiment of this application, a multi-level processing mechanism is used to optimize the optimal initial boundary sequence to obtain a dome line boundary sequence, including: determining the effective boundary points in the optimal initial boundary sequence; when the proportion of effective boundary points in multiple initial boundary points is greater than or equal to a first threshold, determining that there is an effective sky region in the optimal initial boundary sequence, and extracting the longitudinal coordinates of the boundary points of the effective sky region, and constructing a set of effective boundary points of the effective sky region based on the longitudinal coordinates; calculating the median of all effective boundary points in the set of effective boundary points, and marking the effective boundary points in the optimal initial boundary sequence whose longitudinal coordinates deviate from the median greater than the maximum permissible deviation threshold as anomalies based on the median and the maximum permissible deviation threshold, to obtain initial anomaly values; determining the missing value column in the optimal initial boundary sequence, and searching for a first effective boundary point based on a first direction of the missing value column and a second effective boundary point based on a second direction of the missing value column, and filling the missing value column using a missing value filling rule based on the first and second effective boundary points to obtain a complete initial boundary sequence; searching for non-anomaly effective boundary points based on the first direction of the initial anomaly values, and correcting the position of the initial anomaly values ​​based on the non-anomaly effective boundary points to obtain the dome line boundary sequence.

[0040] The preset range of iterative gradient thresholds can be configured by those skilled in the art according to specific application scenarios, and is not specifically limited here.

[0041] Specifically, the gradient threshold iteration boundary is initially extracted from the obtained single-channel gradient magnitude map. First, the iteration range, iteration step size, and total number of iterations of the gradient magnitude map are determined, where the minimum value of the iteration range of the gradient magnitude map is determined. Maximum value Iteration step size Total number of iterations , No. Threshold of round iteration ( By iterating through the gradient magnitude map through multiple rounds of threshold iterations, the initial boundary points under each threshold are extracted column by column. In other words, for each round of iteration threshold... traverse the gradient magnitude map column by column Search from top to bottom for the first gradient magnitude greater than 1 in the current column. For each pixel that meets the condition, its vertical coordinate is marked as the initial boundary point of the current column. Columns with no pixels meeting the condition are marked as -1, thus obtaining the initial boundary sequence under this threshold. After all iterations have been completed, the optimal initial boundary sequence under the preset conditions is output, which is the final iteration threshold. The corresponding optimal initial boundary sequence , This allows for the extraction of the optimal initial boundary sequence, ensuring the coverage and effectiveness of the boundary extraction.

[0042] The mathematical expression for the boundary extraction rule is as follows: ; in, Indexing the image column, For the first The initial vertical coordinates of the column's boundary points; -1 indicates that the column has no valid boundary points.

[0043] Furthermore, such as Figure 4 As shown in (a), the robust boundary optimization adopts a multi-level processing mechanism, such as a four-level processing mechanism of effective point statistics and no-sky pre-judgment, robust median estimation, missing value neighborhood filling and outlier secondary correction, to optimize the optimal initial boundary sequence, effectively eliminate the interference caused by noise and false edges, and obtain a robust and continuous initial boundary sequence of the dome line.

[0044] Specifically, firstly, such as Figure 4 As shown in (b), effective point statistics and no-sky pre-judgment are performed on the optimal initial boundary sequence to determine the effective boundary points (vertical coordinates not equal to -1) in the optimal initial boundary sequence, and an indicator function is defined. (Value 1 when the condition is true, value 0 otherwise), calculate the optimal initial boundary sequence. Number of valid boundary points , Calculate the percentage of valid points , When the proportion of valid boundary points among multiple initial boundary points is greater than or equal to a first threshold (e.g., 10% among multiple initial boundary points), for example, the proportion of valid points in this embodiment... If the proportion of valid points is greater than or equal to 10%, it is determined that there are valid points in the optimal initial boundary sequence, and the process proceeds to the next step. If the proportion of valid points is... If the percentage of valid points is less than 10%, it is directly determined that there is no valid sky area, and the subsequent detection process is terminated.

[0045] Secondly, such as Figure 4 As shown in (c), robust median estimation is performed, and the longitudinal coordinates of the effective boundary points are extracted to construct a set of effective boundary points. , The longitudinal coordinates of all valid boundary points are sorted in ascending order to obtain... Calculate the median of the valid points. That is, take the median of the sorted sequence as the benchmark value, and take the median. As a baseline, 20% of the image height is used as the maximum permissible deviation threshold. For example, set the maximum permissible deviation threshold. , ,For example Identify and mark valid boundary points that exceed the maximum permissible deviation threshold to obtain initial outlier values, for example, Then Marked as an initial outlier.

[0046] in, The number of elements in the set of valid points. This is the floor function.

[0047] Again, such as Figure 4 As shown in (d), missing value neighborhood filling is performed to determine the missing value column marked as -1 in the optimal initial boundary sequence, i.e. For each column with missing values, search for the nearest valid boundary point in the first direction (left direction). And, based on the second direction (right direction) of the missing value column, find the nearest second valid boundary point. The calculation formula is as follows: , ; If valid values ​​exist on both sides, the missing value is filled according to the rule, that is, the mean of the valid values ​​on both sides is used to fill the missing value; if a valid value exists on only one side, the valid value of that side is used to fill the missing value, thus completing the boundary sequence. The mathematical expression for the missing value filling rule is: ; Finally, as Figure 4 As shown in (e), outlier correction is performed on the initially marked outlier points. Search for the nearest non-anomaly valid boundary point in the first direction (left direction). The calculation formula is: ; The outlier correction rules are as follows: In other words, the non-anomaly valid boundary point is used. The vertical coordinates are used to replace the initial outliers, and after correction, a robust dome line boundary sequence is obtained. .

[0048] According to one embodiment of this application, polynomial fitting and global linear fitting are performed on the dome line boundary sequence to obtain the final dome line, including: constructing a polynomial fitting model; calculating the polynomial coefficients of the polynomial fitting model using a preset least squares method; and performing polynomial fitting on the dome line boundary sequence based on the polynomial coefficients; calculating the quantile outlier threshold of the dome line boundary sequence at the target location; replacing the initial outlier with fitted values ​​based on the fitting result and the quantile outlier threshold at the target location to obtain a smoothed dome line fitting boundary; and constructing a polynomial linear fitting model and performing global linear fitting on the dome line fitting boundary based on the polynomial linear fitting model to obtain the final dome line.

[0049] The preset least squares method can be configured by those skilled in the art according to specific application scenarios, and no specific limitations are made here.

[0050] Specifically, polynomial fitting boundary correction is performed on the dome line boundary sequence. First, horizontal column indexing is used. For independent variable and boundary vertical coordinate A 5th-order polynomial fitting model is constructed with the variable as the dependent variable. ,in, Indexing image columns , The boundary longitudinal coordinates are the fitted values. For polynomial coefficients, Then, a least-squares loss function is constructed. The polynomial coefficients are solved by minimizing the sum of squared residuals between the fitted values ​​and the actual boundary values. The expression for the loss function is: ; Solving for polynomial coefficient vectors based on the Vandermonde matrix The solution formula is: ,in, Let Vandermonde matrix be a matrix whose elements satisfy the following conditions: , , , For the boundary value vector, the set of vertical coordinates of the effective boundary points After sorting, the target position, such as the 90th percentile, is taken as the outlier threshold. ,in, ,Will Points marked as outliers to be corrected are used. The rounded-down values ​​replace the outliers to be corrected, resulting in the smoothed fitting boundary. Then, based on the fitting results and the quantile outlier threshold at the target location, the initial outliers are replaced with fitted values ​​to obtain the smoothed dome line fitting boundary. The mathematical expression for the correction rule is: ; Furthermore, linear optimization of virtual scene adaptation is performed to fit the boundary. Horizontal column index With as the independent variable and the vertical axis as the dependent variable, a first-order polynomial linear fitting model is constructed, and the model expression is: ; in, Indexing image columns , The boundary longitudinal coordinates of the linear fit. For the intercept term, This is the slope term.

[0051] The fitting coefficients are obtained using the least squares method: first, the mean of the column indices of the column indexes is calculated. With boundary value mean The expression is: , ; slope term With intercept term The solution formula is: , ; The boundary sequence is updated based on the linear fitting results, i.e., using... Replace the integer value All values ​​are used to complete the boundary straightening optimization, resulting in the final celestial dome line. The updated formula is: , ; Linearization effect as Figure 5 As shown in the middle right figure, the dome line is a straight line that is flat, continuous, and adapted to the virtual scene. It can be directly used for virtual scene matching and fusion without additional post-processing.

[0052] In step S103, the mean of the longitudinal coordinates of the dome boundary sequence and the mean of the absolute values ​​of the differences in the longitudinal coordinates of adjacent column boundary points in the dome boundary sequence are calculated. If it is determined that there is a valid sky region in the original image of the virtual scene based on the mean of the longitudinal coordinates and the mean of the absolute values, a binarized sky mask is generated based on the final dome line, and the binarized sky mask is upsampled to obtain the final size of the original image of the virtual scene.

[0053] According to one embodiment of this application, a binarized sky mask is generated based on the final celestial dome, and the binarized sky mask is upsampled to obtain the final size of the original image of the virtual scene. The process includes: constructing a single-channel mask matrix based on the size of the target image and initializing all element values ​​in the single-channel mask matrix; assigning pixel-by-pixel conditional values ​​to the single-channel mask matrix according to the final celestial dome to generate a binarized sky mask; and upsampling the binarized sky mask by a target multiple to obtain the final size of the original image of the virtual scene.

[0054] Specifically, the existence of sky regions in the original image of the virtual scene is determined mainly based on the mean features of the vertical coordinates of the initial boundary sequence and the mean features of the difference between the vertical coordinates of adjacent points. Combined with the preset three-level judgment threshold, the existence of valid sky regions in the image is quantitatively determined to avoid invalid detection results.

[0055] Specifically, first, the sequence of dome line boundaries is calculated. mean of the vertical coordinates of all points Mean absolute value of the difference between the longitudinal coordinates of adjacent column boundary points in the dome boundary sequence Secondly, set three levels of judgment thresholds: the first threshold... ,For example Second threshold ,For example The third threshold The mathematical expression for the rule for determining the existence of sky regions is: ; In this embodiment, if If the condition is met, it is determined that a valid sky area exists, and the process proceeds to the next step; if the condition is met... or satisfy and If the area is not clear, it is determined that there is no valid sky area.

[0056] Furthermore, if a valid sky region is determined to exist, a sky mask is generated and its size is restored. This mainly involves generating a binarized sky mask based on the final dome line, marking the sky region in red in the output image, and performing a 2x upsampling process on the binarized sky mask and the marked detection image to restore it to the size of the original input image, thus completing the final dome line detection and sky mask output.

[0057] Specifically, first, a single-channel mask matrix with the same size as the downsampled target image is created. First, initialize all elements in the single-channel mask matrix to 0; second, based on the final celestial dome line... Generate a binary sky mask. The mathematical expression for the mask generation rule is as follows: ; in, For image row indexing, This is the image column index; 255 is the sky region marker value, and 0 is the non-sky region marker value.

[0058] Then, pixel-by-pixel conditional assignments are performed on the single-channel mask matrix, for example, ... The pixel values ​​of the sky region are set to 255, while the non-sky regions remain unchanged at 0. The generated binarized sky mask is as follows: Figure 5 As shown; furthermore, if a valid sky region is determined to exist, the downsampled original image is copied to the output image. Iterate through all pixels of the image and extract the sky mask. The corresponding sky region in the output image The values ​​are marked in red (B=0, G=0, R=255), resulting in a visual detection result; again, as... Figure 5 As shown, the sky mask generated is processed using the cv::pyrUp function. With visual detection results Perform a 2x upsampling process to restore the image to its original resolution of 1920×1080, obtaining the final sky mask and detection result images. Save the final sky mask and detection result images, and output a detection completion message, thus completing the final sky dome detection and sky mask output. The mathematical expression for upsampling is: ; in, To reduce the detection results in the sampling space, The final output is to restore the original size.

[0059] Based on the above specific embodiments, the implementation effect of this application embodiment is verified as follows: This embodiment tested 100 frames of virtual scene images (1920×1080 resolution) from different scenes on a domestic embedded platform (1.8GHz main frequency, 2G memory). The results were compared with traditional OTSU threshold segmentation and Sobel edge detection methods. The test results are shown in Table 1. Table 1

[0060] The test results show that the detection accuracy (F1 score 0.98) of the method of this invention is much higher than that of the traditional method, the boundary continuity reaches 99%, and the detection result is a straight line, which can be directly adapted to the virtual scene matching and fusion requirements; the average detection time is 9~10ms / frame, and real-time detection (frame rate ≥30 frames / second) can be achieved on the domestic embedded platform, which fully meets the engineering application requirements of the airborne platform.

[0061] It should be noted that the above description is only a preferred embodiment of this application and is not intended to limit this application. For those skilled in the art, this application can have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this invention should be included within the protection scope of this invention. The protection scope of this application is not limited to the above specific embodiments. The core technical solutions of this application as defined by the claims are all within the protection scope of this application.

[0062] In summary, this application can achieve the following beneficial effects: (1) High computational efficiency: The image resolution is greatly reduced by 2x Gaussian pyramid downsampling, which significantly reduces computational complexity and meets the real-time processing requirements of virtual scene; (2) Strong robustness: The four-level boundary optimization mechanism (effective point statistics and pre-judgment, robust estimation of median, missing value neighborhood filling, outlier secondary correction) can effectively handle complex situations such as noise, local occlusion, and weak edges, and the detection accuracy reaches more than 98%. (3) Good scene adaptability: In view of the fact that the sky line of the virtual scene is usually a straight line, a first-order linear fitting is introduced to optimize the global linearization, so that the detection results are more in line with the structural rules of the virtual scene. (4) Accurate determination of effective sky regions: Based on the mean of the vertical coordinates of the boundary sequence and the mean of the difference between adjacent values, combined with multi-level thresholds, the existence of sky regions is intelligently determined, avoiding invalid processing. (5) Complete processing link: It forms an end-to-end detection process from original image input to sky mask output, supports size restoration, and can be directly integrated into various virtual visual systems.

[0063] According to the virtual scene skyline detection method of this application embodiment, the target image is obtained by downsampling the acquired virtual scene original image, and a gradient magnitude map is calculated; the gradient magnitude map is iteratively calculated to obtain the optimal initial boundary sequence, and the skyline boundary sequence is optimized to obtain the skyline boundary sequence; then, polynomial fitting and global linear fitting are performed on the skyline boundary sequence to obtain the final skyline; the mean of the vertical coordinates of the skyline boundary sequence and the mean of the absolute values ​​of the differences in the vertical coordinates of adjacent column boundary points in the skyline boundary sequence are calculated; if it is determined that there is a valid sky region in the virtual scene original image based on the mean of the vertical coordinates and the mean of the absolute values, a binarized sky mask is generated based on the final skyline, and upsampling processing is performed to obtain the final size of the virtual scene original image. This invention addresses the challenges in matching and fusing aerial photoelectric images with 3D virtual scenes, such as differences in sky regions interfering with registration accuracy and the difficulty in balancing real-time performance and detection robustness on embedded platforms. By iteratively extracting initial boundary points column by column through gradient thresholding, and combining a four-level robust optimization mechanism with a polynomial-linear hybrid fitting strategy, it achieves adaptive, highly robust detection and linearization optimization of celestial lines in virtual scene images.

[0064] Next, the virtual visual dome line detection device according to the embodiments of this application is described with reference to the accompanying drawings.

[0065] Figure 6 This is a block diagram of a virtual visual skyline detection device according to an embodiment of this application.

[0066] like Figure 6 As shown, the virtual visual skyline detection device 10 includes: a calculation module 100, a fitting module 200, and a sampling processing module 300.

[0067] The calculation module 100 is used to acquire the original image of the virtual scene, perform downsampling processing on the original image of the virtual scene to obtain the target image, and calculate the gradient magnitude map of the target image. The fitting module 200 is used to iteratively calculate the gradient magnitude map based on a preset iterative gradient threshold range, obtain the optimal initial boundary sequence of the gradient magnitude map after iteration, optimize the optimal initial boundary sequence using a multi-level processing mechanism to obtain the dome line boundary sequence, and perform polynomial fitting and global linear fitting on the dome line boundary sequence to obtain the final dome line. The sampling processing module 300 is used to calculate the mean of the longitudinal coordinates of the dome boundary sequence and the mean of the absolute values ​​of the differences in the longitudinal coordinates of adjacent column boundary points in the dome boundary sequence. If it is determined that there is a valid sky region in the original image of the virtual scene based on the mean of the longitudinal coordinates and the mean of the absolute values, a binarized sky mask is generated based on the final dome line, and the binarized sky mask is upsampled to obtain the final size of the original image of the virtual scene.

[0068] According to one embodiment of this application, the computing module 100 includes: The conversion unit is used to perform grayscale conversion on the target image to obtain the target grayscale image; The first calculation unit is used to calculate the first gradient component of the target grayscale image in the horizontal direction and the second gradient component of the target grayscale image in the vertical direction. The second calculation unit is used to calculate the gradient magnitude of each pixel in the target grayscale image based on the first gradient component and the second gradient component, so as to obtain the gradient magnitude map of the target image.

[0069] According to one embodiment of this application, the fitting module 200 includes: The first determining unit is used to determine the iteration range, iteration step size, and total number of iterations of the gradient magnitude map; The extraction unit is used to iteratively calculate the gradient magnitude map column by column according to the iteration range and iteration step size and the total number of iterations, and extract the initial boundary points of the gradient magnitude map after each iteration to obtain multiple initial boundary points of the target image. The second determining unit is used to determine the optimal initial boundary sequence that satisfies preset conditions based on multiple initial boundary points.

[0070] According to one embodiment of this application, the fitting module 200 includes: The third determining unit is used to determine the effective boundary points in the optimal initial boundary sequence; The first construction unit is used to determine that there is a valid sky region in the optimal initial boundary sequence when the proportion of valid boundary points in multiple initial boundary points is greater than or equal to a first threshold, and to extract the longitudinal coordinates of the boundary points of the valid sky region, and to construct a set of valid point boundary points of the valid sky region based on the longitudinal coordinates. The anomaly marking unit is used to calculate the median of all valid boundary points in the set of valid boundary points, and based on the median and the maximum permissible deviation threshold, marks the valid boundary points in the optimal initial boundary sequence whose vertical coordinates deviate from the median by more than the maximum permissible deviation threshold as anomalies, thus obtaining the initial anomaly values; The filling unit is used to determine the missing value column in the optimal initial boundary sequence, and to find the first valid boundary point based on the first direction of the missing value column and the second valid boundary point based on the second direction of the missing value column. Based on the first and second valid boundary points, the missing value column is filled using the missing value filling rule to obtain the complete initial boundary sequence. The correction unit is used to find non-abnormal valid boundary points based on the first direction of the initial outlier, and to correct the position of the initial outlier based on the non-abnormal valid boundary points to obtain the dome line boundary sequence.

[0071] According to one embodiment of this application, the fitting module 200 includes: The second building unit is used to build a polynomial fitting model, calculate the polynomial coefficients of the polynomial fitting model using the preset least squares method, and perform polynomial fitting on the dome line boundary sequence based on the polynomial coefficients. The third calculation unit is used to calculate the threshold of quantile outliers in the dome boundary sequence at the target location; The fitting unit is used to replace the initial outlier with the fitted value based on the fitting result and the quantile outlier threshold of the target position, so as to obtain the smoothed dome line fitting boundary. The third building unit is used to construct a polynomial linear fitting model and perform global linear fitting on the celestial dome fitting boundary based on the polynomial linear fitting model to obtain the final celestial dome.

[0072] According to one embodiment of this application, the sampling processing module 300 includes: The fourth construction unit is used to construct a single-channel mask matrix based on the size of the target image and initialize all element values ​​within the single-channel mask matrix; The generation unit is used to perform pixel-by-pixel conditional assignment on the single-channel mask matrix based on the final skyline, and generate a binary sky mask. The sampling processing unit is used to upsample the binarized sky mask by a target multiple to obtain the final size of the original image of the virtual scene.

[0073] According to the virtual scene skyline detection device of this application embodiment, a target image is obtained by downsampling the acquired virtual scene original image, and a gradient magnitude map is calculated; the gradient magnitude map is iteratively calculated to obtain the optimal initial boundary sequence, and the skyline boundary sequence is optimized to obtain the skyline boundary sequence; then, polynomial fitting and global linear fitting are performed on the skyline boundary sequence to obtain the final skyline; the mean of the vertical coordinates of the skyline boundary sequence and the mean of the absolute values ​​of the differences in the vertical coordinates of adjacent column boundary points in the skyline boundary sequence are calculated; if it is determined that there is a valid sky region in the virtual scene original image based on the mean of the vertical coordinates and the mean of the absolute values, a binarized sky mask is generated based on the final skyline, and upsampling processing is performed to obtain the final size of the virtual scene original image. This invention addresses the challenges in matching and fusing aerial photoelectric images with 3D virtual scenes, such as differences in sky regions interfering with registration accuracy and the difficulty in balancing real-time performance and detection robustness on embedded platforms. By iteratively extracting initial boundary points column by column through gradient thresholding, and combining a four-level robust optimization mechanism with a polynomial-linear hybrid fitting strategy, it achieves adaptive, highly robust detection and linearization optimization of celestial lines in virtual scene images.

[0074] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include: The memory 701, the processor 702, and the computer program stored on the memory 701 and executable on the processor 702.

[0075] When the processor 702 executes the program, it implements the virtual view skyline detection method provided in the above embodiments.

[0076] Furthermore, electronic devices also include: Communication interface 703 is used for communication between memory 701 and processor 702.

[0077] The memory 701 is used to store computer programs that can run on the processor 702.

[0078] The memory 701 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage.

[0079] If the memory 701, processor 702, and communication interface 703 are implemented independently, then the communication interface 703, memory 701, and processor 702 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized into address buses, data buses, control buses, etc. For ease of representation, Figure 7 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0080] Optionally, in a specific implementation, if the memory 701, processor 702, and communication interface 703 are integrated on a single chip, then the memory 701, processor 702, and communication interface 703 can communicate with each other through an internal interface.

[0081] The processor 702 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0082] This embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the virtual view skyline detection method described above.

[0083] This application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the virtual view skyline detection method as described in the above embodiments.

[0084] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0085] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0086] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0087] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0088] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0089] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware, and the program can be stored in a computer-readable storage medium. When executed, the program includes one or a combination of the steps of the method embodiments.

[0090] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0091] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A method for detecting the dome line in a virtual visual scene, characterized in that, Includes the following steps: The original image of the virtual scene is acquired, and the original image of the virtual scene is downsampled to obtain the target image. The gradient magnitude map of the target image is then calculated. Based on a preset iterative gradient threshold range, the gradient magnitude map is iteratively calculated to obtain the optimal initial boundary sequence of the gradient magnitude map after iteration. The optimal initial boundary sequence is then optimized using a multi-level processing mechanism to obtain the dome line boundary sequence. Finally, the dome line boundary sequence is subjected to polynomial fitting and global linear fitting to obtain the final dome line. Calculate the mean of the longitudinal coordinates of the dome boundary sequence and the mean of the absolute values ​​of the differences in the longitudinal coordinates of adjacent column boundary points in the dome boundary sequence. If it is determined that there is a valid sky region in the original virtual scene image based on the mean of the longitudinal coordinates and the mean of the absolute values, generate a binarized sky mask based on the final dome line, and perform upsampling processing on the binarized sky mask to obtain the final size of the original virtual scene image.

2. The method according to claim 1, characterized in that, The calculation of the gradient magnitude map of the target image includes: The target image is converted to grayscale to obtain a target grayscale image; Calculate the first gradient component of the target grayscale image in the horizontal direction and the second gradient component of the target grayscale image in the vertical direction; The gradient magnitude of each pixel in the target grayscale image is calculated based on the first gradient component and the second gradient component to obtain the gradient magnitude map of the target image.

3. The method according to claim 1, characterized in that, The step of iteratively calculating the gradient magnitude map based on a preset iterative gradient threshold range to obtain the optimal initial boundary sequence of the gradient magnitude map after iteration includes: Determine the iteration range, iteration step size, and total number of iterations for the gradient magnitude map; Based on the iteration range and the iteration step size, the gradient magnitude map is iteratively calculated column by column according to the total number of iterations, and the initial boundary points of the gradient magnitude map after each iteration are extracted column by column to obtain multiple initial boundary points of the target image; Based on the multiple initial boundary points, determine the optimal initial boundary sequence that satisfies the preset conditions.

4. The method according to claim 1, characterized in that, The process of optimizing the optimal initial boundary sequence using a multi-level processing mechanism to obtain the dome line boundary sequence includes: Determine the valid boundary points in the optimal initial boundary sequence; When the proportion of the effective boundary points in the multiple initial boundary points is greater than or equal to a first threshold, it is determined that there is an effective sky region in the optimal initial boundary sequence, and the longitudinal coordinates of the boundary points of the effective sky region are extracted. Based on the longitudinal coordinates, a set of effective point boundary points of the effective sky region is constructed. Calculate the median of all valid boundary points in the set of valid boundary points, and based on the median and the maximum permissible deviation threshold, mark the valid boundary points in the optimal initial boundary sequence whose vertical coordinates deviate from the median by more than the maximum permissible deviation threshold as anomalies to obtain initial outliers; The missing value column in the optimal initial boundary sequence is determined, and a first effective boundary point is found based on a first direction of the missing value column and a second effective boundary point is found based on a second direction of the missing value column. Based on the first effective boundary point and the second effective boundary point, the missing value column is filled using the missing value filling rule to obtain the complete initial boundary sequence. Based on the first direction of the initial outlier, non-outlier valid boundary points are found, and the initial outlier is corrected in position based on the non-outlier valid boundary points to obtain the dome line boundary sequence.

5. The method according to claim 1, characterized in that, The process of performing polynomial fitting and global linear fitting on the dome boundary sequence to obtain the final dome line includes: A polynomial fitting model is constructed, the polynomial coefficients of the polynomial fitting model are calculated using a preset least squares method, and the dome line boundary sequence is fitted with a polynomial based on the polynomial coefficients. Calculate the threshold value of the quantile outlier of the dome boundary sequence at the target location; Based on the fitting results and the quantile outlier threshold of the target location, the initial outlier values ​​are replaced with fitted values ​​to obtain the smoothed dome line fitting boundary. A polynomial linear fitting model is constructed, and a global linear fitting is performed on the fitting boundary of the dome line based on the polynomial linear fitting model to obtain the final dome line.

6. The method according to claim 1, characterized in that, The process of generating a binarized sky mask based on the final skyline and upsampling the binarized sky mask to obtain the final size of the original virtual view image includes: A single-channel mask matrix is ​​constructed based on the size of the target image, and all element values ​​within the single-channel mask matrix are initialized. Based on the final skyline, the single-channel mask matrix is ​​conditionally assigned pixel by pixel to generate a binary sky mask. The binarized sky mask is upsampled by a target factor to obtain the final size of the original virtual scene image.

7. A virtual visual skyline detection device, characterized in that, include: The calculation module is used to acquire the original image of the virtual scene, perform downsampling processing on the original image of the virtual scene to obtain the target image, and calculate the gradient magnitude map of the target image; The fitting module is used to iteratively calculate the gradient magnitude map based on a preset iterative gradient threshold range to obtain the optimal initial boundary sequence of the gradient magnitude map after iteration, and to optimize the optimal initial boundary sequence using a multi-level processing mechanism to obtain the dome line boundary sequence. Finally, the dome line boundary sequence is subjected to polynomial fitting and global linear fitting to obtain the final dome line. The sampling processing module is used to calculate the mean of the longitudinal coordinates of the dome boundary sequence and the mean of the absolute values ​​of the differences in the longitudinal coordinates of adjacent column boundary points in the dome boundary sequence. If it is determined that there is a valid sky region in the original virtual scene image based on the mean of the longitudinal coordinates and the mean of the absolute values, a binarized sky mask is generated based on the final dome, and the binarized sky mask is upsampled to obtain the final size of the original virtual scene image.

8. The apparatus according to claim 7, characterized in that, The computing module includes: A conversion unit is used to perform grayscale conversion on the target image to obtain a target grayscale image; The first calculation unit is used to calculate the first gradient component of the target grayscale image in the horizontal direction and the second gradient component of the target grayscale image in the vertical direction; The second calculation unit is used to calculate the gradient magnitude of each pixel in the target grayscale image based on the first gradient component and the second gradient component, so as to obtain the gradient magnitude map of the target image.

9. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and capable of running on the processor, the processor executing the program to implement the virtual view skyline detection method as described in any one of claims 1-6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the virtual view dome detection method as described in any one of claims 1-6.

Images