A multi-level display area screening method for a mixed reality environment

Abstract

The application discloses a multi-level display area screening method for a mixed reality environment, and relates to the technical field of computer vision and mixed reality, and comprises the following steps: acquiring a scene image; performing target detection on the scene image to extract at least one candidate area; performing a view angle evaluation on the candidate area to obtain a view angle score; performing image segmentation and area evaluation on the candidate area to obtain an area score; performing depth estimation and distance evaluation on the scene image to obtain a distance score; performing preliminary screening on the candidate area based on the view angle score, the area score and the distance score, performing three-dimensional reconstruction on the candidate area that passes the preliminary screening to generate a three-dimensional grid model; performing surface curvature evaluation on the three-dimensional grid model, calculating a surface curvature score, and screening a final display area according to the surface curvature score. The application has the advantages that the multi-level evaluation mechanism is used to solve the challenge of display area screening in the mixed reality environment, so that the visibility, integrity and user experience of display content are improved.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and mixed reality technology, and specifically to a multi-level display area selection method for mixed reality environments. Background Technology

[0002] In immersive mixed reality scenarios, the adaptation of displayed content to the scene is a key factor affecting user experience. The display area in mixed reality environments is no longer limited to regular rectangular screens but extends extensively to the surfaces of complex and irregular objects. This unique complexity in immersive scenarios places higher demands on the adaptability of displayed content. It requires consideration not only of the content's own deformation and layout rationality but also of whether the object surface serving as the display area is suitable for supporting the content. Faced with this challenge, how to select suitable display areas from complex immersive scenes becomes a crucial issue affecting the quality of displayed content presentation.

[0003] Traditional display technologies are typically designed with regular, static rectangular display areas, which struggle to handle the varied shapes, dynamic changes, and surface complexity of display areas in complex environments. The uniqueness of these immersive scenarios places higher demands on the adaptability of displayed content, and unfiltered display areas can severely impact the visibility, integrity, and visual effects of content due to incompatibility. For example, small areas may result in incomplete content presentation, with content potentially being cropped or reduced in size, affecting readability and user experience; areas with off-angle viewing can distract users and reduce information acquisition efficiency; and uneven surfaces can cause content distortion, stretching, or obstruction, making text difficult to read and image details blurry or distorted.

[0004] Specifically, display area selection faces three core challenges: First, the area requirement ensures the display area is large enough to accommodate content and prevent information loss. Second, distance and viewing angle requirements guarantee the display area is within the user's line of sight and offers a good viewing angle, optimizing the viewing experience. Third, surface flatness requirements prevent content distortion due to uneven surfaces, ensuring optimal display quality. Existing technologies often only adjust the displayed content itself, failing to comprehensively address the challenges posed by the complexity of display areas in real-world environments. In dynamically changing mixed reality scenarios, the lack of a systematic selection method that comprehensively considers viewing angle adaptation, area characteristics, spatial distance relationships, and surface geometry leads to poor content presentation on complex object surfaces, significantly impacting the user experience.

[0005] To address the aforementioned issues, existing technologies urgently need improvement. Summary of the Invention

[0006] The purpose of this invention is to provide a multi-level display area selection method for mixed reality environments. This method addresses the challenges of display area selection in mixed reality environments through a multi-level evaluation mechanism, including area requirements, distance and viewing angle requirements, and surface flatness requirements, thereby improving the visibility, integrity, and user experience of the displayed content.

[0007] This invention provides a multi-level display area filtering method for mixed reality environments, comprising the following steps: S1. Acquire scene image; S2. Perform target detection on the scene image and extract at least one candidate region; S3. Perform a viewpoint evaluation on each candidate region to assess the degree of adaptation between the candidate region and the user's line of sight and obtain a viewpoint score. S4. Perform image segmentation on each candidate region, extract pixel-level segmentation masks, and evaluate the area based on the segmentation masks to obtain an area score. S5. Perform depth estimation on the scene image to obtain the weighted average depth value and depth standard deviation of the candidate region; and perform distance evaluation based on the weighted average depth value and depth standard deviation to obtain a distance score; S6. Based on the viewpoint score, area score and distance score, the candidate regions are initially screened, and the candidate regions that pass the initial screening are reconstructed in three dimensions to generate a three-dimensional mesh model. S7. Evaluate the surface curvature of the three-dimensional mesh model, calculate the surface curvature score, and select the final display area based on the surface curvature score.

[0008] In one alternative implementation, in S2, target detection includes: S21. Preprocess the scene image to adapt its size and format to the target detection model; S22. The preprocessed image is inferred using the YOLOv10 model to extract the detection results of at least one candidate region; the detection results include the number of targets, bounding box information, target category, and confidence level; S23. Eliminate candidate regions with confidence levels below the first preset threshold and retain candidate regions with high confidence levels.

[0009] In one alternative implementation, in S3, the viewpoint evaluation includes: S31. Calculate the horizontal and vertical offsets between the center point of the candidate region and the center point of the scene image; S32. Based on the horizontal and vertical offsets, and combined with the width and height of the scene image, calculate the horizontal viewpoint score and the vertical viewpoint score. S33. Introduce a Gaussian distribution function that highlights the importance of the central region of the scene image and calculate the position weights; S34. Calculate the boundary distance factor to evaluate how close the candidate region is to the image boundary; The boundary distance factor is calculated using the following formula: in, The coordinates of the center point of the candidate region. W and H These are the width and height of the scene image, respectively; S35. The view score is calculated by weighting the position weight, horizontal view score, vertical view score and boundary distance factor.

[0010] In an optional implementation, in step S4, image segmentation includes: S41. Using the bounding box information corresponding to the candidate region as input, the EfficientViT-SAM model generates a pixel-level segmentation mask. S42. Post-process the segmentation mask, including removing isolated regions with an area smaller than a preset noise threshold, and optimizing the region boundaries through morphological operations.

[0011] In one alternative implementation, in step S4, the area evaluation includes: S43. Count the number of target pixels in the segmentation mask to obtain the pixel area of ​​the candidate region, and normalize it to the ratio of the total area of ​​the scene image to obtain the normalized area. S44. Calculate the perimeter of the candidate region and calculate the shape factor based on the perimeter and pixel area; The shape factor is calculated using the following formula: in, A The pixel area of ​​the candidate region. The perimeter of the candidate region; S45. The normalized area and shape factor are weighted and summed to obtain the area score.

[0012] In one alternative implementation, in S5, the depth estimation includes: S51. Process the scene image using the Depth Anything model to generate a depth map; S52. Normalize the depth map, mapping the depth values ​​uniformly to... interval; S53. Combining the segmentation mask, the normalized depth values ​​are weighted and averaged to calculate the weighted average depth value of the candidate region; wherein the weights are Gaussian distributed based on the distance between the pixel and the center of the candidate region. S54. Based on the weighted average depth value of the candidate region and the normalized depth map, calculate the depth standard deviation to characterize the uniformity of the depth distribution.

[0013] In one alternative implementation, in step S5, the distance evaluation includes: S55. Calculate the depth mean score based on the deviation between the weighted average depth value and the preset target depth value; S56. Based on the depth standard deviation, calculate the depth uniformity score using an exponential decay function; S57. The depth mean score and the depth uniformity score are weighted and summed to obtain the distance score.

[0014] In an optional implementation, in step S6, the preliminary screening of candidate regions based on the viewpoint score, area score, and distance score includes: S61. Calculate the comprehensive score by weighted summation of the viewpoint score, area score, and distance score. S62. Eliminate candidate regions with a comprehensive score lower than the second preset threshold, and retain candidate regions with a score higher than or equal to the second preset threshold for three-dimensional reconstruction.

[0015] In one alternative implementation, in step S6, the three-dimensional reconstruction includes: S63. Using the segmentation mask as input, process the original scene image using SF3D technology to generate a three-dimensional mesh model of the retained candidate region.

[0016] In one alternative implementation, in S7, the surface curvature evaluation includes: S71. Calculate the principal curvature of each vertex in the three-dimensional mesh model, and obtain the Gaussian curvature based on the product of the principal curvatures; the principal curvatures include the maximum curvature and the minimum curvature; S72. Calculate the distribution deviation of Gaussian curvature in the entire candidate region to quantify the overall flatness of the surface. S73. Based on the distribution deviation of the Gaussian curvature, calculate the surface curvature score using an exponential decay function; S74. The surface curvature score is compared with a preset curvature threshold, and the candidate areas that reach or exceed the preset curvature threshold are used as the final display areas.

[0017] As can be seen from the above, the multi-level display area filtering method for mixed reality environments provided in this application includes acquiring scene images, performing target detection to extract candidate areas, performing viewpoint evaluation, area evaluation and distance evaluation for preliminary filtering, three-dimensional reconstruction to generate models, and surface curvature evaluation to filter the final display area. The multi-level evaluation mechanism solves the challenges of display area filtering in mixed reality environments and improves the visibility, integrity and user experience of the displayed content. Attached Figure Description

[0018] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is an overall flowchart of a multi-level display area filtering method for mixed reality environments according to an embodiment of the present invention. Figure 2 This is a flowchart illustrating the target detection and viewpoint evaluation process according to an embodiment of the present invention. Figure 3 This is a flowchart illustrating the image segmentation and area evaluation process according to an embodiment of the present invention. Figure 4 This is a flowchart illustrating the depth estimation and distance scoring process according to an embodiment of the present invention. Figure 5 This is a flowchart illustrating the preliminary screening, three-dimensional reconstruction, and surface curvature evaluation process in an embodiment of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] In immersive mixed reality scenarios, traditional display technologies suffer from several problems when dealing with complex and irregular object surfaces. These problems include the display area being limited by the regular shape, the interior being easily cropped due to insufficient area, the information becoming blurred due to viewing angle deviation, improper distance affecting visibility, and the content being deformed or distorted due to surface unevenness. These factors affect the visibility, integrity, and user experience of the displayed content.

[0022] In this regard, such as Figure 1As shown, this application proposes a multi-level display area filtering method for mixed reality environments, including the following steps: S1. Acquire scene image; S2. Perform target detection on the scene image and extract at least one candidate region; S3. Evaluate the viewing angle of each candidate area, assess the degree of adaptation between the candidate area and the user's line of sight, and obtain a viewing angle score. S4. Perform image segmentation on each candidate region, extract pixel-level segmentation masks, and evaluate the area based on the segmentation masks to obtain an area score. S5. Perform depth estimation on the scene image to obtain the weighted average depth value and depth standard deviation of the candidate region; and perform distance evaluation based on the weighted average depth value and depth standard deviation to obtain a distance score; S6. Based on viewpoint score, area score and distance score, the candidate regions are initially screened, and the candidate regions that pass the initial screening are reconstructed in three dimensions to generate a three-dimensional mesh model. S7. Evaluate the surface curvature of the 3D mesh model, calculate the surface curvature score, and select the final display area based on the surface curvature score.

[0023] Mixed reality environments refer to display scenarios where computer-generated virtual content (such as text, images, and 3D models) is integrated with the user's real physical environment in real time through head-mounted displays or other perspective display devices, enabling dynamic interaction between the virtual content and the real environment. In this environment, the surfaces of real-world objects (such as walls, roads, furniture, and irregular objects) can serve as the display carriers for virtual content, which must be dynamically adapted according to the geometry, spatial location, and physical characteristics of the real surfaces.

[0024] Scene images refer to two-dimensional image data acquired through image acquisition devices that reflect real-world scenes within a mixed reality environment. These images serve as the basis for subsequent processing, carrying visual information about objects, backgrounds, and spatial layouts within the environment.

[0025] Candidate regions refer to objects or areas in a scene image that are identified through techniques such as object detection and are potentially suitable as display carriers. These regions are potential display targets that have been initially identified and require further evaluation of their suitability.

[0026] Viewpoint evaluation is a quantification process of assessing the fit between a candidate area and the user's line of sight. This evaluation aims to determine whether the candidate area falls within the user's field of vision and whether its viewing angle is appropriate. The viewpoint score is the result of the viewpoint evaluation, expressed numerically as the degree of fit between the candidate area and the user's line of sight.

[0027] Image segmentation is a technique for separating specific target regions from the background in a scene image. Through image segmentation, pixel-level boundaries of candidate regions can be accurately extracted. A segmentation mask is the output of the image segmentation operation, typically a binary image, in which pixels belonging to the target region are marked.

[0028] Area evaluation is the process of quantitatively assessing the size and shape of candidate regions. This evaluation aims to ensure that candidate regions have sufficient area to accommodate displayed content and possess appropriate geometry. The area score is the result of the area evaluation, representing the degree of fit in size and shape of the candidate region in numerical form.

[0029] Depth estimation is a technique that infers the distance between objects in a scene and the observer by analyzing two-dimensional image data. This technique provides three-dimensional spatial information for candidate regions. The weighted average depth value is the result of calculating the weighted average of the depth values ​​of each pixel within the candidate region, reflecting the overall spatial location of the candidate region. The depth standard deviation is an indicator that measures the uniformity of the depth value distribution within the candidate region.

[0030] Distance evaluation is the process of quantitatively assessing the reasonableness of the distance between a candidate area and the user. This evaluation aims to ensure that the candidate area is within the user's optimal viewing distance range. The distance score is the result of the distance evaluation, representing the degree of spatial suitability of the candidate area in numerical form.

[0031] Preliminary screening refers to a pre-screening of candidate regions based on multi-dimensional scoring such as viewpoint, area, and distance, before detailed 3D reconstruction. This step aims to reduce subsequent computation and improve overall efficiency.

[0032] 3D reconstruction refers to the technique of recovering the three-dimensional geometric structure of objects in a scene from two-dimensional image data. Through 3D reconstruction, an accurate 3D model of a candidate region can be obtained. A 3D mesh model is the output of 3D reconstruction, a geometric structure composed of a series of vertices, edges, and faces, used to represent the 3D shape of the candidate region.

[0033] Surface curvature evaluation is the process of quantitatively assessing the surface flatness of a 3D mesh model. This evaluation aims to identify areas with flat surfaces suitable for displaying content. The surface curvature score is the result of the surface curvature evaluation, representing the degree of fit of the candidate region's surface flatness in numerical form.

[0034] The final display area refers to the object surface area that has been selected after multiple rounds of screening as the most suitable for hosting mixed reality display content. This area meets requirements in terms of area, viewing angle, distance, and surface flatness.

[0035] The method proposed in this application effectively addresses the challenges of varied display area shapes and complex surfaces in mixed reality environments through a multi-level screening mechanism. This method comprehensively considers key factors such as the viewing angle, area, distance, and surface flatness of candidate areas, thereby avoiding problems such as cropping, blurring, deformation, or distortion of displayed content due to incompatible areas. Therefore, this method can provide a high-quality, highly adaptable display medium for mixed reality applications, improving the user's visual experience and information acquisition efficiency in immersive scenarios.

[0036] In an optional implementation, this application further proposes a specific method for object detection in step S2 above. First, the scene image is preprocessed to adapt it to the size and format of the object detection model. Specifically, since current mainstream deep learning object detection models, such as the YOLOv10 model, typically have specific requirements for the size, pixel value range, and channel order of the input image, the preprocessing operation aims to convert the original scene image into an input format acceptable to the model. In this embodiment, the longest side of the original scene image is limited to 1024 pixels, and it is scaled proportionally according to the original aspect ratio to ensure that the size of the input data adapts to the model's requirements. Furthermore, if the original scene image is in RGBA format, the alpha channel is removed, retaining the RGB three-channel data to ensure that the input format is consistent with the YOLOv10 model. This preprocessing ensures that the object detection model can correctly and efficiently process the input data, thus laying the foundation for subsequent detection tasks.

[0037] Subsequently, the preprocessed image is inferred using the YOLOv10 model to extract the detection results of at least one candidate region. B Test results B It typically includes the number of targets, bounding box information, target category, and confidence score, and its calculation expression is: ; in, The number of targets detected. Indicates the center point of the bounding box. For the width and height of the bounding box, For the target category, , where is the confidence level.

[0038] The YOLOv10 model, as an advanced single-stage object detector, is renowned for its exceptional speed and accuracy, capable of identifying and locating multiple objects in images in real time. During inference, the model outputs the bounding box coordinates (e.g., top-left corner coordinates, width, and height) for each detected object, the predicted object category (e.g., "person," "vehicle," "monitor," etc.), and a confidence score representing the model's confidence in the detection result as a true object. These detailed detection results provide rich and accurate data for subsequent screening and evaluation.

[0039] Based on this, to further improve the quality of candidate regions, candidate regions with confidence scores below a first preset threshold are eliminated, while high-confidence candidate regions are retained. During target detection, the model may generate low confidence scores for some ambiguous or unclear objects; these low-confidence detection results are often false alarms or inaccuracies. By setting a "first preset threshold," such as 0.6 or 0.7, detection results that the model considers uncertain can be filtered out. For example, if a candidate region has a confidence score of 0.5 and the first preset threshold is 0.6, then that candidate region will be eliminated. This screening mechanism effectively reduces the number of candidate regions in subsequent processing, ensuring that only targets that the model is highly confident in are included in subsequent stages such as viewpoint evaluation, area evaluation, and distance evaluation, thereby significantly improving the efficiency and accuracy of the entire screening method. In this embodiment, the first preset threshold is... .

[0040] Through the above technical solution, this application utilizes the efficient and high-precision YOLOv10 model for target detection, combined with a confidence-based screening mechanism. This not only ensures the high accuracy and reliability of candidate regions extracted from scene images, avoiding interference from low-quality or irrelevant regions in subsequent processing, but also effectively reduces the complexity and resource consumption of subsequent calculations by eliminating low-confidence regions. Therefore, this solution can more efficiently provide high-quality potential display regions for mixed reality environments, laying a solid foundation for subsequent 3D reconstruction and surface curvature evaluation, thereby improving the overall performance and user experience of the entire multi-level display region screening method.

[0041] In one alternative implementation, such as Figure 2 As shown, this application further proposes the following steps for perspective evaluation: S31. Calculate the horizontal and vertical offsets between the center point of the candidate region and the center point of the scene image. The calculation expressions are as follows: ; in, This is the horizontal offset; This is the vertical offset; Indicates the coordinates of the center point of the candidate region; This indicates the coordinates of the center point of the scene image.

[0042] S32. Based on the horizontal and vertical offsets, and combined with the width and height of the scene image, calculate the horizontal and vertical viewpoint scores. The calculation expressions are as follows: ; ; in, and These are the width and height of the scene image, used to normalize the offset values; Score based on horizontal perspective; Score based on vertical perspective.

[0043] S33. Introduce a Gaussian distribution function to highlight the importance of the central region of the scene image and calculate the position weights. .

[0044] Specifically, to highlight the importance of the central region of the scene image, positional weights are further introduced. Its form is a Gaussian distribution: in, These are the control parameters for the weight distribution. The smaller the value, the higher the weight of the central region of the scene image.

[0045] S34. Calculate the boundary distance factor to evaluate how close the candidate region is to the image boundary.

[0046] In addition, to prevent candidate areas from being too close to the field of view boundary and affecting the browsing experience of the displayed content, a boundary distance factor is added. Defined as: in, The coordinates of the center point of the candidate region. W and H These represent the width and height of the scene image, respectively.

[0047] S35. The viewpoint score is calculated by weighting the position weight, horizontal viewpoint score, vertical viewpoint score and boundary distance factor.

[0048] Specifically, by combining the aforementioned location weights, horizontal view scores, vertical view scores, and boundary distance factors, the final view score of the candidate region is determined. for: ; in, and This is a weighting coefficient used to balance the influence of horizontal and vertical scores. In this embodiment, it is set as the aspect ratio coefficient of the preprocessed scene image.

[0049] Specifically, when calculating the horizontal and vertical offsets between the center point of the candidate region and the center point of the scene image, this step aims to quantify the deviation of the candidate region from the center position of the entire scene image. By obtaining the geometric center coordinates of the candidate region and the geometric center coordinates of the scene image, the absolute distance difference between the two in the horizontal and vertical directions is calculated. These offsets are the foundational data for subsequent evaluation of the centrality of the candidate region, providing a direct quantitative basis for subsequent viewpoint score calculation. Based on the horizontal and vertical offsets, combined with the width and height of the scene image, when calculating the horizontal and vertical viewpoint scores, this step converts the obtained absolute offsets into relative scores to make them comparable. By normalizing the offsets with the width and height of the scene image, the influence of different image sizes can be eliminated, ensuring the objectivity and consistency of the viewpoint scores. When introducing a Gaussian distribution function that highlights the importance of the central region of the scene image to calculate the positional weights, this step aims to simulate the human eye's visual preference for the central region of the image, i.e., targets located in the center of the image are generally considered more visually salient. The Gaussian distribution function, with its central symmetry and outward decay characteristics, can effectively assign positional weights to candidate regions. In practical implementation, the center of the Gaussian distribution can be set as the center of the scene image. A weight value is calculated using a Gaussian function based on the distance between the candidate region's center point and the image center; the closer the distance, the greater the weight, thus giving the central region a higher priority in viewpoint evaluation. When calculating the boundary distance factor to assess how close the candidate region is to the image boundary, this step quantifies the proximity of the candidate region to the scene image edge. This factor effectively identifies candidate regions that are too close to the image edge, as these regions may be partially cropped or located in visual blind spots and are therefore unsuitable for display, thus incurring a penalty in the evaluation. Finally, when calculating the viewpoint score based on the position weight, horizontal viewpoint score, vertical viewpoint score, and boundary distance factor, this step is a comprehensive viewpoint evaluation process, aiming to integrate the aforementioned independent evaluation indicators into a unified viewpoint score. By weighted summing of the position weight, horizontal viewpoint score, vertical viewpoint score, and boundary distance factor, the visual suitability of the candidate region can be comprehensively and objectively reflected. The weights of each indicator can be adjusted according to the actual application scenario and user preferences to ensure that the final perspective score can accurately guide the subsequent selection of display areas, giving priority to those areas that are more visually attractive and more in line with the user's observation habits.

[0050] In one alternative implementation, such as Figure 3As shown, this application further proposes that in step S4, image segmentation includes: S41. Using the bounding box information corresponding to the candidate region as input, the EfficientViT-SAM model generates a pixel-level segmentation mask. Specifically, the input to the EfficientViT-SAM model is the candidate region image retained in step S2. Building upon the SAM model, the EfficientViT-SAM model combines an efficient visual transformer structure with powerful semantic segmentation capabilities, enabling it to generate accurate pixel-level segmentation masks for target regions against complex backgrounds. The value of each pixel indicates whether it belongs to the target region, as defined below: ; S42. Post-process the segmentation mask, including removing isolated regions with an area smaller than a preset noise threshold, and optimizing the region boundaries through morphological operations.

[0051] Since the initial mask generated by the segmentation model may contain small isolated regions or details with discontinuous boundaries, this noise can affect the area calculation and shape factor evaluation of candidate regions. Therefore, this embodiment performs post-processing on the segmentation results to ensure the quality of the segmentation mask. First, small isolated regions in the segmentation mask are removed using an area threshold elimination method, retaining only those with an area greater than or equal to a preset noise threshold. Part: ; in, To preset the noise threshold, To segment small, isolated regions within the mask; This is the segmentation mask after post-processing.

[0052] It should be noted that the segmentation masks mentioned in subsequent articles are all post-processed segmentation masks. .

[0053] In this embodiment, a preset noise threshold is used. Set as the total area of ​​the scene image This ensures that the retained candidate regions are sized to meet the display content requirements. Subsequently, morphological operations are used to optimize the boundaries of the candidate regions. Erosion and dilation operations are employed to smooth the region contours, remove noise, and enhance the continuity and regularity of the boundaries.

[0054] Specifically, step S41 aims to transition from coarse-grained object detection results (bounding boxes) to fine-grained pixel-level object representations (segmentation masks), which is crucial for accurate area calculation, depth estimation, and subsequent 3D reconstruction. The EfficientViT-SAM model is an efficient visual Transformer model specifically designed for image segmentation tasks, capable of generating high-quality pixel-level segmentation masks based on given cues (such as bounding boxes). Its efficiency lies in maintaining high segmentation accuracy while reducing computational resource consumption, making it suitable for real-time scenarios in mixed reality environments. This model learns the contextual information and semantic features of the image to distinguish foreground objects from the background and assigns a corresponding category label to each pixel, thereby forming an accurate segmentation mask.

[0055] Because the raw segmentation mask generated by the image segmentation model may have some undesirable characteristics, such as small, irrelevant noise points or regions, and uneven, jagged boundaries, the post-processing steps of S42 aim to eliminate these imperfections and improve the quality and usability of the segmentation mask. Removing isolated regions with an area smaller than a preset noise threshold is typically achieved through connected component analysis, identifying and eliminating excessively small pixel clusters, thereby purifying the segmentation result. Morphological operations are used to optimize region boundaries, employing operations such as erosion, dilation, opening, or closing to smooth the boundaries of the segmentation mask, eliminate jagged edges, fill small holes, or separate adhered objects, making it smoother and more accurate.

[0056] In one alternative implementation, such as Figure 3 As shown, this application further proposes specific steps for area assessment, including: S43. Count the number of target pixels in the segmentation mask to obtain the pixel area of ​​the candidate region, and normalize it to the proportion relative to the total area of ​​the scene image to obtain the normalized area.

[0057] Specifically, the pixel area of ​​the candidate region Statistical segmentation mask The number of target pixels is obtained, and its calculation formula is: ; in, denoted as the pixel area of ​​the candidate region.

[0058] To facilitate comparison between different candidate regions, the area of ​​each candidate region is normalized to a proportion relative to the total area of ​​the scene image, resulting in a normalized area. : ; in, and These represent the width and height of the original scene image, respectively.

[0059] S44. Calculate the perimeter of the candidate region and calculate the shape factor based on the perimeter and pixel area; The shape factor is calculated using the following formula: in, A The pixel area of ​​the candidate region. The perimeter of the candidate region; The shape factor, derived from the geometric properties of a circle, is an important indicator for measuring the regularity of a region. For example... Figure 3 As shown, a circle, as an ideal regular shape, has a shape factor of 1, reflecting its geometric advantage in maximizing area and minimizing perimeter. For rectangles or other irregular shapes, however, the shape factor is typically less than 1 because the perimeter increases relatively faster than the area (e.g., the shape factor of a square is approximately 0.785). This characteristic allows the shape factor to effectively distinguish between regular and elongated, irregular areas, ensuring that the selected candidate areas better meet display requirements.

[0060] S45. Perform a weighted summation of the normalized area and shape factor to obtain the area score. Its calculation expression is: ; in, and For the weighting coefficients, satisfying In this embodiment, the following settings are provided. , This is to highlight the importance of area ratio in the scoring, while also taking into account the influence of shape factor.

[0061] Specifically, the process involves counting the number of target pixels in the segmentation mask to obtain the pixel area of ​​the candidate region. This area is then normalized to a proportion relative to the total area of ​​the scene image. To obtain the normalized area, the pixel area is typically calculated by iterating through all pixels in the segmentation mask and counting the number of pixels marked as targets (foreground). Normalization involves dividing this pixel area by the total pixel area of ​​the entire scene image, resulting in a proportion value between 0 and 1. This normalized area objectively reflects the relative size and visual salience of the candidate region within the entire scene image, avoiding differences in area values ​​caused by varying image resolutions, and ensuring comparability of candidate region area evaluations across different scenes.

[0062] When calculating the perimeter of candidate regions and the shape factor based on the perimeter and pixel area, the perimeter can be calculated by tracing the boundary pixels of the segmentation mask or using edge detection algorithms. The shape factor is a metric that measures the compactness of a region, with a value between 0 and 1, reaching a maximum of 1 when the region is circular. This factor effectively distinguishes between regularly shaped, compact regions and irregularly shaped, elongated regions. For example, for regions of the same area, a circular or square region will have a higher shape factor than an elongated or branching region. In mixed reality environments, regions with regular, compact shapes are generally preferred for display to provide a better visual experience and information presentation.

[0063] When calculating the area score by weighted summation of normalized area and shape factor, weighted summation is a method that integrates multiple evaluation indicators. By assigning different weights to the normalized area and shape factor, their importance in the area score can be adjusted according to the actual application requirements. For example, if more attention is paid to the relative size of the region, a higher weight can be assigned to the normalized area; if more attention is paid to the regularity of the region, a higher weight can be assigned to the shape factor. This flexible weighting mechanism allows the area score to more comprehensively and precisely reflect the overall geometric characteristics of the candidate region, thus providing a more accurate basis for subsequent screening.

[0064] By employing the aforementioned technical solution, after segmenting the candidate regions and extracting pixel-level segmentation masks, not only is the absolute pixel area of ​​the candidate regions considered, but further, by calculating their normalized area relative to the total area of ​​the scene image, their visual proportion within the entire scene can be objectively assessed. Simultaneously, a shape factor is introduced to quantify the geometric regularity and compactness of the candidate regions, effectively distinguishing regions of different shapes. Finally, by weighted summing the normalized area and the shape factor, the size and shape characteristics of the candidate regions are comprehensively considered, resulting in a more comprehensive and accurate area score. This enables more effective identification of regions that are not only of moderate size but also have regular shapes and better visual effects during the initial screening of display areas, significantly improving the precision of display area screening in mixed reality environments and enhancing the user experience.

[0065] In one alternative implementation, such as Figure 4 As shown, this application further proposes a specific implementation method for depth estimation, including the following steps: S51. Process the scene image using the Depth Anything model to generate a depth map.

[0066] Specifically, the Depth Anything model is an advanced monocular depth estimation model capable of predicting scene depth information from a single 2D image. This model is typically based on deep learning architectures, such as convolutional neural networks or Transformers, and establishes a mapping from image features to depth values ​​by learning from a large amount of image-depth pair data. In practical applications, the scene image to be processed is input into the pre-trained Depth Anything model, which outputs a depth map of the same size as the input image, where the value of each pixel represents the depth of that point in 3D space. This model has strong generalization ability and can handle various complex scenes, thus providing high-quality raw data for subsequent depth analysis.

[0067] S52. Normalize the depth map, mapping the depth values ​​uniformly to... Interval.

[0068] Specifically, the original depth values ​​of a depth map may have different ranges and units. Normalization unifies them to the [0,1] interval, which helps eliminate the influence of dimensions, facilitates subsequent calculations and comparisons, and thus improves the stability and robustness of the algorithm. This normalization can be achieved through linear mapping, for example, by finding the minimum and maximum values ​​in the depth map and then linearly scaling each depth value.

[0069] The normalization formula is as follows: in, and Depth map The minimum and maximum values ​​in the range. After normalization, the depth values ​​are limited to... This interval eliminates inconsistencies in depth scale between different candidate regions.

[0070] S53. Combine the segmentation mask and perform a weighted average on the normalized depth values ​​to calculate the weighted average depth value of the candidate region; where the weights are Gaussian distributed based on the distance between the pixel and the center of the candidate region.

[0071] Specifically, based on normalization, to more accurately characterize the depth characteristics of candidate regions, this embodiment introduces a Gaussian distribution-based weighting mechanism to calculate the depth values ​​of candidate regions. This weighting mechanism assigns higher weights to the center pixels of candidate regions, highlighting the contribution of the core parts of the candidate regions to depth calculation. Weighting coefficients The calculation formula is: ; in, Indicates the coordinates of the center point of the candidate region. Represents the pixel coordinates of the candidate region. Control the distribution range of the weights. Smaller Values ​​that concentrate weights closer to the center, while larger values... This value allows more pixels to participate in the overall calculation of the depth value.

[0072] Combined with this weighting coefficient, the weighted average depth value of the candidate region It can be calculated using the following formula: ; in, This represents the segmentation mask for the candidate region. This represents the weighted average depth value of the candidate region. This calculation method not only considers the depth value of each pixel within the candidate region, but also incorporates the importance of pixel position, enabling the result to more accurately reflect the depth characteristics of the candidate region.

[0073] S54. Based on the weighted average depth value of the candidate region and the normalized depth map, calculate the depth standard deviation to characterize the uniformity of the depth distribution.

[0074] Among them, depth standard deviation Defined as: Standard deviation The smaller the value, the more uniform the depth distribution of the candidate region, and the better it is suited to display requirements; while a larger value indicates a more uniform depth distribution. This indicates that there are significant fluctuations in the depth distribution, which may lead to instability in the display effect.

[0075] Specifically, the weighted average depth value of the candidate region aims to more accurately reflect the overall depth information of the region, while highlighting the importance of the region center through weighting and reducing the influence of edge or noise pixels on the average value. Specifically, firstly, the segmentation mask obtained in S4 is used to determine all pixels within the candidate region. Then, for each pixel within the candidate region, its Euclidean distance to the center point of the candidate region is calculated, and a Gaussian weight is generated based on this distance; pixels closer to the center have a larger weight, and pixels farther from the center have a smaller weight. These weights are multiplied by the normalized depth value of the corresponding pixel, and then all weighted depth values ​​are summed and divided by the sum of the total weights to obtain the weighted average depth value. This weighting method effectively reduces the interference of region edges or background noise on depth estimation, making the average depth value more representative.

[0076] The depth standard deviation is used to quantify the dispersion or uniformity of depth values ​​within a candidate region. A smaller standard deviation indicates that the depth values ​​within the region do not vary much, and the surface is relatively flat; a larger standard deviation indicates drastic depth variations, possibly indicating complex geometric structures or depth discontinuities. After obtaining the weighted average depth value of the candidate region, for each pixel within the candidate region, the difference between its normalized depth value and the weighted average depth value is calculated and squared. Then, all squared differences are weighted and averaged, and finally, the square root is taken to obtain the depth standard deviation. This standard deviation effectively reflects the fluctuation of depth values ​​within the candidate region, providing important geometric features for subsequent distance evaluation.

[0077] Through the above technical solution, this invention generates depth maps using a Depth Anything model, achieving high-quality and highly generalizable scene depth information. Normalization of the depth map ensures that depth values ​​are compared and calculated on a uniform scale, enhancing the algorithm's stability and robustness. Furthermore, by combining a segmentation mask and calculating a weighted average depth value for the candidate region based on Gaussian distributions of pixel distances to the candidate region center, the interference of region edges or background noise on depth estimation is effectively reduced, making the obtained average depth value more accurately reflect the true depth of the candidate region. Simultaneously, calculating the depth standard deviation quantifies the uniformity of depth distribution within the candidate region, providing more refined geometric features for subsequent distance evaluation. This refined depth information makes distance evaluation more accurate and robust, thereby improving the effectiveness of initial screening and ensuring the reliability of subsequent 3D reconstruction and final display region selection.

[0078] In one alternative implementation, such as Figure 4 As shown, this application further proposes a method for distance evaluation, including: S55. Calculate the depth mean score based on the deviation between the weighted average depth value and the preset target depth value; For the deviation of the weighted average depth value, this embodiment adopts a linear decay scoring mechanism. If the deviation between the area depth and the target depth is within the allowable range, the score is the highest value of 1; when the deviation exceeds the allowable range, the score will gradually decrease as the deviation increases. The specific scoring formula is as follows: in, The depth average is the score; The preset target depth value represents the target distance of the candidate region under ideal conditions; This represents the allowable depth deviation range, used to control the rate of score decay.

[0079] S56. Calculate the depth uniformity score based on the depth standard deviation using an exponential decay function.

[0080] To reflect the uniformity of the depth distribution of the candidate regions, an exponential decay function based on standard deviation is introduced as a scoring mechanism. The specific formula is: in, For depth uniformity score, To control the impact of depth standard deviation on scoring, a larger scale parameter is used. The value allows for greater depth distribution fluctuations, while smaller values... They have a lower tolerance for fluctuations.

[0081] S57. The distance score is obtained by weighted summation of the mean depth score and the uniformity depth score.

[0082] in, and These are weighting coefficients used to balance the impact of depth mean deviation and distribution uniformity on the total score, and satisfy the following conditions: In this embodiment, the following settings are provided. , This is to better highlight the role of depth mean in distance evaluation.

[0083] Specifically, step S55 aims to quantify the degree of matching between the overall depth of the candidate region and the user's desired display depth. First, a preset target depth value is determined, which typically represents the ideal display distance that is most comfortable or best suited for interaction in a mixed reality environment. For example, this target depth value can be set based on the comfortable viewing distance for the human eye, the optimal focusing range of the mixed reality device, or the needs of a specific application scenario. Then, the deviation between the weighted average depth value of the candidate region and the preset target depth value is calculated. Based on this deviation, a depth mean score is calculated using a predefined function. This score intuitively reflects how close the candidate region is to the ideal display position in the depth dimension. Step S56 is used to evaluate the flatness or uniformity of the depth distribution within the candidate region. The depth standard deviation is an indicator of the dispersion of depth values ​​within a region. The smaller the depth standard deviation, the more uniform the depth and the flatter the surface of the region; conversely, the larger the depth standard deviation, the more drastic the depth variation within the region, and the potentially uneven surface. To convert this uniformity into a quantifiable score, this application uses an exponential decay function for calculation. The exponential decay function effectively maps larger depth standard deviations to lower scores and smaller depth standard deviations to higher scores, thus highlighting the importance of regions with uniform depth. Finally, step S57 aims to comprehensively consider the overall depth position of the candidate region and the uniformity of its internal depth distribution to obtain a comprehensive distance evaluation metric. Weighted summation allows different weights to be assigned to the mean depth score and the depth uniformity score based on actual application needs or user preferences. For example, if more emphasis is placed on matching the display area to the user's ideal distance, a higher weight can be assigned to the mean depth score; if more emphasis is placed on the smoothness of the display area surface to ensure clear content display, a higher weight can be assigned to the depth uniformity score. Through this weighted combination, the distance score can more comprehensively and flexibly reflect the suitability of the candidate region in the depth dimension, providing a more reliable basis for subsequent preliminary screening.

[0084] In one alternative implementation, such as Figure 5 As shown, this application further proposes a specific method for preliminary screening of candidate regions in S6 based on viewpoint score, area score, and distance score. This method includes: S61. Calculate the comprehensive score by weighted summation of the viewpoint score, area score, and distance score. .

[0085] ; in, , and The results are respectively the viewpoint score, area score, and distance score; , , For the weighting coefficients, satisfying To emphasize the priority of area proportion in candidate region selection, this embodiment sets the weight to... , , .

[0086] S62. Eliminate candidate regions with a comprehensive score lower than the second preset threshold, and retain candidate regions with a score higher than or equal to the second preset threshold for three-dimensional reconstruction.

[0087] To control the screening range, a second preset threshold for the comprehensive score is set. .when Setting the value too high will eliminate a large number of candidate regions, potentially resulting in an overly narrow final selection range; setting it too low will retain too many unsuitable candidate regions, increasing the computational burden of subsequent 3D reconstruction stages. In this embodiment, the setting is... To achieve a balance between computing resources and the diversity of candidate regions.

[0088] Through the above technical solution, this application can effectively integrate multi-dimensional candidate region evaluation information. By calculating a comprehensive score, considerations from multiple dimensions such as viewing angle, area, and distance are unified into a single quantitative indicator, thus providing a comprehensive and objective screening criterion. Based on the comparison of this comprehensive score with a second preset threshold, candidate regions that do not meet the requirements can be efficiently eliminated, avoiding time-consuming and resource-intensive 3D reconstruction operations on low-quality or unsuitable regions. This significantly improves the efficiency of the entire multi-level display region screening method, reduces the waste of computational resources, and ensures that only the most promising candidate regions can enter the subsequent fine-tuning stage, thereby improving the quality of the final display region and the user experience.

[0089] In an alternative implementation, this application further proposes that in step S6, the three-dimensional reconstruction includes: S63. Using the segmentation mask as input, the original scene image is processed using SF3D technology to generate a 3D mesh model of the retained candidate region.

[0090] Specifically, the segmentation mask is the pixel-level region extracted in image segmentation step S4, which accurately outlines the contours of candidate regions in a 2D image. Using it as input for 3D reconstruction provides the reconstruction algorithm with precise boundary information of the target object, ensuring a high degree of consistency between the reconstructed 3D model and the actual object in 2D projection, avoiding geometric distortions that may occur when relying solely on coarse bounding boxes for reconstruction. SF3D (Single-View 3D Reconstruction from Segmentation Masks) is a method specifically designed to reconstruct 3D shapes from segmentation masks of a single 2D image. This technique typically utilizes deep learning models to infer the potential 3D geometry of an object from the input 2D segmentation mask by learning a large number of 2D-3D correspondences. SF3D can effectively handle objects of various complex shapes and generate high-fidelity 3D mesh models, making it particularly suitable for mixed reality environments where high object shape accuracy is required. The original scene image contains rich texture, color, and lighting information, which is crucial for SF3D to infer the surface details and geometry of objects during reconstruction. By combining segmentation masks and original scene images, SF3D technology can not only utilize the precise contours provided by the masks but also extract additional visual cues from the images, thereby generating more realistic and detailed 3D mesh models. A 3D mesh model is a geometric data structure composed of vertices, edges, and faces, capable of accurately representing the three-dimensional shape of an object. Generating a 3D mesh model is a prerequisite for surface curvature evaluation (S7), as curvature calculation requires geometric information in three-dimensional space. The retained candidate regions refer to the display areas considered to have high potential after the initial screening (S6). 3D reconstruction of these regions ensures that subsequent detailed evaluation focuses on the most valuable targets.

[0091] By using pixel-level segmentation masks and original scene images as input, and employing SF3D technology for 3D reconstruction, this application efficiently and accurately generates high-fidelity 3D mesh models for initially screened candidate regions. SF3D technology fully utilizes the precise boundary information provided by the segmentation mask and the rich visual cues in the original scene image, overcoming the limitations of traditional methods in reconstructing complex 3D shapes from 2D information. This ensures that the reconstructed 3D model accurately reflects the geometric features of the object. This precise 3D mesh model provides a reliable foundation for subsequent surface curvature evaluation (S7), enabling the system to calculate surface curvature scores more accurately, thereby more effectively selecting the most suitable regions for display in mixed reality environments and significantly improving the accuracy and reliability of display region selection.

[0092] In one alternative implementation, such as Figure 5As shown, this application further proposes that in the above S7, the surface curvature evaluation includes: S71. Calculate the principal curvature of each vertex in the 3D mesh model, and obtain the Gaussian curvature based on the product of the principal curvatures; the principal curvatures include the maximum curvature and the minimum curvature.

[0093] After generating the 3D mesh model, this embodiment further evaluates its surface curvature to quantify the flatness and complexity of the candidate region. Surface curvature is determined by the principal curvature. , Characterization, and These are the maximum and minimum curvatures, respectively, calculated using the second-order derivative matrices of the mesh vertices. Based on the principal curvatures, Gaussian curvature can be further defined. : ; Gaussian curvature describes the geometric properties of a local surface. A schematic diagram of the Gaussian curvature of the model is shown below. Figure 5 As shown. When When, the surface is a convex curved surface; when At that time, the surface is saddle-shaped.

[0094] S72. Calculate the distribution deviation of Gaussian curvature in the entire candidate region to quantify the overall flatness of the surface.

[0095] To evaluate the overall surface smoothness of the candidate region, this embodiment calculates the distribution deviation of Gaussian curvature. Its formula is: ; in, Indicates the first i Gaussian curvature of each vertex This represents the average Gaussian curvature of the candidate region. The number of grid vertices. Deviation. The smaller the value, the smoother the surface of the candidate area, and the more suitable it is as a display area.

[0096] S73. Based on the distribution deviation of Gaussian curvature, the surface curvature score is calculated using an exponential decay function.

[0097] To quantify curvature evaluation into a scoring metric, this embodiment defines a surface curvature score. Its formula is: ; in, This is a control parameter for curvature deviation, used to adjust the sensitivity of the score to curvature fluctuations. This embodiment will... Set it to 0.5, such as Figure 5As shown, this setting can effectively reduce the score of uneven surface areas while maintaining a high tolerance for minor fluctuations.

[0098] S74. Compare the surface curvature score with the preset curvature threshold, and select the candidate areas that reach or exceed the preset curvature threshold as the final display areas.

[0099] To identify eligible regions, a curvature threshold is set for the surface curvature score. .when When the threshold is reached, the candidate region is considered to meet the criteria; otherwise, it is discarded. The threshold setting is crucial to the filtering results. Setting it too high will severely limit the filtering results, retaining only areas with perfectly flat surfaces; however, an excessively high threshold may also exclude some suitable candidate areas that have slight fluctuations. Setting it too low may introduce areas with high surface complexity, affecting the final display effect. Experimental verification in this embodiment shows that... Set as It can eliminate areas with high curvature fluctuations while retaining display areas with surface fluctuations within an acceptable range.

[0100] Through the aforementioned technical solution, this application introduces a refined evaluation mechanism based on surface curvature after reconstructing the candidate region into a 3D mesh model. By calculating the principal curvature and Gaussian curvature of each vertex in the 3D mesh model, and further analyzing the distribution deviation of Gaussian curvature throughout the candidate region, the geometric complexity and surface smoothness of the 3D model can be objectively and accurately quantified. Based on this, the surface curvature score is calculated using an exponential decay function and compared with a preset curvature threshold, thereby effectively eliminating 3D models with overly complex geometry, uneven surfaces, or significant distortions. This ensures that the finally selected display area has good visual consistency, geometric stability, and a lower rendering burden in the mixed reality environment, significantly improving the user's immersion and comfort in the mixed reality environment, and avoiding visual fatigue or discomfort caused by displaying low-quality 3D models.

[0101] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A multi-level display area filtering method for mixed reality environments, characterized in that, Includes the following steps: S1. Acquire scene image; S2. Perform target detection on the scene image and extract at least one candidate region; S3. Perform a viewpoint evaluation on each candidate region to assess the degree of adaptation between the candidate region and the user's line of sight and obtain a viewpoint score. S4. Perform image segmentation on each candidate region, extract pixel-level segmentation masks, and evaluate the area based on the segmentation masks to obtain an area score. S5. Perform depth estimation on the scene image to obtain the weighted average depth value and depth standard deviation of the candidate region; and perform distance evaluation based on the weighted average depth value and depth standard deviation to obtain a distance score; S6. Based on the viewpoint score, area score and distance score, the candidate regions are initially screened, and the candidate regions that pass the initial screening are reconstructed in three dimensions to generate a three-dimensional mesh model. S7. Evaluate the surface curvature of the three-dimensional mesh model, calculate the surface curvature score, and select the final display area based on the surface curvature score.

2. The method according to claim 1, characterized in that, In S2, target detection includes: S21. Preprocess the scene image to adapt its size and format to the target detection model; S22. The preprocessed image is inferred using the YOLOv10 model to extract the detection results of at least one candidate region; the detection results include the number of targets, bounding box information, target category, and confidence level; S23. Eliminate candidate regions with confidence levels below the first preset threshold and retain candidate regions with high confidence levels.

3. The method according to claim 1, characterized in that, In S3, the viewpoint evaluation includes: S31. Calculate the horizontal and vertical offsets between the center point of the candidate region and the center point of the scene image; S32. Based on the horizontal and vertical offsets, and combined with the width and height of the scene image, calculate the horizontal viewpoint score and the vertical viewpoint score. S33. Introduce a Gaussian distribution function that highlights the importance of the central region of the scene image and calculate the position weights; S34. Calculate the boundary distance factor to evaluate how close the candidate region is to the image boundary; The boundary distance factor is calculated using the following formula: in, The coordinates of the center point of the candidate region. W and H These are the width and height of the scene image, respectively; S35. The view score is calculated by weighting the position weight, horizontal view score, vertical view score and boundary distance factor.

4. The method according to claim 2, characterized in that, In S4, image segmentation includes: S41. Using the bounding box information corresponding to the candidate region as input, the EfficientViT-SAM model generates a pixel-level segmentation mask. S42. Post-process the segmentation mask, including removing isolated regions with an area smaller than a preset noise threshold, and optimizing the region boundaries through morphological operations.

5. The method according to claim 4, characterized in that, In S4, Area evaluation includes: S43. Count the number of target pixels in the segmentation mask to obtain the pixel area of ​​the candidate region, and normalize it to the ratio of the total area of ​​the scene image to obtain the normalized area. S44. Calculate the perimeter of the candidate region and calculate the shape factor based on the perimeter and pixel area; The shape factor is calculated using the following formula: in, A The pixel area of ​​the candidate region. The perimeter of the candidate region; S45. The normalized area and shape factor are weighted and summed to obtain the area score.

6. The method according to claim 1, characterized in that, In S5, depth estimation includes: S51. Process the scene image using the Depth Anything model to generate a depth map; S52. Normalize the depth map, mapping the depth values ​​uniformly to... interval; S53. Combining the segmentation mask, the normalized depth values ​​are weighted and averaged to calculate the weighted average depth value of the candidate region; wherein the weights are Gaussian distributed based on the distance between the pixel and the center of the candidate region. S54. Based on the weighted average depth value of the candidate region and the normalized depth map, calculate the depth standard deviation to characterize the uniformity of the depth distribution.

7. The method according to claim 6, characterized in that, In S5, the distance evaluation includes: S55. Calculate the depth mean score based on the deviation between the weighted average depth value and the preset target depth value; S56. Based on the depth standard deviation, calculate the depth uniformity score using an exponential decay function; S57. The depth mean score and the depth uniformity score are weighted and summed to obtain the distance score.

8. The method according to claim 1, characterized in that, In step S6, the preliminary screening of candidate regions based on the viewpoint score, area score, and distance score includes: S61. Calculate the comprehensive score by weighted summation of the viewpoint score, area score, and distance score. S62. Eliminate candidate regions with a comprehensive score lower than the second preset threshold, and retain candidate regions with a score higher than or equal to the second preset threshold for three-dimensional reconstruction.

9. The method according to claim 1, characterized in that, In S6, the three-dimensional reconstruction includes: S63. Using the segmentation mask as input, process the original scene image using SF3D technology to generate a three-dimensional mesh model of the retained candidate region.

10. The method according to claim 9, characterized in that, In S7, the surface curvature evaluation includes: S71. Calculate the principal curvature of each vertex in the three-dimensional mesh model, and obtain the Gaussian curvature based on the product of the principal curvatures; the principal curvatures include the maximum curvature and the minimum curvature; S72. Calculate the distribution deviation of Gaussian curvature in the entire candidate region to quantify the overall flatness of the surface. S73. Based on the distribution deviation of the Gaussian curvature, calculate the surface curvature score using an exponential decay function; S74. The surface curvature score is compared with a preset curvature threshold, and the candidate areas that reach or exceed the preset curvature threshold are used as the final display areas.

Images