Method and system for automatically extracting multi-view perspective tilt images from panoramic videos for cultural relic protection

Abstract

The present application relates to the technical field of cultural relic digital protection, and particularly relates to a panoramic video multi-view angle tilt image automatic extraction method and system for ancient building and cultural relic site digital protection, which comprises the following steps: acquiring panoramic video data of a cultural relic object; analyzing the panoramic video data to obtain a down-sampling ERP panoramic frame; determining the azimuth angle, the pitch angle and the field angle of each virtual camera; performing multi-dimensional feature rapid detection on the down-sampling ERP panoramic frame to obtain an importance score of a key area; determining a frame extraction interval; obtaining a full-resolution ERP panoramic frame based on the frame extraction interval, and generating multiple perspective projection images by using a ray tracing method according to the determined view angle of each virtual camera; performing multi-dimensional quality evaluation on each extracted virtual view angle image; and performing view angle compensation on the virtual view angle images with insufficient quality. The present application improves the image sampling density of key parts by dynamic frame extraction, and improves the availability and the reconstruction success rate by view angle compensation.

Description

Technical Field

[0001] This invention relates to the field of digital preservation technology for cultural relics, specifically to a method and system for automatic extraction of panoramic video multi-view tilted images for the digital preservation of ancient buildings and cultural relics sites. Background Technology

[0002] In the protection of cultural heritage such as ancient buildings, historical sites, and relics, the non-renewable, immovable, and indestructible nature of these objects necessitates high-precision 3D digital acquisition for all aspects, including restoration, surveying and archiving, damage monitoring, digital display, and post-disaster restoration. Currently, commonly used techniques include 3D laser scanning, UAV oblique photogrammetry, close-range photogrammetry, handheld scanning, and manual photography. For example, some technical solutions have proposed combining 3D laser scanning with UAV oblique photogrammetry for the reconstruction of ancient buildings, and other methods have attempted to construct models of ancient buildings based on UAV oblique photogrammetry. While these methods have demonstrated certain value in the field of cultural relic digitization, they still face the following challenges: First, traditional oblique photography is not well-suited for ancient buildings and ruins. This method is typically applicable to open areas, regular plots, or city-level 3D modeling. However, for complex environments such as ancient courtyards, ancestral halls, towers, archways, old sites, mountain ruins, and narrow alley ruins, flight path planning is easily affected by factors such as trees, walls, power lines, visitor access, and protection zones, making it difficult to implement standard crisscross or five-directional flight paths.

[0003] Secondly, multi-lens tilt camera equipment is expensive, which hinders its widespread adoption by grassroots cultural heritage sites. The high cost of such equipment, coupled with the high barriers to operation and post-processing, makes it difficult for many county-level cultural heritage sites, memorial halls, and historical site management agencies to maintain long-term access to professional tilt photography equipment and aerial survey teams.

[0004] Furthermore, ordinary panoramic videos cannot be directly used for 3D modeling of cultural relics. While panoramic videos can quickly record the surrounding environment of buildings and sites, they typically use an equidistant cylindrical projection format, which suffers from problems such as large edge distortion, viewing angles that do not meet the requirements of 3D reconstruction software, and uncontrollable overlap between images. Currently, panoramic videos are mostly used for browsing, display, or VR experiences, and cannot directly address the key requirements of 3D reconstruction of cultural relics, such as viewpoint organization, detailed structural coverage, and accuracy control.

[0005] Finally, cultural relics contain a large number of detailed structures, and ordinary frame-taking methods are prone to missing data. Details commonly found in ancient architecture, such as brackets, eaves, roof tiles, ridge beasts, doors and windows, inscriptions, paintings, and wall defects, as well as targets in cultural relic sites such as slogan walls, gate towers, exhibition spaces, monuments, reliefs, caves, trenches, and old residential courtyards, often suffer from occlusion, unevenness, shadows, and dense local details. Ordinary timed frame-taking methods cannot guarantee multi-angle coverage of these key areas.

[0006] Therefore, there is an urgent need for a dedicated image extraction method for cultural relic protection scenarios, which can quickly acquire panoramic video and automatically generate multi-view image datasets that meet the requirements of oblique photogrammetry modeling and detailed recording of cultural relics. Summary of the Invention

[0007] The purpose of this invention is to provide a method and system for automatic extraction of panoramic video multi-view tilted images for digital protection of ancient buildings and cultural relics sites, in order to solve the technical problems in the prior art, such as high cost of multi-lens tilted photography equipment, limited flight path planning in narrow and complex cultural relic scenes, inability of ordinary panoramic video to be directly used for cultural relic-level 3D modeling, and difficulty in ensuring multi-angle coverage of the detailed structure of cultural relics by conventional frame extraction methods.

[0008] To achieve the above objectives, embodiments of the present invention provide a method for automatic extraction of panoramic video multi-view tilted images for cultural relic protection, comprising: Acquire panoramic video data of cultural relics; The panoramic video data is parsed to obtain downsampled ERP panoramic frames; The azimuth, pitch, and field of view of each virtual camera are determined based on the panoramic video data. Multi-dimensional feature rapid detection is performed on the downsampled ERP panoramic frame to obtain the importance score of key areas; The frame extraction interval is determined based on the importance score of the key areas; Based on the frame extraction interval, full-resolution ERP panoramic frames are obtained, and multiple perspective projection images are generated using ray tracing according to the determined viewpoint of each virtual camera. Perform multidimensional quality assessment on each extracted virtual viewpoint image; Based on the evaluation results, view compensation is performed on virtual view images with insufficient quality in the virtual view images; The quality of the compensated viewpoint is assessed, and qualified compensated images are retained.

[0009] Optionally, determining the azimuth, pitch, and field of view of each virtual camera based on the panoramic video data includes: The parameters of each scene criterion are calculated based on the panoramic video data. The scene criterion includes the scene criterion of inscription or slogan wall, the scene criterion of eaves or bracket details, the scene criterion of roof or courtyard, the scene criterion of building facade, the scene criterion of site passage and the scene criterion of indoor and outdoor transition. The type of the acquisition scene is identified based on the aforementioned scene criterion parameters; Based on the identified scene type, the corresponding multi-view template is called from the view template library to determine the azimuth, pitch, and field of view of each virtual camera.

[0010] Optionally, multi-dimensional feature rapid detection is performed on the downsampled ERP panoramic frame to obtain the importance score of key areas, including: Edge, texture, color anomaly, and key parts of cultural relics are detected in the downsampled ERP panoramic frames to obtain edge intensity score, texture density score, color anomaly score, and key parts of cultural relics matching score; The edge intensity score, texture density score, color anomaly score, and key part matching score of cultural relics are weighted and fused to obtain the importance score of key areas.

[0011] Optionally, determining the frame skipping interval based on the importance score of the key region includes: The frame extraction interval for the normal region is obtained according to formula (1). (1) in, This is the frame extraction interval for normal regions. For heading overlap, The average distance from the equipment to the cultural relic. For horizontal field of view, The moving speed of the data acquisition device; The frame extraction interval is obtained according to formula (2). (2) in, For key areas, the frame extraction rate is increased. Score the importance of the current frame. This is the importance threshold.

[0012] Optionally, based on the frame extraction interval, full-resolution ERP panoramic frames are obtained, and multiple perspective projection images are generated using ray tracing according to the determined viewpoint of each virtual camera, including: Establish spherical coordinate mapping; Construct the output image plane and vertical field of view based on the horizontal field of view; Based on the horizontal and vertical field of view angles, a virtual camera ray direction vector is generated for each pixel; Apply rotation matrices sequentially to the ray direction vector to transform the ray from the camera coordinate system to the world coordinate system; Calculate the spherical longitude and latitude based on the rotated world coordinate system direction vector to obtain the spherical coordinates; Based on the spherical coordinate mapping, the spherical coordinates are converted into ERP panoramic image pixel coordinates; Based on the pixel coordinates of the ERP panoramic image, bilinear interpolation sampling is performed on the panoramic image to obtain a perspective projection image from the virtual camera's viewpoint.

[0013] Optionally, constructing the output image plane and the vertical field of view based on the horizontal field of view includes: The vertical field of view is obtained according to formula (3). (3) in, For vertical field of view, For horizontal field of view, To output the height of the virtual image, To output the width of the virtual image.

[0014] Optionally, generating a virtual camera ray direction vector for each pixel based on the horizontal and vertical field of view angles includes: Normalize the center of each pixel in the output image to a preset normalized plane coordinate system; Calculate the directional components of the ray in the camera coordinate system according to formula (4). (4) in, The normalized x-coordinate of the pixel center. The normalized ordinate of the pixel center. For horizontal field of view, This refers to the vertical field of view. Normalize the virtual camera ray direction vector to a unit direction vector.

[0015] Optionally, the multidimensional quality assessment includes: sharpness assessment, exposure assessment, brightness assessment, shadow ratio assessment, and effective area ratio assessment of image quality.

[0016] Optionally, based on the evaluation results, viewpoint compensation for the failed viewpoints in the virtual viewpoint image includes: Select the corresponding compensation strategy based on the current view template and the name of the failed view. The compensation strategy includes facade template compensation, eaves or bracket template compensation and roof or passage template compensation. Compensated perspective images are generated using ray tracing.

[0017] On the other hand, the present invention also provides an automatic extraction system for panoramic video multi-view tilted images for cultural relic protection, the system comprising: The panoramic video data acquisition module is used to acquire panoramic video data of cultural relics. The parsing module is used to parse the panoramic video data to obtain downsampled ERP panoramic frames; The virtual camera parameter determination module is used to determine the azimuth, pitch, and field of view of each virtual camera based on the panoramic video data. A multi-dimensional feature detection module is used to perform rapid multi-dimensional feature detection on the downsampled ERP panoramic frame to obtain the importance score of key areas; The frame extraction interval determination module is used to determine the frame extraction interval based on the importance score of the key area; The perspective projection image generation module is used to obtain full-resolution ERP panoramic frames based on the frame extraction interval, and generate multiple perspective projection images according to the determined virtual camera viewpoint using ray tracing. The multidimensional quality assessment module is used to perform multidimensional quality assessment on each extracted virtual viewpoint image; The viewpoint compensation module is used to compensate for the viewpoint of the virtual viewpoint image that is of insufficient quality based on the evaluation results.

[0018] The beneficial effects of this invention are: This invention eliminates the need for the regular flight path planning required by traditional oblique photogrammetry. It only requires a single panoramic video capture of the cultural relic to automatically generate oblique perspective images covering multiple directions, demonstrating excellent adaptability to complex scenes with limited space, such as ancient courtyards, historical sites, narrow alleys, and mountain ruins. Furthermore, this invention obtains importance scores for key areas through downsampling analysis and rapid multi-dimensional feature detection, dynamically determining the frame extraction interval accordingly. This results in denser image coverage for key areas of the cultural relic while maintaining a basic acquisition density for ordinary areas. This achieves refined acquisition of key areas while ensuring overall coverage, effectively reducing the generation of invalid data. In addition, this invention performs multi-dimensional quality assessment on the generated virtual perspective images and compensates for insufficient quality perspectives, improving the overall usability of the output images and the success rate of 3D reconstruction.

[0019] Other features and advantages of the embodiments of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0020] The accompanying drawings are provided to further illustrate embodiments of the present invention and form part of the specification. They are used together with the following detailed description to explain the embodiments of the present invention, but do not constitute a limitation thereof. In the drawings: Figure 1 A flowchart of a method for automatic extraction of panoramic video multi-view tilted images for cultural relic protection according to an embodiment of the present invention. Figure 2 A flowchart illustrating a method for determining the azimuth, pitch, and field of view of each virtual camera based on panoramic video data, according to an embodiment of the present invention. Figure 3 A flowchart illustrating a method for generating multiple perspective projection images using ray tracing according to an embodiment of the present invention; Figure 4 This is an architecture diagram of a panoramic video multi-view tilt image automatic extraction system for cultural relic protection according to one embodiment of the present invention. Detailed Implementation

[0021] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the scope of the present invention.

[0022] It should be noted that the acquisition, transmission, storage, use, and processing of data in the technical solution of this application all comply with relevant laws and regulations. In the embodiments of this application, certain existing industry solutions such as software, components, and models may be mentioned. These should be considered exemplary, intended only to illustrate the feasibility of implementing the technical solution of this application, and do not imply that the applicant has already used or necessarily used such solutions.

[0023] like Figure 1 The diagram shows a flowchart of an automatic extraction method for panoramic video multi-view tilted images for cultural relic protection according to an embodiment of the present invention. Figure 1 The automatic image extraction method may include the following steps: In step S10, panoramic video data of the cultural relic is acquired; In step S11, the panoramic video data is parsed to obtain downsampled ERP panoramic frames; In step S12, the azimuth, pitch, and field of view of each virtual camera are determined based on the panoramic video data. In step S13, multi-dimensional feature rapid detection is performed on the downsampled ERP panoramic frame to obtain the importance score of key areas; In step S14, the frame extraction interval is determined based on the importance score of the key area; In step S15, full-resolution ERP panoramic frames are obtained based on the frame extraction interval, and multiple perspective projection images are generated using ray tracing according to the determined viewpoint of each virtual camera. In step S16, a multidimensional quality assessment is performed on each extracted virtual viewpoint image; In step S17, based on the evaluation results, view compensation is performed on virtual view images with insufficient quality in the virtual view images; In step S18, the quality of the compensated viewpoint is assessed, and qualified compensated images are retained.

[0024] In such Figure 1In the illustrated method for automatic extraction of panoramic video multi-view tilted images for cultural relic protection, step S10 is used to acquire panoramic video data of the cultural relic object. In this embodiment, panoramic video of the object to be collected is acquired using a drone equipped with a panoramic camera, a handheld stick, an inspection vehicle, a backpack acquisition device, or a track device. Specifically, a panoramic camera that supports equirectangular projection (ERP) output is selected; during acquisition, the device moves or circles around the cultural relic object at a low speed to ensure continuous coverage; the output is a standard 2:1 ERP panoramic video with a resolution preferably not less than 6000×3000 pixels. The objects to be collected include, but are not limited to, the roofs, eaves, brackets, facades, walls, and courtyards of ancient buildings; old building complexes, memorial facilities, slogan walls, and old residences; steles, archways, pagodas, ancestral halls, temples, and city walls; mountain sites, cave sites, trench sites, and archaeological site spaces.

[0025] Step S11 is used to parse the panoramic video data to obtain downsampled ERP panoramic frames. The panoramic video is decoded, and the ERP panoramic frames and their metadata are extracted in basic interval order. Downsampled versions of all frames are generated first for subsequent rapid scoring. Specifically, in this example, FFprobe can be used to read the video metadata and obtain the resolution. Duration Frame rate Total number of frames According to the basic interval Sequentially extract ERP panoramic frames; convert the decoded frames into a BGR three-channel image matrix. Generate downsampling version (Approximately 1200 pixels wide, maintaining a 2:1 aspect ratio) Used for rapid key point scoring and scene recognition.

[0026] Step S12 is used to determine the azimuth, pitch, and field of view of each virtual camera based on the panoramic video data. In this embodiment, the specific method for determining the azimuth, pitch, and field of view of each virtual camera based on the panoramic video data in step S12 can be of various forms known to those skilled in the art. In one example of the present invention, step S12 may include, for example... Figure 2 The steps shown are described in this. Figure 2 In this context, step S12 may include: In step S20, the parameters of each scene criterion are calculated based on the panoramic video data. The scene criteria include the scene criteria for inscriptions or slogan walls, the scene criteria for details of eaves or brackets, the scene criteria for roofs or courtyards, the scene criteria for building facades, and the scene criteria for indoor-outdoor transitions. In step S21, the type of scene to be collected is identified based on the scene criterion parameters; In step S22, the corresponding multi-view template is called from the view template library according to the identified acquisition scene type to determine the azimuth, pitch and field of view of each virtual camera.

[0027] In such Figure 2 In the method shown, steps S20 and S21 are used to calculate the parameters of each scene criterion based on panoramic video data. Based on the content features of the panoramic image, the current scene is automatically determined to belong to one of the following six types: inscription / slogan wall scene, eaves / bracket details scene, roof / courtyard scene, building facade scene, indoor / outdoor transition scene, and ruins passage scene. All criteria are based on lightweight, computable features, suitable for embedded real-time processing, and do not require a GPU. Optionally, deep learning scene classification models (such as EfficientNet-B0 or ​​ResNet-50) can be used to replace the rule-based criteria.

[0028] In this example, the criterion for determining the scene of inscriptions / slogans can be to use the HSV color space to detect red areas (corresponding to slogans and paintings) and count the proportion of red pixels. The MSER algorithm is used to detect extreme regions (corresponding to text and component outlines) and their proportion is calculated. : ; in, Represents the pixel coordinates of the image. Represents pixels Hue component values ​​in the HSV color space Represents pixels Saturation component values ​​in the HSV color space Represents pixels The lightness component value in the HSV color space Indicates the red-toned area. Image width, Image height, This is an indicator function that takes the value 1 if all the conditions within the parentheses are met, and 0 otherwise. This represents the total number of pixels covered by the extreme region detected by the MSER algorithm.

[0029] when and At that time, the scene corresponding to the current image is determined to be a monument / slogan wall scene. Among them, The threshold for determining the proportion of red pixels. This is the threshold for determining the proportion of extreme value regions. In this example, Configurable range is The priority value is 0.05; Configurable range is The preferred value is 0.10.

[0030] In this example, the scene criterion for eaves / bracket details could be calculating the Laplacian variance of the upper region (e.g., the top 1 / 4). This is used to measure high-frequency textures (such as roof tiles and bracket sets). Simultaneously, the proportion of dark areas in the bottom region is calculated. : ; in, This represents the top 1 / 4 area of ​​the image. For Laplacian operators, Represents the variance function. Indicates the bottom region of the image. Represents pixels grayscale value, when and At that time, the scene corresponding to the current image is determined to be a scene of eaves / bracket details. Among them, The threshold for determining texture variance. This is the threshold for determining the proportion of the dark area at the bottom. In this example, The configurable range is The preferred value is 200; The configurable range is The preferred value is 0.35.

[0031] In this example, the criterion for a rooftop / courtyard scene could be calculating the average gray level of the upper region (e.g., the top 1 / 4). : ; in, The set of pixels representing the top region of the image.

[0032] when At that time, the scene corresponding to the current image is determined to be a rooftop / courtyard scene. Among them, This indicates the criteria for determining the bright area. In this example, The configurable range is The preferred value is 180.

[0033] In this example, the criterion for determining a building facade scene could be to calculate the horizontal gradient of the central region using the Sobel operator. Statistical analysis of the percentage of pixels at the vertical edges : ; in, This represents the central region of the image, typically the middle half.

[0034] when At that time, the scene corresponding to the current image is determined to be a building facade scene. Among them, In this example, the threshold for determining the percentage of pixels at the vertical edge is... The configurable range is The preferred value is 0.35.

[0035] In this example, the criterion for determining the transition between indoor and outdoor scenes could be calculating the average grayscale ratio of the upper and lower regions. : ; in, This represents the average gray level of the upper region. This represents the average gray level of the lower region.

[0036] when This indicates a strong contrast in brightness (e.g., dark inside and bright outside of a cave / gateway / passage entrance), suggesting the current image corresponds to a transitional scene between indoors and outdoors. In this example, the luminance ratio threshold is used to determine the brightness ratio. The configurable range is The priority value is .

[0037] If none of the above criteria are met, the scene corresponding to the current image is determined to be the archaeological site passage scene.

[0038] Step S22 is used to retrieve the corresponding multi-view template from the viewpoint template library based on the identified scene type, and determine the azimuth, pitch, and field of view of each virtual camera. Specifically, in this example, the correspondence between scene type and template is shown in Table 1: Table 1. Correspondence between Scene Types and Templates

[0039] Each template contains five virtual camera views, consisting of yaw, pitch, roll, and an optional horizontal field of view. )definition.

[0040] In this example, the specific parameters of the roof / courtyard template (A) are as follows:

[0041] The specific parameters for facade / wall formwork (B) are as follows:

[0042] The specific parameters for the eaves / bracket set / beam frame formwork (C) are as follows:

[0043] The specific parameters for the archaeological site passage / lane template (D) are as follows:

[0044] By adaptively selecting perspective templates based on the type of cultural relic, invalid images are reduced, multi-angle coverage of key parts of the cultural relic is improved, and the success rate of feature matching and the ability to express details in subsequent 3D reconstruction are enhanced.

[0045] Step S13 is used to perform multi-dimensional feature rapid detection on the downsampled ERP panoramic frame to obtain the importance score of key areas. In this embodiment, the downsampled ERP panoramic frame... Rapid detection of multi-dimensional features such as edges, textures, color anomalies, and key parts of cultural relics is performed, and an importance score of key areas is obtained through a weighted fusion model. Specifically, in this example, let the importance score of the current frame be... It is composed of a weighted sum of four-dimensional features: ; in, For edge strength scoring, Scoring the texture density Score the color anomaly. Assign scores to key parts of cultural relics; The weighting coefficients satisfy the normalization condition. In this example, the weighting coefficient can take the following values: , , , When the primary data collection target is a slogan wall, the data collection capacity can be increased. The value was increased to 0.35 to enhance the detection of red slogans.

[0046] Furthermore, edge strength scoring The calculation method is as follows: First, the gradient magnitude of the image is calculated using the Sobel operator: ; in, Represents pixels The horizontal Sobel gradient value at that point. Represents pixels The Sobel gradient value in the vertical direction at that point.

[0047] Next, the statistical gradient magnitude is greater than the threshold. The pixel ratio, normalized to : ; .

[0048] The closer the value is to 1.0, the richer the image edge information and the clearer the texture.

[0049] Furthermore, texture density scoring The calculation method is as follows: Calculate the second derivative variance of the image using the Laplacian operator: ; Normalization to Texture density score obtained: .

[0050] Furthermore, color anomaly / disease score The calculation method is as follows: Detect red areas (HSV color space, corresponding to slogans, paintings, and rust): ; in, This represents the set of pixels that satisfy the low-value red range. This represents the set of pixels that satisfy the high-value red range.

[0051] Next, the proportion of original red pixels is calculated, and a color anomaly score is obtained: ; .

[0052] The closer the value is to 1.0, the higher the coverage of red-toned content such as red slogans, paintings, or rust in the image.

[0053] Furthermore, the matching and scoring of key parts of cultural relics. The calculation method is as follows: The MSER (Maximum Stable Extreme Region) algorithm is used to detect extreme regions (approximately corresponding to text and component outlines): ; in, This represents the pixel percentage of the original MSER region, reflecting the coverage of extreme stable regions in the image. The larger the value, the larger the area occupied by key parts of the cultural relic (text, carving, component outlines, etc.) in the image. This represents the total number of pixels covered by the extreme value region detected by the MSER algorithm. The closer the value is to 1.0, the richer the details of the cultural relics in the image, such as the inscriptions, component outlines, and carved patterns.

[0054] Step S14 is used to determine the frame extraction interval based on the importance score of key areas. Frame extraction interval Overlap with heading The quantitative relationship between them is as follows: set up The moving speed of the data acquisition device (unit: m / s). The average distance from the equipment to the cultural relic is expressed in meters. Given the horizontal field of view (unit: degrees), the ground coverage width of a single image is: Adjacent frame device offset: The heading overlap rate is: .

[0055] The required frame extraction interval is calculated by working backward from the target overlap rate: .

[0056] According to the frame extraction interval Overlap with heading The quantitative relationship between them can be used to obtain the frame skipping interval of ordinary regions. .

[0057] Then score the importance of the current frame. Importance threshold The comparison is used to determine the frame extraction interval. The configurable range is The preferred value is 0.55.

[0058] ; in, For key areas, the frame extraction rate is increased. (Frequency increased by 4 times).

[0059] In this example, with parameters m / s, m, For example, calculate: m; Normal area (target) ): s. Actual conservative value s, corresponding to actual overlap It satisfies CH / Z3005-2010 greater than Requirements.

[0060] Encrypted area (target) ): s. Actual conservative value s, corresponding to actual overlap The detailed reconstruction suggestion is greater than Requirements.

[0061] By encrypting and extracting frames in key areas, images of important parts such as inscriptions, slogans, brackets, and cracks can be obtained from more angles and with higher overlap, while ordinary areas maintain basic density. This achieves adaptive sampling, ensuring the accuracy of detailed reconstruction of cultural relics while controlling the total amount of data.

[0062] Step S15 is used to acquire full-resolution ERP panoramic frames based on the frame-sampling interval, and generate multiple perspective projection images using ray tracing according to the determined virtual camera viewpoint. In this embodiment, the specific method for generating multiple perspective projection images using ray tracing in step S15 can be of various forms known to those skilled in the art. In one example of the present invention, step S15 may include, for example... Figure 3 The steps shown are described in this. Figure 3 In this context, step S15 may include: In step S30, a spherical coordinate mapping is established; In step S31, the output image plane and the vertical field of view are constructed based on the horizontal field of view. In step S32, a virtual camera ray direction vector is generated for each pixel based on the horizontal and vertical field of view angles; In step S33, rotation matrices are applied sequentially to the ray direction vector to transform the ray from the camera coordinate system to the world coordinate system; In step S34, the spherical longitude and latitude are calculated based on the rotated world coordinate system direction vector to obtain the spherical coordinates; In step S35, based on spherical coordinate mapping, the spherical coordinates are converted into ERP panoramic image pixel coordinates; In step S36, bilinear interpolation sampling is performed on the panoramic image based on the pixel coordinates of the ERP panoramic image to obtain a perspective projection image from the virtual camera's perspective.

[0063] In such Figure 3 In the method shown, step S30 is used to establish a spherical coordinate mapping, mapping the pixels in the panoramic image to spherical coordinates, and establishing two-dimensional pixel coordinates. The bidirectional correspondence between the coordinates and the three-dimensional spatial orientation is used to extract arbitrary perspective views from the panoramic image. Specifically, in this example, a right-handed Cartesian coordinate system is established with the center of the sphere as the origin: x-axis: pointing directly forward of the sphere. y-axis: points directly upwards on the sphere, towards the zenith. z-axis: points to the right of the sphere. .

[0064] Let the width of the panoramic image be... The height is (ERP format meets) The horizontal direction angle (longitude) of the sphere is The vertical direction angle (latitude) is .

[0065] The mapping from spherical coordinates to pixel coordinates is as follows: ; in, The horizontal pixel coordinates of the panoramic image; The vertical pixel coordinates of the panoramic image; It is the horizontal direction angle; This is the vertical direction angle.

[0066] The conversion from spherical coordinates to three-dimensional Cartesian coordinates: ; The conversion from 3D Cartesian coordinates to spherical coordinates: ; This mapping establishes a two-way correspondence between pixels in a two-dimensional panoramic image and directions in three-dimensional space.

[0067] Step S31 is used to construct the output image plane and vertical field of view based on the horizontal field of view. First, Euler angles are defined: the azimuth angle yaw is the rotation around the y-axis. The z-axis represents the front, with positive values ​​pointing to the right; the pitch angle is a rotation around the z-axis, with positive values ​​indicating upward tilt and negative values ​​indicating downward tilt. Directly below; the roll angle (roll) is the rotation around the x-axis, typically... In this example, each virtual camera is defined by three Euler angles and a field of view: the azimuth angle (yaw) ranges from... The range of pitch angle is: The roll angle is generally... Horizontal field of view Default is .create The blank perspective image plane of pixels. The vertical field of view is calculated based on the horizontal field of view and the output aspect ratio. : .

[0068] Step S32 generates a virtual camera ray direction vector for each pixel based on the horizontal and vertical field of view angles. Specifically, in this example, the center of each pixel in the output image is... Normalization to Plane coordinates: ; Calculate the direction components of the ray in the camera coordinate system based on the field of view: ; Normalized to a unit direction vector: .

[0069] Step S33 involves applying rotation matrices sequentially to the ray direction vectors to transform the rays from the camera coordinate system to the world coordinate system. Specifically, in this example, the rotation matrices are defined as follows: ; in, , .

[0070] The complete rotational transformation is: .

[0071] Step S34 is used to determine the direction vector of the rotated world coordinate system. Calculate spherical longitude and latitude: .

[0072] Step S35 is used to convert spherical coordinates into ERP panoramic image pixel coordinates based on spherical coordinate mapping. Specifically, the mapping formula established in step S30 is used to convert the spherical coordinates... Convert to ERP panoramic image pixel coordinates : ; For longitude coordinates Perform periodic processing For latitude coordinates Make boundary clipping .

[0073] Step S36 is used to perform bilinear interpolation sampling on the panoramic image based on the ERP panoramic image pixel coordinates to obtain a perspective projection image from the virtual camera's viewpoint. Specifically, in this example, according to The panoramic image is sampled using bilinear interpolation, with periodic boundaries applied in the horizontal direction.

[0074] set up For floating-point coordinates, the definition is: ; Horizontal mold taking: , The interpolation result is: ; After traversing all pixels, output the perspective projection image of the virtual camera viewpoint. Repeat steps S31 to S36 for each viewpoint in the template.

[0075] Step S16 is used to perform a multi-dimensional quality assessment on each extracted virtual viewpoint image, comprehensively judging its effectiveness and providing a triggering basis for viewpoint compensation. In this example, the multi-dimensional quality assessment includes sharpness assessment, exposure assessment, brightness assessment, shadow ratio assessment, and effective area ratio assessment of image quality.

[0076] Furthermore, a specific method for sharpness assessment could be to first calculate the Laplacian variance of the image: ; in, This represents the grayscale image to be evaluated.

[0077] when When the current image is blurred, it is determined that the current image is blurred. The range is Preferred .

[0078] Furthermore, a specific method for exposure assessment could be: first, statistically analyze the percentage of overexposed pixels. ( ) and underexposed pixel ratio ( ): ; when If this happens, the current image is determined to be underexposed.

[0079] Furthermore, a specific method for brightness evaluation could be to calculate the average gray level of the image. .like It determines whether the current image is too dark or too bright.

[0080] Furthermore, a specific method for evaluating shadow proportions could be to statistically analyze the percentage of pixels in dark areas. ( ), used for compensation trigger determination in eaves / bracket sets.

[0081] Furthermore, the method for calculating the proportion of effective image quality region can be to define the effective quality region as a set of pixels that are properly exposed and not dark. : ; Image quality effective area ratio for: ; If the clarity is not up to standard, then... .

[0082] Finally, a comprehensive validity determination is made, and the necessary and sufficient condition for the validity of this viewpoint image is that it simultaneously satisfies the following: (Clarity meets requirements); (The exposure was satisfactory); (The brightness is reasonable); , The range is Preferred .

[0083] Step S17 is used to compensate for the insufficient quality of virtual viewpoint images based on the evaluation results. When the quality or effective area of ​​a certain virtual camera viewpoint is insufficient, a set of compensation viewpoints is automatically generated, and the probability of selecting available projection viewpoints is increased by fine-tuning the angle or scaling the field of view. The compensation viewpoints are generated based on different projection directions of the same panoramic frame, without changing the original acquired data. The triggering condition is: meeting the following conditions. Or eaves / bracket sets Triggered at time.

[0084] In this example, the specific steps for automatically generating the compensated viewpoint are as follows: First, determine the template type of the failed view. Based on the current view template (roof / facade / eaves / passage) and the name of the failed view, select the corresponding compensation strategy.

[0085] The facade template compensation strategy is as follows: when the effective area of ​​the view from the front is insufficient, a candidate view with a small angle offset is generated, as shown in Table 2. Table 2 Facade Formwork Compensation Strategy

[0086] The compensation strategy for eaves / bracket sets is as follows: when the shadow area from the lower viewpoint is too large, candidate images are generated from multiple upward viewing angles, as shown in Table 3. Table 3 Compensation Strategy for Eaves / Bracket Formwork

[0087] Local magnification compensation reduces the horizontal field of view to (Approximately the default) (Half of the original image) increases the proportion of cultural relics in the output image without changing the camera position.

[0088] The roof / passage template compensation strategy is as follows: when overexposed directly below the roof, increase the angle of view (pitch= Increase the pitch angle when the passage is obstructed from the front (pitch= ).

[0089] Next, step S15 is called to generate a compensated perspective image, which is then marked as a compensated viewpoint in the naming and metadata.

[0090] Step S17, viewpoint compensation, increases the number of available viewpoint candidates, reduces model voids caused by single viewpoint failures, and increases the proportion of cultural relic areas in the output image. Furthermore, without re-acquiring data, it automatically fills in the gaps in candidate images from failed viewpoints, improving image coverage integrity in complex cultural relic environments and reducing the void rate and detail loss rate of the 3D reconstruction model.

[0091] Step S18 is used to assess the quality of the compensated viewpoint and retain qualified compensated images.

[0092] Finally, the cultural relics archiving dataset is output. The qualified images are saved in a structured manner by object, location, perspective and time, and a JSON metadata file containing the acquisition parameters and summary metadata is generated.

[0093] Specifically, the structured naming convention uses a six-dimensional hierarchical encoding format for output file naming: [Project Number]_[Cultural Relic Object Number]_[Part Type]_[Timestamp]_[Viewpoint Type]_[Frame Number].jpg. Here, the project number (e.g., GMYZ001) comes from manual input or the data acquisition task configuration file; the cultural relic object number (e.g., MainHall) comes from manual input, QR code scanning, or GIS coordinates; the part type (e.g., Roof / Facade / Eave / Corridor) is automatically inferred from the scene template; the timestamp comes from the video analysis in step S11; the viewpoint type (e.g., Front / LeftFront / CompLeft15) comes from the viewpoint template name mapping; and the frame number is a six-digit serial number that automatically increments in the system.

[0094] Generate a JSON file with the same name for each image, containing: { "project_id":"GMYZ001", "object_id":"MainHall", "component_type":"Eave", "view_type":"LeftOblique", "is_compensation":false, "frame_time":"00:00:25.400", "timestamp_sec":25.4, "frame_index":125, "yaw":-35.0, "pitch":40.0, "roll":0.0, "quality_score": 0.87, "key_region_score":0.92, "sharpness": 285.4, "mean_brightness": 142.3, "is_valid":true } Finally, a summary metadata file, metadata_summary.json, is generated, which records project information, total number of images, configuration parameters, and all image indexes. This facilitates the establishment of digital archives for cultural relics, comparison of defects, comparison of before and after restoration, and long-term monitoring databases.

[0095] Example 1: Digital Modeling of Ancient Building Courtyards and Roofs A panoramic video was captured of an ancient architectural courtyard. The acquisition equipment moved slowly along the center of the courtyard and the outer edges of the buildings to obtain continuous panoramic video.

[0096] First, the video is decoded and a spherical mapping is established, and the average gray level at the top is detected. The scene was identified as a "roof / courtyard scene," and a roof / courtyard template was invoked. Next, the downsampled ERP frames were scored. Enter encrypted mode ( s). Then, five perspectives are generated based on the selected time points: directly below, in front / back / left / right bottom. Overexposure directly below is detected to trigger perspective compensation (pitch= (High depression angle), and finally output the data by naming it according to the six-dimensional encoding.

[0097] Example 2: Digital Archiving of Slogan Walls at Old Sites A panoramic video was captured of a slogan wall at an old site. The proportion of the area marked in red was measured. And the MSER text area The scene was identified as a "stone inscription / slogan wall"; the facade template was called; five perspectives were generated: front, left front, right front, upward, and downward; due to the red area... rise, Exceeding the threshold triggers encrypted frame dropping; it is determined that reflection from directly in front is the cause. At that time, generate left front / right front And looking up / looking down Four compensation perspectives.

[0098] Example 3: Detailed collection of bracket sets and eaves of ancient buildings Panoramic video was captured below the eaves of the ancient building. High-frequency textures on the upper part were detected. And the bottom dark area The scene is identified as "eaves / bracket details"; the eaves template is called; five perspectives are generated: front, left, right, and front angle upward view and local magnification; when the shadow ratio exceeds the standard, five compensation perspectives are generated: left / right angle upward view, left / right angle upward view and local magnification, to increase the number of available candidates for upward view direction.

[0099] Example 4: Real-world measurement of a panoramic video system Input data: Panoramic video 0610.mp4, resolution 6000×3000 pixels (standard 2:1 ERP), duration 83.7 seconds, frame rate 30fps. System configuration: Normal frame skipping interval. s, key area multiplier Importance threshold Effective region threshold Output perspective resolution Pixel.

[0100] First, the video specifications were checked at 6000×3000@30fps, and a spherical coordinate mapping (x-forward / y-upward / z-right-handed system) was established. The first frame was identified as a "ruins passage scene" (the upper grayscale / vertical edge / bottom dark area did not reach the threshold of each criterion, and the brightness ratio did not exceed the limit). Ruins passage / alley templates (forward, left-forward, right-forward, left-side, right-side) were applied. The walls of the ruins passage had rich textures. Stable at Encryption frame skipping is triggered throughout the entire process. Encryption interval. s, number of processing time points One; some side views are affected by the black border at the edge of the ERP. This triggers the generation of compensated viewpoints; the first step generates 67 time points, with 5 basic viewpoints for each time point, totaling 335 images, and 52 compensated viewpoint images, generating a total of 387 structured named images and corresponding JSON metadata.

[0101] The overlap verification is shown in Table 4. m / s, m, ): Table 4 Overlap Verification

[0102] System performance verification is shown in Table 5: Table 5 System Performance Verification

[0103] The verification results show that the complete process of the method of the present invention is feasible and can automatically extract structured multi-view tilted image datasets from real panoramic videos.

[0104] On the other hand, the present invention also provides an automatic extraction system for panoramic video multi-view tilted images for cultural relic protection, the system architecture diagram of which is shown below. Figure 4 As shown, it includes a panoramic video data acquisition module, a parsing module, a virtual camera parameter determination module, a multi-dimensional feature detection module, a frame extraction interval determination module, a perspective projection image generation module, a multi-dimensional quality assessment module, and a viewpoint compensation module.

[0105] The system comprises the following modules: a panoramic video data acquisition module for acquiring panoramic video data of the cultural relic; a parsing module for parsing the panoramic video data to obtain downsampled ERP panoramic frames; a virtual camera parameter determination module for determining the azimuth, pitch, and field of view of each virtual camera based on the panoramic video data; a multi-dimensional feature detection module for rapidly detecting multi-dimensional features in the downsampled ERP panoramic frames to obtain an importance score for key areas; a frame extraction interval determination module for determining the frame extraction interval based on the importance score of key areas; a perspective projection image generation module for acquiring full-resolution ERP panoramic frames based on the frame extraction interval, and generating multiple perspective projection images using ray tracing according to the determined virtual camera viewpoints; a multi-dimensional quality assessment module for performing multi-dimensional quality assessment on each extracted virtual viewpoint image; and a viewpoint compensation module for compensating for insufficient quality virtual viewpoint images based on the assessment results.

[0106] The beneficial effects of this invention are: This invention is more suitable for scenarios with limited space for cultural relic protection. It does not require the regular flight path planning necessary for traditional oblique photography; only a single panoramic video capture at the cultural relic site is needed to automatically generate oblique perspective images covering multiple directions. It is highly adaptable to complex environments with confined spaces and limited flight paths, such as ancient building courtyards, old sites, narrow alleys, and mountain ruins. Simultaneously, this invention effectively reduces acquisition costs and operational barriers. Multi-directional oblique images can be generated using a consumer-grade panoramic camera, eliminating the need for expensive multi-lens oblique photography equipment or professional aerial survey teams. Its ease of operation makes it particularly suitable for widespread use by grassroots cultural relic protection units. Regarding image coverage, this invention uses multi-feature weighted scoring to assess the importance of each panoramic frame and dynamically adjusts the frame extraction interval accordingly. This ensures high-overlap, dense image coverage of key areas such as inscriptions, brackets, and eaves, while maintaining only basic overlap in ordinary areas. This allows for focused acquisition of detailed areas while ensuring overall coverage. Furthermore, this invention employs a two-stage processing strategy. The first stage rapidly filters high-value timestamps at downsampling resolution, while the second stage generates full-resolution perspective projection only from the filtered timestamps. This avoids processing invalid frames at the source, effectively reducing computational resource consumption and data storage volume. Finally, this invention uses a six-dimensional encoding naming rule and binds metadata files to each output image, fully recording information such as the cultural relic object, location, acquisition time, and camera pose. This ensures traceability for each image, facilitating comparative analysis of data from different periods and meeting long-term protection needs such as monitoring cultural relic damage and comparing data before and after restoration.

[0107] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0108] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0109] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0110] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0111] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0112] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0113] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0114] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0115] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for automatically extracting multi-view tilted images from panoramic videos for cultural relic preservation, characterized in that, The method includes: Acquire panoramic video data of cultural relics; The panoramic video data is parsed to obtain downsampled ERP panoramic frames; The azimuth, pitch, and field of view of each virtual camera are determined based on the panoramic video data. Multi-dimensional feature rapid detection is performed on the downsampled ERP panoramic frame to obtain the importance score of key areas; The frame extraction interval is determined based on the importance score of the key areas; Based on the frame extraction interval, full-resolution ERP panoramic frames are obtained, and multiple perspective projection images are generated using ray tracing according to the determined viewpoint of each virtual camera. Perform multidimensional quality assessment on each extracted virtual viewpoint image; Based on the evaluation results, view compensation is performed on virtual view images with insufficient quality in the virtual view images; The quality of the compensated viewpoint is assessed, and qualified compensated images are retained.

2. The method according to claim 1, characterized in that, The azimuth, pitch, and field of view of each virtual camera are determined based on the panoramic video data, including: The parameters of each scene criterion are calculated based on the panoramic video data. The scene criterion includes the scene criterion of inscription or slogan wall, the scene criterion of eaves or bracket details, the scene criterion of roof or courtyard, the scene criterion of building facade, the scene criterion of site passage and the scene criterion of indoor and outdoor transition. The parameter identification and acquisition scene type are based on the scene criteria. Based on the identified scene type, the corresponding multi-view template is called from the view template library to determine the azimuth, pitch, and field of view of each virtual camera.

3. The method according to claim 1, characterized in that, The downsampled ERP panoramic frame is subjected to rapid multi-dimensional feature detection to obtain the importance score of key areas, including: Edge, texture, color anomaly, and key parts of cultural relics are detected in the downsampled ERP panoramic frames to obtain edge intensity score, texture density score, color anomaly score, and key parts of cultural relics matching score; The edge intensity score, texture density score, color anomaly score, and key part matching score of cultural relics are weighted and fused to obtain the importance score of key areas.

4. The method according to claim 1, characterized in that, Determining the frame extraction interval based on the importance score of the key region includes: The frame extraction interval for the normal region is obtained according to formula (1). ，（1） in, This is the frame extraction interval for normal regions. For heading overlap, The average distance from the equipment to the cultural relic. For horizontal field of view, The moving speed of the data acquisition device; The frame extraction interval is obtained according to formula (2). ，（2） in, For key areas, the frame extraction rate is increased. Score the importance of the current frame. This is the importance threshold.

5. The method according to claim 1, characterized in that, Based on the frame extraction interval, full-resolution ERP panoramic frames are obtained. Multiple perspective projection images are generated using ray tracing according to the determined viewpoint of each virtual camera, including: Establish spherical coordinate mapping; Construct the output image plane and vertical field of view based on the horizontal field of view; Based on the horizontal and vertical field of view angles, a virtual camera ray direction vector is generated for each pixel; Apply rotation matrices sequentially to the ray direction vector to transform the ray from the camera coordinate system to the world coordinate system; Calculate the spherical longitude and latitude based on the rotated world coordinate system direction vector to obtain the spherical coordinates; Based on the spherical coordinate mapping, the spherical coordinates are converted into ERP panoramic image pixel coordinates; Based on the pixel coordinates of the ERP panoramic image, bilinear interpolation sampling is performed on the panoramic image to obtain a perspective projection image from the virtual camera's viewpoint.

6. The method according to claim 5, characterized in that, The output image plane and vertical field of view are constructed based on the horizontal field of view, including: The vertical field of view is obtained according to formula (3). ，（3） in, For vertical field of view, For horizontal field of view, To output the height of the virtual image, To output the width of the virtual image.

7. The method according to claim 5, characterized in that, Based on the horizontal and vertical field of view angles, generating a virtual camera ray direction vector for each pixel includes: Normalize the center of each pixel in the output image to a preset normalized plane coordinate system; Calculate the directional components of the ray in the camera coordinate system according to formula (4). ，（4） in, The normalized x-coordinate of the pixel center. The normalized ordinate of the pixel center. For horizontal field of view, This refers to the vertical field of view. Normalize the virtual camera ray direction vector to a unit direction vector.

8. The method according to claim 1, characterized in that, The multidimensional quality assessment includes: sharpness assessment, exposure assessment, brightness assessment, shadow ratio assessment, and effective area ratio assessment of image quality.

9. The method according to claim 1, characterized in that, Based on the evaluation results, viewpoint compensation for the failed viewpoints in the virtual viewpoint image includes: Select the corresponding compensation strategy based on the current view template and the name of the failed view. The compensation strategy includes facade template compensation, eaves or bracket template compensation and roof or passage template compensation. Compensated perspective images are generated using ray tracing.

10. A panoramic video multi-view tilt image automatic extraction system for cultural relic protection, characterized in that, The system includes: The panoramic video data acquisition module is used to acquire panoramic video data of cultural relics. The parsing module is used to parse the panoramic video data to obtain downsampled ERP panoramic frames; The virtual camera parameter determination module is used to determine the azimuth, pitch, and field of view of each virtual camera based on the panoramic video data. A multi-dimensional feature detection module is used to perform rapid multi-dimensional feature detection on the downsampled ERP panoramic frame to obtain the importance score of key areas; The frame extraction interval determination module is used to determine the frame extraction interval based on the importance score of the key area; The perspective projection image generation module is used to obtain full-resolution ERP panoramic frames based on the frame extraction interval, and generate multiple perspective projection images according to the determined virtual camera viewpoint using ray tracing. The multidimensional quality assessment module is used to perform multidimensional quality assessment on each extracted virtual viewpoint image; The viewpoint compensation module is used to compensate for the viewpoint of the virtual viewpoint image that is of insufficient quality based on the evaluation results.

Images