Engineering aerial photography result and three-dimensional model linkage display method and device and readable medium

Abstract

This disclosure presents a method, apparatus, and readable medium for the linked display of engineering aerial photography results and 3D models. One specific implementation of the method includes: loading a pre-constructed twin 3D scene model; acquiring aerial video files, flight trajectory data, and time-synchronized location datasets for the target aerial photography sortie; performing coordinate transformation processing on the flight trajectory data; rendering a spatial trajectory curve corresponding to the trajectory data in the scene coordinate system within the twin 3D scene model; displaying the trajectory overlaid with the twin 3D scene model on a preset linked page; on the preset linked page, as the playback timeline of the aerial video files progresses, aerial images from the corresponding aerial image sequence are played sequentially. Each time an aerial image is played, a dynamic positioning icon is continuously updated and displayed along the spatial trajectory curve in the trajectory overlaid with the twin 3D scene model based on the time-synchronized location dataset. This implementation improves the user experience.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the field of computer technology, specifically to a method, apparatus, and readable medium for the linked display of engineering aerial photography results and 3D models. Background Technology

[0002] With the widespread application of drone aerial photography technology in large-scale engineering projects such as transportation, municipal works, and water conservancy, project managers need to view aerial photography results (such as inspection videos, photos, and panoramic views) through digital twin platforms to monitor construction progress and identify potential safety hazards. Linking aerial photography results with 3D models is one technology for showcasing aerial photography achievements. Currently, when identifying information about food plates, the common method is to display aerial videos and photos as a two-dimensional list, and represent the flight path with lines on a two-dimensional satellite map.

[0003] However, when using the above methods to display aerial photography results, the following technical problems often arise: The system only displays aerial videos and photos in a two-dimensional list and uses lines to represent the flight path on a two-dimensional satellite map. It lacks the ability to synchronize and link the collected videos and images with the drone's flight trajectory and the spatial position of the three-dimensional model. This results in a disconnect between the video frame content and the three-dimensional spatial position when the drone was shooting. Engineering managers find it difficult to quickly and accurately determine the specific location of engineering parts (such as a section of cofferdam or a gate chamber) in the video in the three-dimensional model, leading to a poor user experience.

[0004] The information disclosed in this background section is only intended to enhance the understanding of the background of the inventive concept, and therefore may contain information that does not form prior art known to those skilled in the art. Summary of the Invention

[0005] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0006] Some embodiments of this disclosure propose methods, devices, electronic devices, and computer-readable media for linking and displaying engineering aerial photography results with 3D models to solve one or more of the technical problems mentioned in the background section above.

[0007] In a first aspect, some embodiments of this disclosure provide a method for linking and displaying engineering aerial photography results with a 3D model. This method includes: loading a pre-constructed twin 3D scene model; acquiring aerial video files, flight trajectory data, and time-synchronized location datasets for a target aerial photography flight; performing coordinate transformation processing on the flight trajectory data to obtain scene coordinate system trajectory data; rendering a spatial trajectory curve corresponding to the scene coordinate system trajectory data in the twin 3D scene model to obtain a trajectory-overlay twin 3D scene model; displaying the trajectory-overlay twin 3D scene model on a preset linkage page; and on the preset linkage page, as the playback timeline of the aerial video files progresses, sequentially playing aerial images from the aerial image sequence corresponding to the aerial video files, wherein for each frame of an aerial image played, based on the time-synchronized location dataset, continuously updating and displaying a dynamic positioning icon along the spatial trajectory curve in the trajectory-overlay twin 3D scene model.

[0008] Secondly, some embodiments of this disclosure provide a device for linking and displaying engineering aerial photography results with a 3D model. The device includes: a loading unit configured to load a pre-constructed twin 3D scene model; an acquisition unit configured to acquire aerial video files, flight trajectory data, and time-synchronized location datasets of a target aerial photography flight; a coordinate transformation processing unit configured to perform coordinate transformation processing on the flight trajectory data to obtain scene coordinate system trajectory data; a rendering unit configured to render a spatial trajectory curve corresponding to the scene coordinate system trajectory data in the twin 3D scene model to obtain a trajectory overlay twin 3D scene model; a first display unit configured to display the trajectory overlay twin 3D scene model on a preset linkage page; and a second display unit configured to, on the preset linkage page, sequentially play aerial images from the aerial image sequence corresponding to the aerial video files as the playback timeline of the aerial video files progresses, wherein, for each frame of aerial image played, a dynamic positioning icon is continuously updated and displayed along the spatial trajectory curve in the trajectory overlay twin 3D scene model based on the time-synchronized location dataset.

[0009] Thirdly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation of the first aspect above.

[0010] Fourthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method described in any of the implementations of the first aspect above.

[0011] The above-described embodiments of this disclosure have the following beneficial effects: the method for linking and displaying engineering aerial photography results with 3D models in some embodiments of this disclosure improves the user experience. Specifically, the reason for the poor user experience is that: aerial videos and photos are only displayed in a two-dimensional list, and the flight path is represented by lines on a two-dimensional satellite map. There is a lack of synchronization and linkage between the collected videos and images and the drone's flight trajectory and the spatial position of the 3D model. This results in a disconnect between the video frame content and the 3D spatial position when the drone was shooting, making it difficult for engineering managers to quickly and accurately determine the specific location of engineering parts (such as a section of cofferdam or a gate chamber) appearing in the video in the 3D model, leading to a low user experience. Based on this, the method for linking and displaying engineering aerial photography results with 3D models in some embodiments of this disclosure first loads a pre-constructed twin 3D scene model. This allows loading a twin 3D scene model of the real engineering environment (terrain, BIM components, roads, water systems, etc.). Next, the aerial video files, flight trajectory data, and time-synchronized location datasets of the target aerial photography flights are acquired. Then, the flight trajectory data is processed by coordinate transformation to obtain scene coordinate system trajectory data. Therefore, flight trajectory data can be converted into trajectory data that can be directly used within the twin scene, allowing the trajectory line to correctly fit the twin 3D scene model. Then, a spatial trajectory curve corresponding to the trajectory data in the scene coordinate system is rendered in the twin 3D scene model, resulting in a trajectory-overlayed twin 3D scene model. This produces a visualized, interactive 3D spatial curve, allowing users to intuitively see the 3D path the drone has flown. Next, the trajectory-overlayed twin 3D scene model is displayed on a preset linked page. Then, on the preset linked page, as the playback timeline of the aerial video file progresses, aerial images from the corresponding aerial image sequence are played sequentially. For each frame of the aerial image played, based on a time-synchronized location dataset, a dynamic positioning icon is continuously updated and displayed along the spatial trajectory curve in the trajectory-overlayed twin 3D scene model. This achieves real-time synchronization between each frame of the video content and the corresponding shooting coordinates in the 3D scene, ensuring the dynamic positioning icon moves continuously along the trajectory curve, with its movement progress corresponding to the video playback progress. Because it employs a synchronization mechanism based on time-synchronized location datasets, retrieving and updating icon positions for each frame played, a dynamic mapping between the video timeline and the UAV's 3D spatial coordinates is established. This allows users to directly and in real-time see the specific landing point of the shooting location corresponding to the current frame in the 3D model, reducing the disconnect between video content and spatial location. This helps in determining the 3D model location corresponding to engineering parts appearing in the video, improving the efficiency of using aerial photography results for construction progress verification and safety hazard investigation, thereby enhancing the user experience. Attached Figure Description

[0012] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0013] Figure 1 This is a flowchart of some embodiments of the method for linking and displaying engineering aerial photography results with 3D models according to this disclosure; Figure 2 These are schematic diagrams of some embodiments of the aerial photography results and 3D model linkage display device based on this disclosure; Figure 3 This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure; Figure 4 These are internal test effect diagrams of some embodiments based on the linkage between the aerial photography results and the 3D model of the project disclosed herein. Detailed Implementation

[0014] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0015] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0016] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0017] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0018] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0019] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0020] Figure 1 A flowchart 100 is shown, illustrating some embodiments of the method for linking and displaying engineering aerial photography results with 3D models according to this disclosure. This method for linking and displaying engineering aerial photography results with 3D models includes the following steps: Step 101: Load the pre-built twin 3D scene model.

[0021] In some embodiments, the executing entity (e.g., a computing device) of the method for linking and displaying engineering aerial photography results with 3D models can load a pre-built twin 3D scene model. In practice, the executing entity can read the pre-built twin 3D scene model from a local disk, network storage, or database server using a 3D rendering engine. The 3D rendering engine refers to a software framework or library used to generate, manage, and display 3D scenes, such as Unreal Engine or a web graphics engine (e.g., Three.js for the web). The twin 3D scene model can be a virtual 3D scene representing a real engineering environment (including terrain, buildings, roads, water systems, vegetation, etc.) created through 3D digitization technology. The twin 3D scene model includes at least one of the following: an engineering BIM model (Building Information Modeling, BIM) and a 3D terrain elevation model. The engineering BIM model (Building Information Modeling) can be a digital representation of various components of an engineering project (such as bridge piers, beams, gate chambers, pipelines, etc.) integrated through 3D digital technology. The aforementioned three-dimensional terrain elevation model can be a terrain three-dimensional model used to represent the undulating shape of the ground. For example, for a waterway engineering project, the twin three-dimensional scene model can include a BIM model of the main structure of the lock and a DEM terrain grid covering a 10-kilometer range of the entire project.

[0022] Step 102: Obtain the aerial video files, flight trajectory data, and time-synchronized location datasets for the target aerial photography flights.

[0023] In some embodiments, the aforementioned executing entity can acquire aerial video files, flight trajectory data, and time-synchronized location datasets for the target aerial photography sortie. Each location data point in the location dataset records the UAV's spatial coordinates corresponding to a playback time point; the location data includes the playback time point and the UAV's spatial coordinates. The target aerial photography sortie can be a preset flight sortie, which can serve as an identifier to uniquely distinguish different batches of aerial photography missions. The aerial video file can be a video file continuously recorded by the UAV's onboard camera during flight, its content being a sequence of video frames arranged in chronological order, i.e., an aerial image sequence. The location dataset refers to a data set that correlates the playback time points of the aerial video files with the UAV's spatial coordinates one-to-one. Each aerial image in the aerial image sequence corresponds to a location data point in the location dataset, enabling the retrieval of the UAV's spatial location (i.e., UAV spatial coordinates) based on the playback time point when playing any frame of the video corresponding to the aerial video file. The playback time point refers to the moment the current video frame is located on the timeline of the aerial video file (e.g., 5 seconds after the start of the video corresponding to the aerial video file). The aforementioned flight trajectory data refers to the sequence of UAV spatial coordinates that record the UAV's flight path. Each UAV spatial coordinate represents the coordinate data of the UAV's spatial position in real three-dimensional space. These UAV spatial coordinates can be coordinates in a geographic coordinate system (longitude, latitude, and altitude). The aforementioned coordinate data can include longitude, latitude, and altitude.

[0024] Step 103: Perform coordinate transformation on the flight trajectory data to obtain trajectory data in the scene coordinate system.

[0025] In some embodiments, the aforementioned execution entity can perform coordinate transformation processing on the flight trajectory data to obtain scene coordinate system trajectory data. In practice, for each UAV spatial coordinate included in the flight trajectory data, the aforementioned execution entity can use Gauss-Kruger projection or Universal Transverse Mercator projection (UTM) technology to convert the UAV spatial coordinates in the geographic coordinate system into trajectory point coordinates in the local coordinate system corresponding to the twin 3D scene model. Then, the aforementioned execution entity can determine the obtained trajectory point coordinates as scene coordinate system trajectory data.

[0026] Step 104: Render the spatial trajectory curve corresponding to the trajectory data of the scene coordinate system in the twin 3D scene model to obtain the trajectory superimposed twin 3D scene model.

[0027] In some embodiments, the execution entity can render a spatial trajectory curve corresponding to the scene coordinate system trajectory data in the twin 3D scene model to obtain a trajectory-overlay twin 3D scene model. The scene coordinate system trajectory data includes a sequence of trajectory point coordinates. Each trajectory point coordinate can represent the spatial position of the UAV in the corresponding local coordinate system of the twin 3D scene model. The trajectory point coordinates include an abscissa, ordinate, and ordinate. The abscissa and ordinate represent planar position. The ordinate represents altitude. The spatial trajectory curve is a continuous three-dimensional line formed by connecting the trajectory point coordinates in the scene coordinate system trajectory data in chronological order, used to visualize the UAV's flight path. In practice, the execution entity can call the drawing interface of a 3D rendering engine (such as Three.js or Unreal Engine) to generate a curve based on the scene coordinate system trajectory data, add this curve as an independent model to the scene graph of the twin 3D scene model, and then re-render the entire scene to obtain a model containing the trajectory line as the trajectory-overlay twin 3D scene model.

[0028] In some optional implementations of certain embodiments, the aforementioned execution entity can render a spatial trajectory curve corresponding to the trajectory data of the aforementioned scene coordinate system in the aforementioned twin 3D scene model through the following steps, thereby obtaining a trajectory-overlaid twin 3D scene model: The first step is to perform trajectory smoothing interpolation on the aforementioned scene coordinate system trajectory data to obtain smooth trajectory data. In practice, the execution entity can iterate through each adjacent trajectory point coordinate pair in the scene coordinate system trajectory data, calculate the Euclidean distance between the two points represented by the adjacent trajectory point coordinate pairs, and if the distance is greater than or equal to a preset threshold (e.g., 0.2 meters), then linear interpolation points are inserted in step sizes (e.g., 0.1 meters) to obtain the interpolation point coordinates. These interpolation point coordinates are then used as the coordinates between the inserted trajectory point and the adjacent trajectory point coordinate pairs in the trajectory point coordinate sequence to update the trajectory point coordinate sequence. Finally, the updated trajectory point coordinate sequence can be determined as the smooth trajectory data. For example, if the trajectory point coordinate sequence can be {(0, 0, 0), (0.2, 0, 0)}, then the interpolation point coordinates between (0, 0, 0) and (0.2, 0, 0) can be (0.1, 0, 0). The updated trajectory point coordinate sequence, i.e., the smooth trajectory data, can be {(0, 0, 0), (0.1, 0, 0), (0.2, 0, 0)}.

[0029] The second step involves invoking the rendering interface of the aforementioned smooth trajectory data and preset geometric attribute information to render the geometry of the spatial trajectory curve within the twin 3D scene model. The preset geometric attribute information refers to a set of parameters defining the appearance of the curve, including but not limited to at least one of the following: line color (e.g., yellow: RGB value (255, 255, 0)), line width (e.g., 2 pixels), and material type (e.g., luminous material or solid line material). The rendering interface refers to the preset interface provided by the 3D engine for drawing lines.

[0030] The third step is to determine the twin 3D scene model containing the spatial trajectory curve as a trajectory superimposed twin 3D scene model.

[0031] Step 105: Display the trajectory overlaid twin 3D scene model on the preset linkage page.

[0032] In some embodiments, the aforementioned execution entity can display a trajectory-overlaid twin 3D scene model on a preset linked page. This preset linked page refers to a predefined graphical user interface that includes a main viewport area for rendering the 3D model and an interactive panel for controlling video playback and browsing the results list.

[0033] Step 106: On the preset linkage page, as the playback timeline of the aerial video file progresses, the aerial images in the aerial image sequence corresponding to the aerial video file are played sequentially.

[0034] In some embodiments, the aforementioned executing entity can, on the aforementioned preset linkage page, sequentially play aerial images from the aerial image sequence corresponding to the aforementioned aerial video file as the playback timeline of the aerial video file progresses. Specifically, for each frame of the aerial image played, a dynamic positioning icon is continuously updated and displayed along the spatial trajectory curve in the trajectory-overlayed twin 3D scene model based on a time-synchronized location dataset. Each aerial image in the aforementioned aerial image sequence has a corresponding playback time point, and each location data in the aforementioned location dataset records the drone's spatial coordinates corresponding to the playback time point. The location data includes the playback time point and the drone's spatial coordinates. The aforementioned dynamic positioning icon refers to a movable visual element in the 3D scene, used to indicate the drone's shooting position corresponding to the current video frame in the trajectory-overlayed twin 3D scene model.

[0035] In some optional implementations of certain embodiments, the aforementioned execution entity can, on the aforementioned preset linkage page, sequentially play aerial images from the aerial image sequence corresponding to the aforementioned aerial video file as the playback timeline of the aforementioned aerial video file progresses: The first step is to obtain the frame rate information of the aforementioned aerial image sequence. The frame rate information refers to the number of aerial images displayed per second, measured in frames per second (fps). For example, 30fps means 30 frames are played per second. In practice, the executing entity can read the frame rate from the metadata corresponding to the aerial video file using a video decoder.

[0036] The second step is to set the timer interval based on the frame rate information. This timer interval refers to the waiting time between the playback of two adjacent frames. In practice, the executing entity can determine the frame interval as the reciprocal of the frame rate represented by the frame rate information. Then, the frame interval can be set as the timer interval.

[0037] The third step involves reading and displaying aerial images sequentially from the above-mentioned aerial image sequence, using the aforementioned timer interval as the period.

[0038] In some optional implementations of certain embodiments, the aforementioned execution entity may continuously update and display a dynamic positioning icon by overlaying the spatial trajectory curve in the twin 3D scene model along the aforementioned trajectory, based on a time-synchronized location dataset, for each frame of aerial image played: The first step is to determine the playback time point corresponding to the first frame of the aerial image sequence as the reference time point when playing the first frame of the aerial image sequence.

[0039] The second step is to determine the location data whose playback time point in the above location dataset is the reference time point as the reference location data.

[0040] The third step involves displaying a pre-created positioning icon as a dynamic positioning icon at the location corresponding to the reference position data on the spatial trajectory curve in the aforementioned trajectory-overlay twin 3D scene model. In practice, the executing entity performs coordinate transformation based on the UAV spatial coordinates in the reference position data (the specific implementation method of coordinate transformation is the same as step 103) to obtain scene coordinates, and then sets and displays the positioning icon at the position in the aforementioned scene coordinates within the trajectory-overlay twin 3D scene model. The aforementioned scene coordinates can be the coordinates obtained by converting the UAV spatial coordinates in the reference position data from the geographic coordinate system to the aforementioned local coordinate system.

[0041] Fourth, when playing the aerial images that are not the first frame in the above aerial image sequence one by one, perform the following steps: The first sub-step is to determine the UAV spatial coordinates included in the above reference position data as the reference UAV spatial coordinates.

[0042] The second sub-step involves generating coordinates in the twin 3D scene model based on the aforementioned reference UAV spatial coordinates, which are then used as the current spatial coordinates. In practice, the executing entity can call a coordinate transformation function (such as the Gauss-Kruger projection formula) to map the reference UAV spatial coordinates to the coordinates in the twin 3D scene model as the current spatial coordinates.

[0043] The third sub-step is to determine the coordinates of the aforementioned dynamic positioning icon in the twin 3D scene model as historical spatial coordinates.

[0044] The fourth sub-step involves determining the distance between the current spatial coordinates and the historical spatial coordinates as the movement distance. This movement distance refers to the Euclidean distance between the current spatial coordinates and the historical spatial coordinates.

[0045] The fifth sub-step involves continuing to display the dynamic positioning icon at the location represented by the aforementioned historical spatial coordinates, in response to determining that the movement distance is less than a preset threshold. The preset threshold is a pre-defined distance value (e.g., 0.1 meters or 0.5 meters) used to determine whether the drone has undergone sufficiently significant displacement. If the movement distance is less than this threshold, the icon position change is considered negligible, and the icon position is not updated to conserve computing resources.

[0046] The sixth sub-step, in response to determining that the movement distance is greater than or equal to the aforementioned preset threshold, updates the dynamic positioning icon to its current spatial coordinate representation on the spatial trajectory curve in the aforementioned trajectory overlaid twin 3D scene model. In practice, the interface for moving objects in a 3D rendering engine can be called (e.g., `sprite.position.set(x, y, z)` in Three.js or `transform.position = new Vector3(x, y, z)` in Unity) to directly assign the position attribute of the dynamic positioning icon to the value of the current spatial coordinates. Then, a scene redraw is triggered (e.g., via `renderer.render(scene, camera)`) to display the dynamic positioning icon at its current spatial coordinate representation. The center coordinates of the dynamic positioning icon can be the aforementioned current spatial coordinates.

[0047] Optionally, the aforementioned implementing entity may also perform the following steps: The first step is to acquire the aerial panoramic image dataset. Each aerial panoramic image in this dataset includes the panoramic image itself and its location (latitude and longitude) information. The aerial panoramic image refers to a 360°×180° full-view image file (e.g., JPEG or PNG format) stitched together from multiple photos taken by a drone while hovering and rotating its camera. The location (latitude and longitude) information refers to the longitude, latitude, and altitude of the drone when the panoramic image was taken. In practice, the aforementioned entity can read the aerial panoramic image dataset locally.

[0048] The second step is to perform the following steps for each aerial panoramic image in the above aerial panoramic image dataset: The first sub-step involves performing coordinate transformation on the latitude and longitude information included in the aerial panoramic image data to obtain the spatial location information of the aerial panoramic image within the twin 3D scene model. In practice, the executing entity can call a coordinate transformation function (e.g., Gauss-Kruger projection) to map the coordinates (longitude, latitude, altitude) in the geographic coordinate system to three-dimensional coordinates in the local coordinate system used by the twin 3D scene model, thus providing the spatial location information of the aerial panoramic image within the twin 3D scene model. This local coordinate system can be a right-handed Cartesian coordinate system with a fixed control point at the project site as the origin, the X-axis pointing due east, the Y-axis pointing due north, and the Z-axis representing vertical elevation (positive upwards).

[0049] The second sub-step involves creating a panoramic image marker icon at the location corresponding to the panoramic spatial location information in the aforementioned twin 3D scene model. This panoramic image marker icon is a visual 2D or 3D icon (e.g., a camera model or a red thumbtack) used to indicate to the user that a viewable panoramic image exists at that location. In practice, the executing entity can generate an icon object that always faces the camera, set its position to the panoramic spatial location, and add it to the scene graph of the twin 3D scene model. The aforementioned icon object refers to a renderable graphical element used to visually mark a specific location in the 3D scene, such as a 2D sprite, a plane geometry, or a simple 3D model (e.g., a sphere or toroidal model).

[0050] The third sub-step involves, in response to a detected trigger action on the aforementioned panoramic image icon, popping up a panoramic image viewing window on the preset linked page. This window then renders and outputs the aerial panoramic image included in the aerial panoramic image data, supporting panoramic image rotation viewing. The trigger action can be a left mouse click, touch click, or other interactive action. The panoramic image viewing window is a floating layer or a standalone panel used to display the panoramic image and provide interactive controls. In practice, the executing entity can listen for click events on the panoramic image icon. When the event occurs, an HTML div element is dynamically created as a floating window, and a panoramic image viewer component (e.g., a panoramic viewer based on Three.js or A-Frame) is embedded within it, loading the aerial panoramic image as a texture and rendering it.

[0051] In some optional implementations of certain embodiments, the aforementioned execution entity may render and output the aerial panoramic image included in the aforementioned aerial panoramic image data in the aforementioned panoramic image viewing window through the following steps to support panoramic image rotation viewing: The first step involves performing isometric cylindrical projection decoding on the aerial panoramic images included in the aforementioned aerial panoramic image data to obtain spherical texture mapping data. The aforementioned aerial panoramic image refers to an omnidirectional image file with a 360° horizontal field of view and a 180° vertical field of view, stitched together from multiple photos taken by a drone while hovering and rotating its camera. This file is stored in an isometric cylindrical projection format, meaning the image width corresponds to a horizontal 360° azimuth angle and the height corresponds to a vertical 180° elevation angle; common formats are JPEG or PNG. The aforementioned isometric cylindrical projection decoding process refers to the process of decompressing the isometric cylindrical projection format panoramic image file and converting it into pixel data usable by the graphics engine. The aforementioned spherical texture mapping data refers to the decoded image pixel matrix (e.g., a one-dimensional or two-dimensional array in RGBA format). This matrix can be used as a texture map and bound to the inner surface of a spherical mesh model to achieve a spherical display of the panoramic image. In practice, the aforementioned execution entity can call an image decoding library (such as the browser's built-in Image object or Three.js's TextureLoader) to read the panoramic image file, and after decoding, directly obtain the ImageData or Texture object. The pixel data in this object is the spherical texture mapping data.

[0052] The second step involves creating a panoramic spherical mesh model at the center of the panoramic view window and attaching the spherical texture mapping data to the inner surface of this spherical mesh model. The panoramic view window refers to the overlay or independent panel displaying the panoramic image. The panoramic spherical mesh model is a three-dimensional geometry with a fixed radius (e.g., 1 meter) facing the center of the sphere, typically using a SphereGeometry. The inner surface refers to the inner surface of the sphere, as the observer needs to view the texture on the inner wall when at the center. In practice, the execution entity performs the following operations through the 3D rendering engine: First, it calls the geometry generation function to create a spherical mesh object with a preset radius (e.g., 1 meter) and a preset number of segments (e.g., 64 segments). Then, it creates a base material object, sets the surface rendering direction of this material object to the back side so that the texture is visible when the camera is inside the sphere, and assigns the spherical texture mapping data to the texture properties of this material. Finally, it combines the spherical mesh object and the material object into a complete mesh model. Finally, the scene addition method is called to add the mesh model to the root node of the scene tree in the panoramic view window, and its position coordinates are set to the origin (i.e., the center position) of the window space. The geometry generation function mentioned above is an interface in the 3D rendering engine used to create basic geometric shapes (such as spheres and cubes). The radius mentioned above refers to the distance from the center of the sphere to its surface. The number of segments mentioned above refers to how many small planes (mesh faces) the sphere's surface is divided into. The sphere mesh object mentioned above refers to a data structure composed of vertices and triangle faces, defining only the geometry. The basic material object mentioned above refers to a set of material parameters that are unaffected by scene lighting and only display the original texture color. The surface rendering direction mentioned above refers to which side (front, back, or both sides) the material is applied to. The back side mentioned above refers to the opposite side of the triangle face (the normal points inwards). The texture attribute mentioned above refers to the field in the material used to receive texture data. The mesh model mentioned above refers to a visual entity generated after binding geometry to materials, which can be added to the scene. The scene addition method mentioned above refers to the function provided by the 3D rendering engine to add objects to the scene graph.

[0053] The third step involves obtaining the mouse movement offset in response to a detected mouse drag event. The mouse drag event refers to a sequence of events triggered continuously by the operating system or browser when the user presses the left mouse button and moves the mouse across the screen. The mouse movement offset refers to the pixel difference between the previous event trigger point and the current event trigger point, moving along the horizontal (X-axis) and vertical (Y-axis) directions of the screen. This offset includes both horizontal and vertical offsets. For example, when the user presses the left mouse button, the initial screen coordinates are recorded, and a drag flag is set to true. When the mouse moves, if the drag flag is true, the difference between the current screen coordinates and the initial coordinates is calculated to obtain the offset. To achieve a continuous dragging effect, the offset is updated after each mouse movement event and immediately used for the next rotation calculation. For example, if the user drags 100 pixels to the right, the offset is (horizontal offset: 100, vertical offset: 0); if dragging 50 pixels upwards, the offset is (horizontal offset: 0, vertical offset: -50). Then, update the starting coordinates to the current coordinates for use in the next movement event.

[0054] The fourth step is to determine the Euler angles of the panoramic spherical mesh model based on the aforementioned offset. These Euler angles describe the angles used to describe the rotation of a 3D object around a coordinate axis, including at least one of the following: the horizontal rotation angle (yaw angle) around the Y-axis and the vertical rotation angle (pitch angle) around the X-axis. First, the execution entity can preset a sensitivity coefficient (e.g., 0.005) to convert the pixel offset on the screen into a change in rotation angle. Then, the horizontal offset is multiplied by the sensitivity coefficient to obtain the change in yaw angle. The vertical offset is multiplied by the sensitivity coefficient to obtain the change in pitch angle. Next, the existing yaw and pitch angles of the current panoramic spherical mesh model are read. Then, the sum of the change in yaw angle and the existing yaw angles can be determined as the horizontal rotation angle (yaw angle). Next, the sum of the change in pitch angle and the existing pitch angles can be determined as the vertical rotation angle (pitch angle) around the X-axis. Finally, the horizontal and vertical rotation angles can be determined as the Euler angles of rotation. It should be noted that in order to prevent the vertical viewing angle from exceeding the reasonable range (for example, causing the image to be upside down), the vertical rotation angle needs to be limited: if the vertical rotation angle is greater than π / 2 (i.e., 90 degrees), then it is taken as π / 2; if the vertical rotation angle is less than -π / 2 (i.e., -90 degrees), then it is taken as -π / 2.

[0055] The fifth step involves updating the rotation matrix of the panoramic spherical mesh model based on the aforementioned Euler angles and re-rendering the panoramic spherical mesh model to support rotating viewing of the panoramic image. In practice, the execution entity can add a preset roll angle (e.g., 0 radians) to the aforementioned Euler angles to update them. For example, the updated Euler angles could be: vertical rotation angle: -0.5 radians (pitch angle), horizontal rotation angle (yaw angle): 1 radian, roll angle: 0 radians. In practice, the Euler angles can be converted into a rotation matrix using the aforementioned 3D rendering engine, and the converted rotation matrix can be applied to the panoramic spherical mesh model of the panoramic image. Then, the renderer's rendering function (e.g., renderer.render(scene, camera)) is called to update the current rotation state of the panoramic spherical mesh model, and the updated panoramic spherical mesh model is rendered onto the screen, allowing the user to see the rotated and adjusted panoramic image. The aforementioned rotation matrix can be a 3×3 rotation matrix obtained by multiplying the three basic rotation matrices corresponding to the vertical rotation angle, horizontal rotation angle, and roll angle in the Euler rotation angle (i.e., the rotation matrix around the X-axis (pitch angle), the rotation matrix around the Y-axis (yaw angle), and the rotation matrix around the Z-axis (roll angle)) in a preset order (e.g., first rotate the yaw angle around the Y-axis, then rotate the pitch angle around the X-axis, and finally rotate the roll angle around the Z-axis).

[0056] In the process of adopting technical solutions to address the technical problems mentioned above, for the application scenario: interactive inspection and multi-dimensional viewing of drone aerial photography results in the digital twin platform of large-scale engineering construction projects (such as roads, municipal works, and waterway projects), the following technical problems often arise: aerial videos only provide continuous images from a fixed perspective of the drone camera, and cannot show the all-round environment around the same shooting location (such as the area behind the lens, or hidden structures on the side). Although existing panoramic images can provide 360°×180° full field of view information, there is a lack of dynamic spatial and temporal correlation between the video and the panoramic image. Users need to manually pause the video, estimate the current drone coordinates, and then search the panoramic image library separately, resulting in many operation steps, slow response speed, and a low user experience. Requirements: The digital twin platform needs to support users in pausing the video with one click by clicking on a moving dynamic positioning icon during inspection video playback. This will automatically bring up an aerial panoramic view that matches the current position of the icon (i.e., the drone's spatial coordinates corresponding to the current video frame), thereby improving the efficiency and ease of operation for engineering management personnel to fully perceive on-site construction details, safety hazards, and environmental changes.

[0057] Optionally, the aforementioned implementing entity may also perform the following steps: The first step is to identify the aerial image sequence corresponding to the aforementioned aerial video file as the target aerial image sequence. Each target aerial image in the target aerial image sequence has a corresponding playback time point, and each location data point in the aforementioned location dataset records the UAV's spatial coordinates corresponding to that playback time point. The location data includes both the playback time point and the UAV's spatial coordinates.

[0058] The second step is to perform the following steps for each location data in the above location dataset: The first sub-step involves retrieving an aerial panoramic image matching the drone's spatial coordinates from a pre-stored panoramic image library, based on the location data included above. This panoramic image library can store drone spatial coordinates and aerial panoramic images. Figure 1 A pre-defined database with one-to-one correspondence.

[0059] The second sub-step involves determining the aforementioned location data, including the playback time point, as the target playback time point.

[0060] The third sub-step involves identifying the target aerial images in the above target aerial image sequence whose playback time point is the target playback time point as associated aerial images.

[0061] The fourth sub-step involves determining the aforementioned aerial panoramic image and the aforementioned target playback time point as the dynamic positioning icon association information corresponding to the aforementioned associated aerial image.

[0062] The fifth sub-step involves defining the associated information of each dynamic positioning icon as a set of dynamic positioning icon association information. Each displayed dynamic positioning icon corresponds to one piece of dynamic positioning icon association information in this set, which includes the UAV's spatial coordinates and the aerial panoramic image.

[0063] The sixth sub-step involves pausing the aerial video file in response to an detected interactive action on the displayed dynamic positioning icon, and displaying a panoramic photo viewing control on the aforementioned preset linked page. This interactive action can be a left-click on the icon. The panoramic photo viewing control is a button or prompt bar that asks the user whether to view the associated panoramic image. In practice, the execution entity is bound to a click event listener for the dynamic positioning icon; when a click occurs, the video player's pause method is invoked, and a "View Panoramic Image" button is displayed at the bottom of the page or near the icon.

[0064] The seventh sub-step involves detecting an interactive operation performed on the panoramic photo viewing control, and determining the dynamic positioning icon association information of the above-mentioned dynamic positioning icon association information and the dynamic positioning icon association information corresponding to the displayed dynamic positioning icon as the target dynamic positioning icon association information.

[0065] The eighth sub-step involves displaying an aerial panoramic view in the above-mentioned trajectory-overlay twin 3D scene model, which includes information associated with the above-mentioned target dynamic positioning icon.

[0066] The above-described technical solution and its related content, as an inventive point of this disclosure, solve the technical problem of "low user experience". Factors leading to a low user experience often include: aerial videos only provide continuous images from a fixed perspective of the drone camera, failing to display the all-around environment around the same shooting location (such as areas behind the lens, lateral concealed structures). Although existing panoramic images can provide 360°×180° full-view information, there is a lack of dynamic spatial and temporal correlation between the video and the panoramic image. Users need to manually pause the video, estimate the current drone coordinates, and then search the panoramic image library, resulting in numerous operation steps and slow response speed, leading to a low user experience. Solving these factors can improve the user experience. To achieve this, firstly, the aerial image sequence corresponding to the above-described aerial video file is determined as the target aerial image sequence. Each target aerial image in the target aerial image sequence has a corresponding playback time point, and each location data in the location dataset records the drone's spatial coordinates corresponding to the playback time point. The location data includes the playback time point and the drone's spatial coordinates. Then, for each location data in the aforementioned location dataset, the following steps are performed: First, based on the UAV spatial coordinates included in the aforementioned location data, retrieve an aerial panoramic image matching the spatial coordinates from a pre-stored panoramic image library. This yields a panoramic image matching the current video frame position. Second, determine the playback time point included in the aforementioned location data as the target playback time point. Third, determine the target aerial image in the aforementioned target aerial image sequence whose playback time point is the aforementioned target playback time point as the associated aerial image. Fourth, determine the aforementioned aerial panoramic image and the aforementioned target playback time point as the dynamic positioning icon association information corresponding to the aforementioned associated aerial image. This yields the panoramic image mapping relationship corresponding to each dynamic positioning icon during video playback. Then, the determined dynamic positioning icon association information is defined as a dynamic positioning icon association information set, where each displayed dynamic positioning icon corresponds to one dynamic positioning icon association information in the aforementioned dynamic positioning icon association information set. The dynamic positioning icon association information includes the UAV spatial coordinates and the aerial panoramic image. This yields the panoramic image mapping relationship corresponding to each dynamic positioning icon during video playback. Subsequently, in response to the detection of an interaction action performed on the displayed dynamic positioning icon, the aerial video file playback is paused, and a panoramic photo viewing control is displayed on the aforementioned preset linked page. Thus, the user can trigger interaction by clicking the dynamic icon. Next, in response to the detection of an interaction action performed on the panoramic photo viewing control, the dynamic positioning icon association information is combined with the dynamic positioning icon association information corresponding to the displayed dynamic positioning icon to determine the target dynamic positioning icon association information. This allows the panoramic image data corresponding to the user-selected icon to be located.Then, the aerial panoramic image, which includes the information associated with the aforementioned target dynamic positioning icon, is displayed in the above-mentioned trajectory-overlaid twin 3D scene model. This allows switching from video frames to the corresponding panoramic image display. Because a pre-established mapping information set of video frames, drone coordinates, and panoramic images is used, and the video is paused and the associated panoramic image automatically pops up in response to the user's click on the dynamic positioning icon, the fixed perspective of the video is dynamically linked to the omnidirectional perspective of the panoramic image. This significantly reduces the user's operation steps and search time, allowing users to quickly and intuitively obtain omnidirectional environmental information from the same shooting location, thus improving the user experience.

[0067] In the process of adopting technical solutions to address the technical problems mentioned above, for the application scenario: in the digital twin platform of large-scale engineering construction projects, dynamically displaying aerial panoramic images associated with video frames in a 3D scene often presents the following technical problems: when directly displaying the image on the spherical grid of the panoramic image, the inconsistent lighting environment (such as sun direction, brightness, and color temperature) during the panoramic image capture with the preset virtual light source parameters in the 3D scene leads to a visual incoordination between the panoramic image and the surrounding 3D model. Users find it difficult to accurately compare the geometric texture differences between the real image and the virtual model, resulting in a low user experience. At the same time, if outdated panoramic image display objects are not destroyed and resources are not released, clicking the dynamic positioning icon at different locations multiple times will cause multiple panoramic image grids to exist in the scene simultaneously, resulting in a continuous increase in video memory usage and a waste of video memory resources. The following requirements are necessary for this application scenario: The digital twin platform needs to automatically analyze the lighting characteristics (brightness, color temperature, main light source direction, ambient light intensity) of the panoramic image and adjust the virtual light source parameters of the 3D scene in real time to ensure that the lighting environment during panoramic image display is consistent with the original shooting lighting, thereby improving the accuracy of virtual-real contrast. To address these technical challenges, we have decided to adopt the following solution: In some optional implementations of certain embodiments, the aforementioned execution entity may display an aerial panoramic image, including information associated with the aforementioned target dynamic positioning icon, in the aforementioned trajectory-overlay twin 3D scene model through the following steps: The first step is to perform a mapping transformation on the UAV spatial coordinates included in the aforementioned target dynamic positioning icon association information to obtain the transformed coordinates. In practice, the aforementioned execution entity can use Gauss-Kruger projection or Universal Transverse Mercator (UTM) projection technology to convert the UAV spatial coordinates (longitude, latitude, altitude) in the geographic coordinate system into three-dimensional coordinates in the local coordinate system corresponding to the twin 3D scene model as the transformed coordinates.

[0068] The second step involves generating the display location information of the aerial panoramic image within the aforementioned twin 3D scene model based on the transformed coordinates. In practice, the executing entity can determine the display location information using the transformed coordinates. Alternatively, the executing entity can combine the transformed coordinates as the center point coordinates with a preset radius (e.g., 0.5 meters) as the display size to form the display location information.

[0069] The third step involves performing an existence detection on the panoramic display objects in the aforementioned trajectory-overlay twin 3D scene model to obtain existence detection information. Here, the panoramic display object refers to the panoramic spherical mesh or other visualization elements already existing in the current twin 3D scene model. This existence detection information is a Boolean value (true or false). In practice, the execution entity can check whether the twin 3D scene model contains a node with a preset name (e.g., "panoramaObject"). In response to determining that a node with a preset name exists in the twin 3D scene model, a Boolean value (e.g., true) indicating the existence of a panoramic display object is determined as the existence detection information. In response to determining that a node with a preset name does not exist in the twin 3D scene model, a Boolean value (e.g., false) indicating the absence of a panoramic display object is determined as the existence detection information.

[0070] Fourth, in response to the determination that the aforementioned existence detection information indicates the presence of a panoramic image display object, a destruction operation is performed on the panoramic image display object in the trajectory overlay twin 3D scene model. In practice, the above-mentioned execution entity can delete the panoramic image display object by calling the remove method in the scene, i.e., the trajectory overlay twin 3D scene model, and release GPU resources by calling the dispose methods of textures and geometry.

[0071] The fifth step involves extracting lighting features from the aerial panoramic image associated with the aforementioned target dynamic positioning icon to obtain the panoramic image's lighting feature information. This information includes the mean luminance histogram, the main light source direction angle, the mean color temperature, and the ambient light intensity value of the panoramic image. The mean luminance histogram refers to the average luminance of all pixels in the aerial panoramic image (range 0-255, normalized to 0-1). The executing entity can determine the main light source direction angle using a method based on luminance integration along the longitude direction. The main light source direction angle refers to the horizontal direction of illumination from the main light source (e.g., the sun, searchlight, or strong lighting at a construction site) in the aerial panoramic image. The mean color temperature refers to the Kelvin temperature value corresponding to the overall color tone of the aerial panoramic image. The ambient light intensity value of the panoramic image refers to the average luminance of the dark areas (the 10% of pixels with the lowest luminance) in the aerial panoramic image. In practice, the executing entity can use OpenCV to analyze the aerial panoramic image: first, convert the panoramic image to grayscale, and then calculate the mean of the histogram as the mean luminance histogram. Then, each pixel in the panoramic image is grouped by longitude (horizontal angle), and the sum of the brightness of all pixels within each longitude interval is calculated. The angle corresponding to the longitude interval with the largest total brightness is the direction angle of the main light source. After that, the RGB channels are white balanced and mapped to color temperature, and the average value of the 10% of pixels with the lowest brightness is taken as the ambient light intensity.

[0072] Step 6: Read the virtual light source parameter set of the above-mentioned trajectory-overlay twin 3D scene model to obtain virtual lighting parameter data. This virtual lighting parameter data includes the current brightness value, current color temperature value, directional light angle, and ambient light intensity value. Specifically, the virtual lighting parameter data includes the current brightness value (overall scene brightness, normalized 0-1), the current color temperature value (stored in Kelvin or RGB format), the directional light angle (horizontal angle of the main light source, 0-360°), and the ambient light intensity value (ambient light brightness). In practice, the above-mentioned execution entity can read these parameters from the 3D engine's lighting manager.

[0073] The seventh step is to determine the panoramic image's lighting characteristics as the target virtual light source parameter data.

[0074] Step 8: Based on the aforementioned target virtual light source parameter data, update the aforementioned virtual light source parameter set to update the virtual light source of the aforementioned trajectory-overlay twin 3D scene model, obtaining a trajectory-overlay twin 3D scene model with dynamically adjusted lighting. In practice, the aforementioned execution entity can convert the values ​​in the target virtual light source parameter data (mean value of brightness histogram, main light source direction angle, mean color temperature, and ambient light intensity value of the panoramic image) into corresponding parameters in the virtual light source parameter set. The virtual light source parameters of the 3D scene have been updated to target values ​​consistent with the lighting characteristics of the panoramic image. Subsequently, the 3D rendering engine re-executes lighting rendering based on the updated virtual light source parameter set to update the virtual light source of the aforementioned trajectory-overlay twin 3D scene model, obtaining a trajectory-overlay twin 3D scene model with dynamically adjusted lighting.

[0075] Step nine involves generating a panoramic spherical support mesh within the adjusted trajectory-overlay twin 3D scene model, based on the aforementioned display position information. This panoramic spherical support mesh is a 3D mesh object with a fixed radius and its face facing the center of the sphere; it is typically a sphere geometry, and its position is set to the display position information. The executing entity calls the geometry creation function of the 3D rendering engine to generate the panoramic spherical support mesh within the adjusted trajectory-overlay twin 3D scene model, based on the aforementioned display position information.

[0076] Step 10: The aerial panoramic image, included in the target dynamic positioning icon association information, is used as a texture and bound to the panoramic image spherical support mesh to obtain a texture-bound panoramic image mesh. Here, the texture refers to an image resource that can be attached to the surface of a 3D model after 2D image data is loaded into a graphics processing unit (GPU). Binding refers to assigning the texture to the material properties of the mesh object, enabling the material to use the texture map during rendering. In practice, the execution entity first converts the aerial panoramic image file into a GPU-recognizable texture object using a texture loader (e.g., TextureLoader). Then, a basic material object (e.g., MeshBasicMaterial) is created, its map property is set to the texture object, and the rendering face of the material is set to the backside. Finally, this material is assigned to the panoramic image spherical support mesh generated in step 9 to obtain the texture-bound panoramic image mesh.

[0077] Step 11: Invoke the rendering pipeline of the aforementioned trajectory-overlay twin 3D scene model to perform a single-draw operation on the texture-bound panoramic mesh. This generates an embedded panoramic display window at the spatial location corresponding to the aforementioned display position information to display the aerial panoramic image. In practice, the executing entity can add the texture-bound panoramic mesh to the scene root node of the adjusted trajectory-overlay twin 3D scene model, and then call the renderer's rendering function (e.g., renderer.render(scene, camera)). The engine will then draw the mesh onto the screen in the next frame. Since the mesh's position is fixed at the spatial coordinates corresponding to the display position information, and its inner surface is covered with a panoramic texture, the user can see a spherical panoramic display window floating near the engineering component in the 3D scene. The user can observe it from the outside by rotating the camera viewpoint, or click to enter and view it in detail.

[0078] The above-described technical solution and its related content, as an inventive point of this disclosure, solve the technical problem of "low user experience and wasted video memory resources." Factors leading to low user experience and wasted video memory resources often include: when directly displaying textures on the panoramic image's spherical mesh, the inconsistency between the actual lighting environment (such as sun direction, brightness, and color temperature) during panoramic image capture and the preset virtual light source parameters in the 3D scene results in visual disharmony between the panoramic image and the surrounding 3D model. Users find it difficult to accurately compare the geometric texture differences between the real image and the virtual model, leading to a low user experience. Furthermore, if outdated panoramic image display objects are not destroyed and their resources released, repeatedly clicking dynamic positioning icons at different locations will cause multiple panoramic image meshes to exist simultaneously in the scene, resulting in continuously increasing video memory usage and wasted video memory resources. Solving these factors can achieve the effect of lowering user experience and reducing video memory waste. To achieve this effect, firstly, the drone spatial coordinates, including the information associated with the aforementioned target dynamic positioning icons, are mapped and transformed to obtain the transformed coordinates. Then, based on the transformed coordinates, the display position information of the aforementioned aerial panoramic image in the aforementioned twin 3D scene model is generated. Therefore, the display position information of the panoramic image in the 3D scene can be obtained. Then, the existence of the panoramic image display object in the trajectory-overlay twin 3D scene model is checked to obtain existence detection information. Next, in response to the existence detection information indicating the existence of the panoramic image display object, a destruction operation is performed on the panoramic image display object in the trajectory-overlay twin 3D scene model. This ensures that only the latest panoramic image is displayed each time, and releases the GPU texture and mesh resources occupied by older objects, avoiding memory accumulation and reducing memory waste. Next, the aerial panoramic image, which is associated with the target dynamic positioning icon, is processed for lighting feature extraction to obtain panoramic image lighting feature information, including the average brightness histogram, the direction angle of the main light source, the average color temperature, and the ambient light intensity value of the panoramic image. This allows for the quantification of the actual lighting parameters during panoramic image capture. Then, the virtual light source parameter set of the trajectory-overlay twin 3D scene model is read to obtain virtual lighting parameter data, including the current brightness value, the current color temperature value, the direction angle of the directional light, and the ambient light intensity value. Next, the panoramic image's lighting characteristics are used as target virtual light source parameter data. Then, based on this target virtual light source parameter data, the virtual light source parameter set is updated to update the virtual light source in the trajectory-overlay twin 3D scene model, resulting in a dynamically adjusted trajectory-overlay twin 3D scene model. This ensures that the lighting environment of the 3D scene (brightness, color temperature, light source direction, ambient light) remains consistent with the actual lighting during panoramic image capture. Finally, based on the aforementioned display position information, a panoramic spherical support mesh is generated within the adjusted trajectory-overlay twin 3D scene model.Next, the aerial panoramic image, which is associated with the aforementioned target dynamic positioning icon, is used as a texture and bound to the aforementioned panoramic image spherical support mesh, resulting in a texture-bound panoramic image mesh. Finally, the rendering pipeline of the aforementioned trajectory overlay twin 3D scene model is invoked to perform a single-draw operation on the aforementioned texture-bound panoramic image mesh, generating an embedded panoramic image display window at the spatial location corresponding to the aforementioned display position information to display the aerial panoramic image. Because the old object is destroyed before a new object is created, multiple panoramic image objects are avoided from existing simultaneously, saving video memory resources. Simultaneously, by extracting the panoramic image's lighting characteristics and dynamically adjusting the 3D scene's virtual light source, the lighting differences between the panoramic image and the 3D model are eliminated, allowing users to compare images from the same angle under a unified lighting environment, thereby improving the user experience.

[0079] The above-described embodiments of this disclosure have the following beneficial effects: the method for linking and displaying engineering aerial photography results with 3D models in some embodiments of this disclosure improves the user experience. Specifically, the reason for the poor user experience is that: aerial videos and photos are only displayed in a two-dimensional list, and the flight path is represented by lines on a two-dimensional satellite map. There is a lack of synchronization and linkage between the collected videos and images and the drone's flight trajectory and the spatial position of the 3D model. This results in a disconnect between the video frame content and the 3D spatial position when the drone was shooting, making it difficult for engineering managers to quickly and accurately determine the specific location of engineering parts (such as a section of cofferdam or a gate chamber) appearing in the video in the 3D model, leading to a low user experience. Based on this, the method for linking and displaying engineering aerial photography results with 3D models in some embodiments of this disclosure first loads a pre-constructed twin 3D scene model. This allows loading a twin 3D scene model of the real engineering environment (terrain, BIM components, roads, water systems, etc.). Next, the aerial video files, flight trajectory data, and time-synchronized location datasets of the target aerial photography flights are acquired. Then, the flight trajectory data is processed by coordinate transformation to obtain scene coordinate system trajectory data. Therefore, flight trajectory data can be converted into trajectory data that can be directly used within the twin scene, allowing the trajectory line to correctly fit the twin 3D scene model. Then, a spatial trajectory curve corresponding to the trajectory data in the scene coordinate system is rendered in the twin 3D scene model, resulting in a trajectory-overlayed twin 3D scene model. This produces a visualized, interactive 3D spatial curve, allowing users to intuitively see the 3D path the drone has flown. Next, the trajectory-overlayed twin 3D scene model is displayed on a preset linked page. Then, on the preset linked page, as the playback timeline of the aerial video file progresses, aerial images from the corresponding aerial image sequence are played sequentially. For each frame of the aerial image played, based on a time-synchronized location dataset, a dynamic positioning icon is continuously updated and displayed along the spatial trajectory curve in the trajectory-overlayed twin 3D scene model. This achieves real-time synchronization between each frame of the video content and the corresponding shooting coordinates in the 3D scene, ensuring the dynamic positioning icon moves continuously along the trajectory curve, with its movement progress corresponding to the video playback progress. Because it employs a synchronization mechanism based on time-synchronized location datasets, retrieving and updating icon positions for each frame played, a dynamic mapping between the video timeline and the UAV's 3D spatial coordinates is established. This allows users to directly and in real-time see the specific landing point of the shooting location corresponding to the current frame in the 3D model, reducing the disconnect between video content and spatial location. This helps in determining the 3D model location corresponding to engineering parts appearing in the video, improving the efficiency of using aerial photography results for construction progress verification and safety hazard investigation, thereby enhancing the user experience.

[0080] Figure 4These are internal test effect diagrams of some embodiments based on the linkage between the aerial photography results and the 3D model of the project disclosed herein.

[0081] Further reference Figure 2 As an implementation of the methods shown in the figures, this disclosure provides some embodiments of a device for linking engineering aerial photography results with a 3D model for display. These device embodiments are similar to... Figure 1 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices.

[0082] like Figure 2 As shown, the engineering aerial photography results and 3D model linkage display device 200 in some embodiments includes: a loading unit 201, an acquisition unit 202, a coordinate transformation processing unit 203, a rendering unit 204, a first display unit 205, and a second display unit 206. The system includes the following components: a loading unit, configured to load a pre-built twin 3D scene model; an acquisition unit, configured to acquire aerial video files, flight trajectory data, and time-synchronized location datasets for the target aerial flight; a coordinate transformation processing unit, configured to perform coordinate transformation processing on the flight trajectory data to obtain scene coordinate system trajectory data; a rendering unit, configured to render a spatial trajectory curve corresponding to the scene coordinate system trajectory data in the twin 3D scene model to obtain a trajectory-overlay twin 3D scene model; a first display unit, configured to display the trajectory-overlay twin 3D scene model on a preset linked page; and a second display unit, configured to play aerial images from the aerial image sequence corresponding to the aerial video files sequentially on the preset linked page as the playback timeline of the aerial video files progresses. For each frame of aerial image played, a dynamic positioning icon is continuously updated and displayed along the spatial trajectory curve in the trajectory-overlay twin 3D scene model based on the time-synchronized location dataset.

[0083] It is understandable that the units described in the device 200 are related to the reference. Figure 1 The steps in the method described above correspond to each other. Therefore, the operations, features, and beneficial effects described above for the method also apply to the device 200 and the units contained therein, and will not be repeated here.

[0084] The following is for reference. Figure 3 It shows a schematic diagram of the structure of an electronic device 300 suitable for implementing some embodiments of the present disclosure. Figure 3 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0085] like Figure 3As shown, the electronic device 300 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. The RAM 303 also stores various programs and data required for the operation of the electronic device 300. The processing unit 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0086] Typically, the following devices can be connected to I / O interface 305: input devices 306 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 308 including, for example, magnetic tapes, hard disks, etc.; and communication devices 309. Communication device 309 allows electronic device 300 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 3 An electronic device 300 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 3 Each box shown can represent a device or multiple devices as needed.

[0087] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 309, or installed from storage device 308, or installed from ROM 302. When the computer program is executed by processing device 301, it performs the functions defined in the methods of some embodiments of this disclosure.

[0088] It should be noted that, in some embodiments of this disclosure, the computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0089] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0090] The computer-readable medium may be included in an electronic device or may exist independently without being assembled into the electronic device. The computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: load a pre-built twin 3D scene model; acquire aerial video files, flight trajectory data, and time-synchronized location datasets for the target aerial flight; perform coordinate transformation processing on the flight trajectory data to obtain scene coordinate system trajectory data; render a spatial trajectory curve corresponding to the scene coordinate system trajectory data in the twin 3D scene model to obtain a trajectory-overlay twin 3D scene model; display the trajectory-overlay twin 3D scene model on a preset linked page; and on the preset linked page, as the playback timeline of the aerial video files progresses, aerial images from the corresponding aerial image sequence are played sequentially, wherein for each frame of aerial image played, a dynamic positioning icon is continuously updated and displayed along the spatial trajectory curve in the trajectory-overlay twin 3D scene model based on the time-synchronized location dataset.

[0091] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0092] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0093] The units described in some embodiments of this disclosure can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including a loading unit, an acquisition unit, a coordinate transformation processing unit, a rendering unit, a first display unit, and a second display unit. The names of these units do not necessarily limit the specific unit; for example, a loading unit may also be described as "a unit that loads a pre-built twin 3D scene model."

[0094] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0095] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of technical features, but should also cover other technical solutions formed by arbitrary combinations of technical features or their equivalents without departing from the inventive concept. For example, technical solutions formed by substituting features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A method for linking and displaying engineering aerial photography results with 3D models, comprising: Load the pre-built twin 3D scene model; Acquire aerial video files, flight trajectory data, and time-synchronized location datasets for the target aerial photography flights; The flight trajectory data is subjected to coordinate transformation to obtain trajectory data in the scene coordinate system; In the twin 3D scene model, a spatial trajectory curve corresponding to the trajectory data of the scene coordinate system is rendered to obtain a trajectory superimposed twin 3D scene model; The trajectory is overlaid with a twin 3D scene model and displayed on a preset linked page; On the preset linkage page, as the playback timeline of the aerial video file progresses, aerial images from the corresponding aerial image sequence of the aerial video file are played sequentially. For each frame of the aerial image played, based on the time-synchronized location dataset, the spatial trajectory curve in the twin 3D scene model is continuously updated and a dynamic positioning icon is displayed along the trajectory.

2. The method of claim 1, wherein, The method further includes: Obtain an aerial panoramic image dataset, wherein each aerial panoramic image in the dataset includes an aerial panoramic image and location latitude and longitude information; For each aerial panoramic image in the aerial panoramic image dataset, perform the following steps: The positioning latitude and longitude information included in the aerial panoramic image data is subjected to coordinate transformation processing to obtain the panoramic spatial location information of the aerial panoramic image in the twin 3D scene model. Create a panoramic image identifier icon at the location corresponding to the panoramic image spatial location information in the twin 3D scene model; In response to the detection of a trigger operation on the panoramic image icon, a panoramic image viewing window pops up on the preset linked page, and the aerial panoramic image included in the aerial panoramic image data is rendered and output in the panoramic image viewing window to support panoramic image rotation viewing.

3. The method of claim 1, wherein, The scene coordinate system trajectory data includes the coordinates of each trajectory point, and the spatial trajectory curve corresponding to the scene coordinate system trajectory data is rendered in the twin 3D scene model to obtain a trajectory superimposed twin 3D scene model, including: The trajectory data in the scene coordinate system is subjected to trajectory smoothing interpolation to obtain smooth trajectory data; Based on the smooth trajectory data and preset geometric attribute information, the rendering interface of the twin 3D scene model is called to render the geometry of the spatial trajectory curve in the twin 3D scene model. The twin 3D scene model containing the geometric object with the spatial trajectory curve is defined as the trajectory superimposed twin 3D scene model.

4. The method according to claim 2, wherein, The aerial panoramic image data, including the aerial panoramic image, is rendered and output in the panoramic image viewing window to support panoramic image rotation viewing, including: The aerial panoramic image data includes equidistant cylindrical projection decoding processing to obtain spherical texture mapping data; Create a panoramic spherical mesh model at the center of the panoramic view window, and attach the spherical texture mapping data to the inner surface of the spherical mesh model; In response to a detected mouse drag event, obtain the offset of the mouse movement; Based on the offset, determine the Euler angle of rotation of the panoramic spherical mesh model; Based on the Euler angles of rotation, update the rotation matrix of the panoramic spherical mesh model and re-render the panoramic spherical mesh model to support panoramic rotation viewing.

5. The method according to claim 1, wherein, On the preset linked page, as the playback timeline of the aerial video file progresses, aerial images from the corresponding aerial image sequence are played sequentially, including: Obtain the frame rate information of the aerial image sequence; Set the timer interval based on the frame rate information; Aerial images are read and displayed sequentially from the aerial image sequence, with the timer interval as the period.

6. The method according to claim 1, wherein, Each aerial image in the aerial image sequence has a corresponding playback time point, and each location data in the location dataset records the drone's spatial coordinates corresponding to the playback time point. The location data includes the playback time point and the drone's spatial coordinates. Furthermore, for each frame of the aerial image played, based on the time-synchronized location dataset, the spatial trajectory curve in the twin 3D scene model is continuously updated and dynamically positioned along the trajectory, including: When playing the first frame of the aerial image sequence, the playback time point corresponding to the aerial image is determined as the reference time point; The location data in the location dataset whose playback time point is the reference time point is determined as the reference location data; In the trajectory overlay twin 3D scene model, the position corresponding to the reference position data on the spatial trajectory curve is displayed as a pre-created positioning icon as a dynamic positioning icon. When playing the aerial images that are not the first frame in the sequence of aerial images one by one, the following steps are performed: The UAV spatial coordinates included in the reference position data are determined as the reference UAV spatial coordinates; Based on the reference UAV spatial coordinates, coordinates generated in the twin 3D scene model are used as the current spatial coordinates; The coordinates of the dynamic positioning icon in the twin 3D scene model are determined as historical spatial coordinates; The distance between the current spatial coordinates and the historical spatial coordinates is determined as the movement distance; In response to determining that the moving distance is less than a preset threshold, a dynamic positioning icon continues to be displayed at the location represented by the historical spatial coordinates; In response to determining that the moving distance is greater than or equal to the preset threshold, the dynamic positioning icon is moved and updated to the position represented by the current spatial coordinates on the spatial trajectory curve in the trajectory overlay twin 3D scene model.

7. A device for linking and displaying engineering aerial photography results with a 3D model, comprising: The loading unit is configured to load a pre-built twin 3D scene model; The acquisition unit is configured to acquire aerial video files, flight trajectory data, and time-synchronized location datasets for the target aerial photography sortie. A coordinate transformation processing unit is configured to perform coordinate transformation processing on the flight trajectory data to obtain trajectory data in the scene coordinate system. The rendering unit is configured to render a spatial trajectory curve corresponding to the trajectory data of the scene coordinate system in the twin 3D scene model, so as to obtain a trajectory superimposed twin 3D scene model. The first display unit is configured to display a trajectory-overlaid twin 3D scene model on a preset linked page; The second display unit is configured to, on the preset linkage page, sequentially play aerial images from the aerial image sequence corresponding to the aerial video file as the playback timeline of the aerial video file progresses. Specifically, for each frame of the aerial image played, the spatial trajectory curve in the twin 3D scene model is continuously updated and displayed based on the time-synchronized location dataset and superimposed along the trajectory.

8. An electronic device, comprising: One or more processors; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 6.

9. A computer-readable medium having a computer program stored thereon, wherein, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 6.

Images