Methods, apparatus, equipment, media, and procedures for integrating continuous panoramic images with BIM.

Abstract

The present application relates to a method, device, equipment, medium and program for the fusion of continuous panoramic images and BIM, the method comprising: obtaining a BIM model of a target building, and obtaining a panoramic image taken at a target shooting point in the target building, rendering a model view of the BIM model at the target shooting point through a BIM rendering engine, measuring the viewpoint distance from each pixel point in the model view to the target shooting point, pixel-aligning the panoramic image with the model view, and copying the viewpoint distance of each pixel point in the aligned model view to the aligned pixel point in the panoramic image, measuring the observation distance and deflection between the target shooting point and a preset observation point, deforming the panoramic image and the model view based on the viewpoint distance, observation distance and deflection of each pixel in the panoramic image, and obtaining an image of a preset observation point. The present invention realizes the fusion effect of continuously displaying panoramic photos and BIM models in space.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, device, equipment, storage medium and computer program for fusing continuous panoramic images with BIM. Background Art

[0002] Currently, the fusion technology of panoramic photos and BIM models has many applications in the fields of architecture, urban planning, real estate, etc. The fusion of panoramic photos and BIM models uses advanced digital technology to combine the wide-angle perspective and meticulous capture of panoramic photography with the precise geometry and attribute data of BIM models. This integration not only improves the efficiency of architectural design and construction management, but also realizes more intuitive and real-time project visualization and simulation through virtual reality and augmented reality technology, providing project teams with new decision-making support and execution tools.

[0003] Panoramic photos and BIM model data usually come from different sources and formats. Processing large amounts of panoramic photos and complex BIM model data requires a lot of computing resources and high-performance processors. Panoramic photos and BIM models require high accuracy, especially during the construction and maintenance phases. It is necessary to ensure that the model is as consistent as possible with the actual situation. This may require additional calibration and verification steps, which is a technical challenge for some projects and scenarios.

[0004] However, traditional fusion methods can only fuse the shooting points of panoramic photos with BIM models. Currently, there is a problem that the fusion effect can only be displayed at certain fixed positions, and the fusion effect of panoramic photos and BIM models cannot be continuously displayed in space. Summary of the invention

[0005] The present application provides a method, device, equipment, storage medium and computer program for fusing continuous panoramic images with BIM to solve the problem that the fusion effect can only be displayed at certain fixed positions and the fusion effect of panoramic photos and BIM models cannot be continuously displayed in space.

[0006] In a first aspect, the present application provides a method for fusing continuous panoramic images with BIM, comprising:

[0007] Obtaining a BIM model of a target building, and obtaining a panoramic image taken from a target shooting point in the target building;

[0008] Rendering a model view of the BIM model corresponding to the target shooting point by a BIM rendering engine;

[0009] Measuring the viewpoint distance from each pixel point in the model view to the target shooting point;

[0010] Pixel-aligning the panoramic image with the model view, and copying the viewpoint distance of each pixel point in the model view after pixel alignment to the aligned pixel points in the panoramic image;

[0011] Measuring the observation distance and deflection angle between the target shooting point and the preset observation point;

[0012] Based on the viewpoint distance of each pixel in the panoramic image and the observation distance and the deflection angle, the panoramic image and the model view are deformed to obtain the image of the preset observation point.

[0013] In some embodiments, pixel-aligning the panoramic image with the model view comprises:

[0014] Extracting characteristic pixel points of the panoramic image;

[0015] Selecting target feature pixels from the feature pixels one by one;

[0016] Constructing a basic matrix using the target feature pixels;

[0017] Calculate the epipolar lines of the target feature pixel points in the panoramic image in the model view based on the basic matrix;

[0018] Calculate the straight-line distance from the target feature pixel to the epipolar line;

[0019] The target feature pixel points whose straight-line distance is less than a preset distance threshold are mapped to the model view to obtain a pixel-aligned model view.

[0020] In some embodiments, the step of deforming the panoramic image and the model view to obtain the image of the preset observation point includes:

[0021] Determining whether the preset observation point is a target shooting point;

[0022] When the preset observation point is a target shooting point, determining a panoramic image and a model view corresponding to the shooting point position where the preset observation point is located as an image of the preset observation point;

[0023] When the preset observation point is not the target shooting point, the panoramic image and the model view are scaled, and the scaled panoramic image and model view are used as the image of the preset observation point.

[0024] In some embodiments, scaling the panoramic image and the model view includes:

[0025] Obtaining two-dimensional coordinates of pixels of the panoramic image;

[0026] Acquire the observation distance and deflection angle between the preset observation point and the target shooting point and the initial distance from the target shooting point to the pixel point of the panoramic image;

[0027] Converting the two-dimensional coordinates of the pixels of the panoramic image into spherical coordinates;

[0028] The update distance from the preset observation point to the pixel point of the panoramic image is calculated according to the following formula:

[0029]

[0030] in, is the update distance from the preset observation point to the pixel point of the panoramic image, is the square value of the distance between the preset observation point and the target shooting point, is the initial distance from the target shooting point to the pixel point of the panoramic image, is the cosine value of the deflection angle between the preset observation point and the target shooting point;

[0031] Calculate a deformation scale factor according to the initial distance and the updated distance;

[0032] Update and calculate the spherical coordinates of the pixel points of the panoramic image according to the deformation scale factor to obtain updated coordinates;

[0033] Performing a two-dimensional coordinate conversion on the updated coordinates, and redrawing the panoramic image and the model view using the converted two-dimensional coordinates;

[0034] The redrawn panoramic image and model view are pixel-filled using an interpolation algorithm.

[0035] In some embodiments, converting the two-dimensional coordinates of the pixels of the panoramic image into spherical coordinates includes:

[0036] The two-dimensional coordinates of the pixels of the panoramic image are converted into a preset spherical coordinate system using the following formula to obtain spherical coordinates:

[0037]

[0038]

[0039] in, is the length of the panoramic image, is the width of the panoramic image, is the distance from the two-dimensional coordinate position to the width of the panoramic image, is the distance from the two-dimensional coordinate position to the panoramic image length, is the celestial angle of the two-dimensional coordinate in the spherical coordinate system, is the azimuth of the two-dimensional coordinate in the spherical coordinate system, is the spherical coordinate system.

[0040] In some embodiments, the pixel supplementation of the redrawn panoramic image and the model view using an interpolation algorithm includes:

[0041] Determine a target pixel position in the panoramic image and the model view where the pixel density is less than a preset density threshold;

[0042] Obtain pixel information of all pixels within a preset radius area of ​​the target pixel position;

[0043] Taking the row where the target pixel position is located as a horizontal line, performing horizontal linear interpolation on the target pixel using pixel information on the horizontal line to obtain a horizontal linear interpolation result;

[0044] The horizontal linear interpolation result is subjected to vertical linear interpolation to obtain a pixel value of a target pixel, and the panoramic image and the model view are supplemented by the pixel value.

[0045] In a second aspect, the present application provides a continuous panoramic image and BIM fusion device, including: a data acquisition module, used to acquire a BIM model of a target building, and to acquire a panoramic image taken at a target shooting point in the target building; a model view rendering module, used to render a model view of the BIM model corresponding to the target shooting point through a BIM rendering engine; a viewpoint distance measurement module, used to measure the viewpoint distance from each pixel point in the model view to the target shooting point; a pixel alignment module, used to pixel-align the panoramic image with the model view, and copy the viewpoint distance of each pixel point in the model view after pixel alignment to the aligned pixel points in the panoramic image; an observation distance and deflection measurement module, used to measure the observation distance and deflection between the target shooting point and a preset observation point; an image deformation module, used to deform the panoramic image and the model view based on the viewpoint distance of each pixel in the panoramic image and the observation distance and deflection, to obtain an image of the preset observation point.

[0046] In a third aspect, the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described in the above aspects.

[0047] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the method described in the above aspects when executed by a processor.

[0048] In a fifth aspect, the present application provides a computer program product, including a computer program / instructions, which implements the steps of the method described in the above aspects when the computer program is executed by a processor.

[0049] The present application provides a method, device, equipment, storage medium and computer program for the fusion of continuous panoramic images and BIM, which obtains the BIM model of a target building and the panoramic image taken at a target shooting point in the target building, renders the model view of the BIM model corresponding to the target shooting point through a BIM rendering engine, measures the viewpoint distance from each pixel point in the model view to the target shooting point, performs pixel alignment on the panoramic image and the model view, and copies the viewpoint distance of each pixel point in the model view after pixel alignment to the aligned pixel points in the panoramic image, measures the observation distance and deflection between the target shooting point and a preset observation point, and performs deformation processing on the panoramic image and the model view based on the viewpoint distance of each pixel in the panoramic image and the observation distance and deflection to obtain the image of the preset observation point. This solves the problem that the fusion effect can only be displayed at certain fixed positions and the fusion effect of the panoramic photo and the BIM model cannot be continuously displayed in space, provides a more intuitive and interactive experience, and can optimize data display and improve user participation.

[0050] 1. Pixel supplementation through interpolation algorithm can achieve a smoothing effect, making the panoramic image rendering deformation process more natural and continuous, and improving the image resolution and clarity.

[0051] 2. The spatial coordinate conversion method can quickly convert the pixel points in the panoramic image. When the preset observation point is not the target shooting point of the panoramic image, the panoramic image is deformed to improve the efficiency of the panoramic image deformation. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The present application will be described in more detail below based on embodiments and with reference to the accompanying drawings:

[0053] Figure 1 A schematic diagram of a flow chart of a method for fusing continuous panoramic images with BIM provided in an embodiment of the present application;

[0054] Figure 2 A schematic diagram of a model view generation process of a continuous panoramic image and BIM fusion provided in an embodiment of the present application;

[0055] Figure 3 A three-dimensional schematic diagram of the fusion of a continuous panoramic image and BIM provided in an embodiment of the present application;

[0056] Figure 4A schematic diagram of functional modules of a continuous panoramic image and BIM fusion device provided in an embodiment of the present application;

[0057] Figure 5 A schematic diagram of the structure of an electronic device for fusing continuous panoramic images with BIM provided in an embodiment of the present application.

[0058] In the drawings, the same reference numerals are used for the same components, and the drawings are not drawn to scale. DETAILED DESCRIPTION

[0059] In order to enable those skilled in the art to better understand the technical solution of the present application, and to fully understand and implement how the present application applies technical means to solve technical problems and achieve the corresponding technical effects, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only embodiments of a part of the present application, not all embodiments. The embodiments of the present application and the various features in the embodiments can be combined with each other without conflict, and the technical solutions formed are all within the scope of protection of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work should belong to the scope of protection of the present application.

[0060] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0061] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0062] Based on the above background, this application proposes a continuous panoramic image and BIM fusion solution, which can adjust panoramic photos using interpolation algorithms and spatial coordinate transformations, and display them to users in real time, providing users with a more intuitive and interactive experience, and can optimize data display and enhance user participation.

[0063] Embodiment 1

[0064] Figure 1 The following is a flow chart of a method for integrating continuous panoramic images with BIM provided in an embodiment of the present application. Figure 1 As shown, a method for fusing continuous panoramic images with BIM includes:

[0065] S1. Obtain a BIM model of a target building, and obtain a panoramic image taken from a target shooting point in the target building.

[0066] In the embodiment of the present invention, the target building may include a residential building, a commercial building, an industrial building, a public building, an infrastructure building, etc.

[0067] In detail, the BIM model is a panoramic building information model, which is a method of digital building design and management, presenting the building project in a digital way, including the building's geometric shape, spatial relationship, materials, components and related attribute information. Through the BIM model, the building design team, construction team, owner and other stakeholders can work together on a unified platform and update and view the project status at any time.

[0068] Specifically, the BIM model of the target building can be provided by the designer or architect, obtained from a contractor or construction team, or downloaded and used from an open BIM model library.

[0069] In the embodiment of the present invention, the panoramic image is also called a panoramic map, which can fully display the 360° field of view of the shooting point. The observer can view different parts of the panoramic image by adjusting the viewing angle (determined by the position, direction and focal length of the observation camera).

[0070] In the embodiment of the present invention, a panoramic camera and panoramic shooting software are combined to obtain a panoramic image at a target shooting point in the target building, and a panoramic image corresponding to the target shooting point position in the target building is obtained to ensure that the entire area inside the target building is covered.

[0071] Figure 2 A schematic diagram of a model view generation process for integrating continuous panoramic images with BIM provided in an embodiment of the present application. Figure 2 The left picture refers to the shooting point location selected in the actual scene. Figure 2 The picture on the right refers to mapping the shooting point position of a specific scene into the BIM model to form a model view.

[0072] S2. Rendering a model view of the BIM model corresponding to the target shooting point through a BIM rendering engine.

[0073] In an embodiment of the present invention, rendering the model view of the BIM model corresponding to the target shooting point by a BIM rendering engine includes:

[0074] Using the BIM rendering engine to simulate the viewing angle of a camera according to the target shooting point, and rendering a model view of the BIM model corresponding to the target shooting point from the viewing angle to obtain a rendered image;

[0075] Pixel supplementation is performed on the rendered image to obtain a model view of the BIM model.

[0076] In detail, the BIM rendering engine supports multi-threaded rendering and GPU acceleration to achieve high frame rate rendering performance.

[0077] In an embodiment of the present invention, rendering the BIM model corresponding to the target shooting point from the perspective refers to setting the required parameters in the BIM rendering engine, such as image resolution, rendering quality, lighting settings, etc.; starting the rendering engine in the BIM software to start rendering the BIM model corresponding to the target shooting point, and the BIM rendering engine uses multi-threading technology to simultaneously process different parts of the BIM model corresponding to the target shooting point; through GPU acceleration, part of the calculation in rendering the BIM model corresponding to the target shooting point can be transferred to the graphics processor for acceleration, and a rendered image is obtained after the accelerated rendering is completed.

[0078] In an embodiment of the present invention, the method for supplementing pixels of the rendered image is to supplement pixels through an interpolation algorithm. This is because during the coordinate transformation process, due to the adjustment of the deformation scale factor during the image deformation process, some areas in the panoramic image may become blank or have insufficient pixel density.

[0079] In detail, bilinear interpolation is used to supplement the pixels of the rendered image. Bilinear interpolation is faster than bicubic interpolation in calculation speed, and the algorithm is simple, easy to implement and understand. Supplementing pixels through the interpolation algorithm can achieve a smoothing effect, making the panoramic image rendering deformation process more natural and continuous, and improving the resolution and clarity of the image.

[0080] Rendering the model view of the BIM model corresponding to the target shooting point through the BIM rendering engine can bring about perspective consistency, improved visualization effect and convenient interactive analysis, thereby improving the efficiency and effect of the computer in the model view generation process.

[0081] S3. Measuring the viewpoint distance from each pixel point in the model view to the target shooting point.

[0082] In an embodiment of the present invention, a spherical coordinate system is established with the shooting point as the origin, and the BIM engine is used to simulate the viewing angle of the camera, so that the model view is positioned and captured in three-dimensional space, and the distance between the simulated camera and each pixel point in the model view is obtained, that is, the viewpoint distance from each pixel point in the model view to the target shooting point is obtained.

[0083] Measuring the viewpoint distance from each pixel in the model view to the target shooting point helps determine the depth and perspective relationship of the viewing angle. By accurately measuring the viewpoint distance, image rendering and projection can be optimized, and the realism and three-dimensionality of the visual effect can be improved. At the same time, the accurate position and size of objects in the panoramic image can be ensured, making the objects in the panoramic image more realistic and credible in the BIM model.

[0084] S4. Pixel-align the panoramic image with the model view, and copy the viewpoint distance of each pixel point in the model view after pixel alignment to the aligned pixel points in the panoramic image.

[0085] In the embodiment of the present invention, the pixel-aligning the panoramic image with the model view includes:

[0086] Extracting characteristic pixel points of the panoramic image;

[0087] Selecting target feature pixels from the feature pixels one by one;

[0088] Constructing a basic matrix using the target feature pixels;

[0089] Calculate the epipolar lines of the target feature pixel points in the panoramic image in the model view based on the basic matrix;

[0090] Calculate the straight-line distance from the target feature pixel to the epipolar line;

[0091] The target feature pixel points whose straight-line distance is less than a preset distance threshold are mapped to the model view to obtain a pixel-aligned model view.

[0092] In an embodiment of the present invention, a basic matrix describes the geometric relationship between a panoramic image and a model view, and can be used to map target feature pixel points from the panoramic image to the epipolar lines in the model view; the epipolar lines are calculated using the basic matrix and the coordinates of the feature points in the panoramic image; a preset distance threshold is set to filter out target feature pixel points whose straight-line distance is less than the preset distance threshold, thereby ensuring that only pixel points near the epipolar lines are retained, thereby improving the accuracy of pixel alignment.

[0093] Fundamental Matrix Verification is a computer vision technology that improves the accuracy and reliability of feature pixel matching through geometric consistency verification. It can ensure the accurate geometric relationship between images or scenes, such as the polar line constraints of corresponding points, thereby improving the accuracy of pixel alignment between panoramic images and model views.

[0094] In an embodiment of the present invention, after verification by the basic matrix, the target feature pixel points whose straight-line distance is less than a preset distance threshold are mapped to the model view to obtain a pixel-aligned model view, and the viewpoint distance of each pixel point in the model view is synchronized to the corresponding pixel point in the panoramic image, so that each pixel point in the panoramic image has a corresponding viewpoint distance.

[0095] The panoramic image is pixel-aligned with the model view, and the viewpoint distance of each pixel in the pixel-aligned model view is copied to the aligned pixel in the panoramic image, ensuring that the spatial position and visual depth of the model view in the panoramic image feel real and accurate, thereby enhancing the interactivity and reality of the user with the virtual content.

[0096] S5. Measure the observation distance and deflection angle between the target shooting point and the preset observation point.

[0097] In the embodiment of the present invention, in the rendered image, the target shooting point is used as the origin of the coordinate system, and the distance and deflection angle between the target shooting point and any selected observation point are measured using the BIM engine.

[0098] Measuring the observation distance and deflection angle between the target shooting point and the preset observation point not only helps to improve the efficiency of panoramic photo post-rendering, but also ensures the integrity of panoramic photo post-rendering, optimizes the visual effect of panoramic photo post-rendering, and enhances the audience's immersion and visual experience.

[0099] S6. Based on the viewpoint distance of each pixel in the panoramic image, the observation distance and the deflection angle, the panoramic image and the model view are deformed to obtain an image of the preset observation point.

[0100] In an embodiment of the present invention, since the viewpoint distances from the preset observation point to each pixel in the panoramic image and the model view are different, the panoramic image and the model view are deformed based on the viewpoint distance of each pixel in the panoramic image and the observation distance and the deflection angle.

[0101] In the embodiment of the present invention, the step of performing deformation processing on the panoramic image and the model view to obtain the image of the preset observation point includes:

[0102] Determining whether the preset observation point is a target shooting point;

[0103] When the preset observation point is a target shooting point, determining a panoramic image and a model view corresponding to the shooting point position where the preset observation point is located as an image of the preset observation point;

[0104] When the preset observation point is not the target shooting point, the panoramic image and the model view are scaled, and the scaled panoramic image and model view are used as the image of the preset observation point.

[0105] In detail, the image scaling of the panoramic image and the model view includes:

[0106] Obtaining two-dimensional coordinates of pixels of the panoramic image;

[0107] Acquire the observation distance and deflection angle between the preset observation point and the target shooting point and the initial distance from the target shooting point to the pixel point of the panoramic image;

[0108] Converting the two-dimensional coordinates of the pixels of the panoramic image into spherical coordinates;

[0109] The update distance from the preset observation point to the pixel point of the panoramic image is calculated according to the following formula:

[0110]

[0111] in, is the update distance from the preset observation point to the pixel point of the panoramic image, is the square value of the distance between the preset observation point and the target shooting point, is the initial distance from the target shooting point to the pixel point of the panoramic image, is the cosine value of the deflection angle between the preset observation point and the target shooting point;

[0112] Calculate a deformation scale factor according to the initial distance and the updated distance;

[0113] Update and calculate the spherical coordinates of the pixel points of the panoramic image according to the deformation scale factor to obtain updated coordinates;

[0114] Performing a two-dimensional coordinate conversion on the updated coordinates, and redrawing the panoramic image and the model view using the converted two-dimensional coordinates;

[0115] The redrawn panoramic image and model view are pixel-filled using an interpolation algorithm.

[0116] In detail, the converting the two-dimensional coordinates of the pixel points of the panoramic image into spherical coordinates includes:

[0117] The two-dimensional coordinates of the pixels of the panoramic image are converted into a preset spherical coordinate system using the following formula to obtain spherical coordinates:

[0118]

[0119]

[0120] in, is the length of the panoramic image, is the width of the panoramic image, is the distance from the two-dimensional coordinate position to the width of the panoramic image, is the distance from the two-dimensional coordinate position to the panoramic image length, is the celestial angle of the two-dimensional coordinate in the spherical coordinate system, is the azimuth of the two-dimensional coordinate in the spherical coordinate system, is the spherical coordinate system.

[0121] In detail, it is assumed that the two-dimensional coordinates of the pixels of the panoramic image are , the coordinates of the target shooting point are , let the coordinates of the preset observation point be , the observation distance between the preset observation point and the target shooting point is , the deflection angle is .

[0122] The two-dimensional coordinates of the pixels of the panoramic image Convert to spherical coordinates , the initial distance from the target shooting point to the pixel point of the panoramic image , let the update distance from the preset observation point to the pixel point of the panoramic image be , according to the law of cosines:

[0123]

[0124] in, is the update distance from the preset observation point to the pixel point of the panoramic image, is the square value of the distance between the preset observation point and the target shooting point, is the initial distance from the target shooting point to the pixel point of the panoramic image, It is the cosine value of the deviation angle between the preset observation point and the target shooting point.

[0125] According to the updated distance from the preset observation point to the pixel point of the panoramic image and the initial distance from the target shooting point to the pixel point of the panoramic image , determine the deformation scaling factor :

[0126]

[0127] in, is the update distance from the preset observation point to the pixel point of the panoramic image, is the initial distance from the target shooting point to the pixel point of the panoramic image.

[0128] For each pixel, calculate the updated coordinates of the pixel in the panoramic image ,in and According to the deformation scale factor Make adjustments:

[0129]

[0130]

[0131] in, is the horizontal coordinate of the spherical coordinate corresponding to the two-dimensional coordinate of the pixel point of the panoramic image, is the ordinate of the spherical coordinate corresponding to the two-dimensional coordinate of the pixel point of the panoramic image, is the deformation scale factor.

[0132] The coordinates will be updated Convert to update plane coordinates , the coordinate conversion is completed, and the panoramic image is redrawn using the updated plane coordinates.

[0133] Due to the adjustment of the deformation scale factor, blank pixel areas or low pixel density in local areas may appear in the deformed view. Therefore, in order to achieve a more complete perspective picture display, an interpolation algorithm may be used to supplement pixels of the panoramic image and the model view.

[0134] In the embodiment of the present invention, the pixel supplementation of the redrawn panoramic image and the model view by using an interpolation algorithm includes:

[0135] Determine a target pixel position in the panoramic image and the model view where the pixel density is less than a preset density threshold;

[0136] Obtain pixel information of all pixels within a preset radius area of ​​the target pixel position;

[0137] Taking the row where the target pixel position is located as a horizontal line, performing horizontal linear interpolation on the target pixel using pixel information on the horizontal line to obtain a horizontal linear interpolation result;

[0138] The horizontal linear interpolation result is subjected to vertical linear interpolation to obtain a pixel value of a target pixel, and the panoramic image and the model view are supplemented by the pixel value.

[0139] The panoramic image and the model view are deformed using an interpolation algorithm to obtain the image of the preset observation point, and the panoramic image is displayed to the user in real time, providing the user with a more intuitive and interactive experience, and being able to optimize data display and enhance user participation.

[0140] In detail, assuming that the preset observation point is not the target shooting point of the panoramic image, the picture needs to be deformed, that is, the scaling ratio of objects with a short viewing distance is larger than that of objects with a long viewing distance, so as to simulate the effect of roaming in the panorama.

[0141] The spatial coordinate conversion method can quickly realize the conversion of pixel points in the panoramic image. When the preset observation point is not the target shooting point of the panoramic image, the panoramic image is deformed to improve the efficiency of the panoramic image deformation.

[0142] Figure 3 A three-dimensional schematic diagram of the fusion of continuous panoramic images and BIM provided in an embodiment of the present application. Figure 3 It refers to integrating the established and trained BIM model with panoramic photos to feel the real building environment in the virtual space.

[0143] Based on the interactive user interface platform, users can use a variety of ways to navigate, including mouse dragging, keyboard operation or touch screen, so as to smoothly browse panoramic images and BIM models. Users can deeply understand the details of space and model through basic operations such as rotation, zooming, panning, etc. In addition, it also supports advanced functions, such as hotspot navigation, which allows users to intuitively jump to specific areas or key points; it can also control the displayed layers to better understand the composition and design of the building model.

[0144] The virtual "roaming" mode provides users with a more immersive experience. In this mode, users can walk freely in the building space, and the system will dynamically load and deform the corresponding panoramic images and BIM models according to the user's position, so that users can feel the real building environment in the virtual space. In this way, users can have a deeper understanding of the building design and have a more intuitive understanding of its details.

[0145] In the embodiment of the present invention, the "roaming" mode is to support the user's continuous roaming in three-dimensional space by developing an algorithm. When the user moves close to the next shooting point, the system smoothly transitions to the panoramic photo and BIM rendering result of the point. During the transition process, the bilinear interpolation (BLERP) technology is used to achieve a seamless visual experience.

[0146] In detail, during the bilinear interpolation (BLERP) technology processing process, the newly created pixel value in the target image is calculated by weighted averaging the values ​​of 4 neighboring pixels in a 2*2 area near the source image. The enlarged image has high quality and no discontinuous pixel values.

[0147] Embodiment 2

[0148] like Figure 4 , which is a functional module diagram of a continuous panoramic image and BIM fusion device 100 provided for this embodiment.

[0149] The continuous panoramic image and BIM fusion device 100 of the present invention can be installed in an electronic device. According to the functions to be implemented, the continuous panoramic image and BIM fusion device 100 can include a data acquisition module 101, a model view rendering module 102, a viewpoint distance measurement module 103, a pixel alignment module 104, an observation distance and deflection angle measurement module 105, and an image deformation module 106. The module of the present invention can also be called a unit, which refers to a series of computer program segments that can be executed by an electronic device processor and can complete fixed functions, which are stored in the memory of the electronic device.

[0150] In this embodiment, the functions of each module / unit are as follows:

[0151] The data acquisition module 101 is used to acquire a BIM model of a target building and to acquire a panoramic image taken from a target shooting point in the target building;

[0152] A model view rendering module 102, configured to render a model view of the BIM model corresponding to the target shooting point through a BIM rendering engine;

[0153] A viewpoint distance measurement module 103 is used to measure the viewpoint distance from each pixel point in the model view to the target shooting point;

[0154] A pixel alignment module 104 is used to perform pixel alignment between the panoramic image and the model view, and copy the viewpoint distance of each pixel point in the model view after pixel alignment to the aligned pixel points in the panoramic image;

[0155] The observation distance and deflection angle measurement module 105 is used to measure the observation distance and deflection angle between the target shooting point and the preset observation point;

[0156] The image deformation module 106 is used to perform deformation processing on the panoramic image and the model view based on the viewpoint distance of each pixel in the panoramic image and the observation distance and the deflection angle to obtain the image of the preset observation point.

[0157] Embodiment 3

[0158] Figure 5 A schematic diagram of the structure of an electronic device for fusing continuous panoramic images with BIM provided in an embodiment of the present application.

[0159] On the basis of the above embodiments, this embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described in the above embodiments.

[0160] In some implementations of this embodiment, a computer-readable storage medium is provided, on which a computer program is stored, characterized in that when the computer program is executed by a processor, the steps of the method described in the above embodiment are implemented.

[0161] In some implementations of this embodiment, a computer program product is provided, including a computer program / instructions, characterized in that when the computer program is executed by a processor, the steps of the method described in the above embodiment are implemented.

[0162] The processor may include, but is not limited to, one or more processors or microprocessors, etc. Each processor may be an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a controller, a microcontroller, a microprocessor or other electronic components to execute the methods in the above embodiments.

[0163] The computer-readable storage medium may be implemented by any type of volatile or non-volatile storage device or a combination thereof, and the computer-readable storage medium may include but is not limited to, for example, random access memory (RAM), read-only memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, computer storage media (e.g., hard disk, floppy disk, solid-state drive, removable disk, CD ROM, DVD ROM, Blu-ray disc, etc.).

[0164] The computer-readable storage medium may also store at least one computer executable program / instruction, which may be, for example, a computer-readable instruction. The computer-readable storage medium includes, but is not limited to, for example, a volatile memory and / or a non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The computer-readable storage medium may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. For example, a non-transitory computer-readable storage medium may be connected to a computing device such as a computer, and then, when the computing device runs the computer-readable instructions stored on the computer-readable storage medium, the various methods described above may be performed.

[0165] In addition, the computer device may also include (but not limited to) a data bus, an input / output (I / O) bus, a display, and input / output devices (eg, keyboard, mouse, speaker, etc.), etc.

[0166] The processor may communicate with a communication interface of an external device via an I / O bus via a wired or wireless network.

[0167] In one embodiment, the at least one computer executable instruction may also be compiled into or constitute a software product / computer program product, wherein one or more computer executable instructions are executed by a processor to perform the various functions and / or method steps in the embodiments described in the present technology.

[0168] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the above-mentioned module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based device that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0169] It should be noted that, in this application, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element limited by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0170] Although the implementation methods disclosed in this application are as above, the above contents are only the implementation methods adopted for facilitating the understanding of this application, and are not intended to limit this application. Any technician in the technical field to which this application belongs can make any modifications and changes in the form and details of the implementation without departing from the spirit and scope disclosed in this application, but the scope of patent protection of this application shall still be based on the scope defined in the attached claims.

Claims

1. A method for fusing continuous panoramic images with BIM, characterized in that: include: Obtaining a BIM model of a target building, and obtaining a panoramic image taken from a target shooting point in the target building; Rendering a model view of the BIM model corresponding to the target shooting point by a BIM rendering engine; Measuring the viewpoint distance from each pixel point in the model view to the target shooting point; Pixel-aligning the panoramic image with the model view, and synchronizing the viewpoint distance of each pixel point in the pixel-aligned model view to the corresponding aligned pixel points in the panoramic image; Measuring the observation distance and deflection angle between the target shooting point and the preset observation point; Based on the viewpoint distance of each pixel in the panoramic image and the observation distance and the deflection angle, the panoramic image and the model view are deformed to obtain the image of the preset observation point; Specifically include: Determining whether the preset observation point is a target shooting point; When the preset observation point is a target shooting point, determining a panoramic image and a model view corresponding to the shooting point position where the preset observation point is located as an image of the preset observation point; When the preset observation point is not the target shooting point, the panoramic image and the model view are scaled, and the scaled panoramic image and the model view are used as the image of the preset observation point; scaling the panoramic image and the model view includes: Obtaining two-dimensional coordinates of pixels of the panoramic image; Acquire the observation distance and deflection angle between the preset observation point and the target shooting point and the initial distance from the target shooting point to the pixel point of the panoramic image; Calculating an updated distance from a preset observation point to a pixel point of the panoramic image according to an observation distance between the preset observation point and the target shooting point and an initial distance from the target shooting point to a pixel point of the panoramic image; Converting the two-dimensional coordinates of the pixels of the panoramic image into spherical coordinates; Calculate a deformation scale factor according to the initial distance and the updated distance; Update and calculate the spherical coordinates of the pixel points of the panoramic image according to the deformation scale factor to obtain updated coordinates; Performing a two-dimensional coordinate conversion on the updated coordinates, and redrawing the panoramic image and the model view using the converted two-dimensional coordinates; The redrawn panoramic image and model view are pixel-filled using an interpolation algorithm.

2. The method according to claim 1, characterized in that: The pixel-aligning the panoramic image with the model view comprises: Extracting characteristic pixel points of the panoramic image; Selecting target feature pixels from the feature pixels one by one; Constructing a basic matrix using the target feature pixels; Calculate the epipolar lines of the target feature pixel points in the panoramic image in the model view based on the basic matrix; Calculate the straight-line distance from the target feature pixel to the epipolar line; The target feature pixel points whose straight-line distance is less than a preset distance threshold are mapped to the model view to obtain a pixel-aligned model view.

3. The method according to claim 1, characterized in that The calculating the updated distance from the preset observation point to the pixel point of the panoramic image according to the observation distance between the preset observation point and the target shooting point and the initial distance from the target shooting point to the pixel point of the panoramic image comprises: The update distance from the preset observation point to the pixel point of the panoramic image is calculated according to the following formula: in, is the update distance from the preset observation point to the pixel point of the panoramic image, is the square value of the distance between the preset observation point and the target shooting point, is the initial distance from the target shooting point to the pixel point of the panoramic image, It is the cosine value of the deviation angle between the preset observation point and the target shooting point.

4. The method according to claim 3, characterized in that: The converting the two-dimensional coordinates of the pixels of the panoramic image into spherical coordinates comprises: The two-dimensional coordinates of the pixels of the panoramic image are converted into a preset spherical coordinate system using the following formula to obtain spherical coordinates: in, is the length of the panoramic image, is the width of the panoramic image, is the distance from the two-dimensional coordinate position to the width of the panoramic image, is the distance from the two-dimensional coordinate position to the panoramic image length, is the celestial angle of the two-dimensional coordinate in the spherical coordinate system, is the azimuth of the two-dimensional coordinate in the spherical coordinate system, is the spherical coordinate system.

5. The method according to claim 3, characterized in that: The method of supplementing pixels of the redrawn panoramic image and the model view by using an interpolation algorithm includes: Determine a target pixel position in the panoramic image and the model view where the pixel density is less than a preset density threshold; Obtain pixel information of all pixels within a preset radius area of ​​the target pixel position; Taking the row where the target pixel position is located as a horizontal line, performing horizontal linear interpolation on the target pixel using pixel information on the horizontal line to obtain a horizontal linear interpolation result; The horizontal linear interpolation result is subjected to vertical linear interpolation to obtain a pixel value of a target pixel, and the panoramic image and the model view are supplemented by the pixel value.

6. A continuous panoramic image and BIM fusion device, characterized in that: The device comprises: A data acquisition module is used to acquire a BIM model of a target building and to acquire a panoramic image taken from a target shooting point in the target building; A model view rendering module, used to render a model view of the BIM model corresponding to the target shooting point through a BIM rendering engine; A viewpoint distance measurement module, used to measure the viewpoint distance from each pixel point in the model view to the target shooting point; A pixel alignment module, used to perform pixel alignment on the panoramic image and the model view, and synchronize the viewpoint distance of each pixel point in the model view after pixel alignment to the aligned pixel points in the panoramic image; An observation distance and deflection angle measurement module is used to measure the observation distance and deflection angle between the target shooting point and a preset observation point; An image deformation module is used to deform the panoramic image and the model view based on the viewpoint distance of each pixel in the panoramic image and the observation distance and deflection angle to obtain the image of the preset observation point; specifically comprising: Determining whether the preset observation point is a target shooting point; When the preset observation point is a target shooting point, determining a panoramic image and a model view corresponding to the shooting point position where the preset observation point is located as an image of the preset observation point; When the preset observation point is not the target shooting point, the panoramic image and the model view are scaled, and the scaled panoramic image and the model view are used as the image of the preset observation point; scaling the panoramic image and the model view includes: Obtaining two-dimensional coordinates of pixels of the panoramic image; Acquire the observation distance and deflection angle between the preset observation point and the target shooting point and the initial distance from the target shooting point to the pixel point of the panoramic image; Calculating an updated distance from a preset observation point to a pixel point of the panoramic image according to an observation distance between the preset observation point and the target shooting point and an initial distance from the target shooting point to a pixel point of the panoramic image; Converting the two-dimensional coordinates of the pixels of the panoramic image into spherical coordinates; Calculate a deformation scale factor according to the initial distance and the updated distance; Update and calculate the spherical coordinates of the pixel points of the panoramic image according to the deformation scale factor to obtain updated coordinates; Performing a two-dimensional coordinate conversion on the updated coordinates, and redrawing the panoramic image and the model view using the converted two-dimensional coordinates; The redrawn panoramic image and model view are pixel-filled using an interpolation algorithm.

7. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images