Method and apparatus for finding a window for a specific landscape

Abstract

The present application relates to a method and device for finding windows for a specific landscape. The method includes: obtaining building information and a multi-view image set in a target area, and constructing a three-dimensional model of the target area based on the building information and the image set; determining a window data set by analyzing frame images in the image set; determining a viewing frustum of a window according to a preset field of view angle and an outward normal vector in geometric space representation data, and determining visible area data of the window generated by projecting the viewing frustum onto the three-dimensional model; constructing a visible area database of all windows in the target area according to the building information, geometric space representation data of each window, and the visible area data; and finding windows that match the multi-modal query information in the visible area database according to the multi-modal query information uploaded by the user. The present application improves the efficiency of finding windows with a specific outdoor landscape.

Description

Technical Field

[0001] The present application relates to the technical field of computer systems, and in particular, to a method and device for finding windows for a specific landscape. Background Art

[0002] With the rapid advancement of the urbanization process and the continuous improvement of people's living quality, in the fields related to architecture, whether it is in scenarios such as residence, commerce, tourism, or urban planning, people's requirements for the spatial environment are becoming increasingly refined and diverse. In the scenarios of residence and hotel accommodation, residents and tourists not only focus on the facilities and layouts inside the building but also expect the space they are in to have a high-quality visual experience so as to enjoy the desired landscape or a good natural environment through the window. For example, hotel guests expect to book rooms with a direct view of a specific landmark landscape, homebuyers pay attention to visual indicators such as the greening rate outside the house window, and urban planners also need to evaluate the sky view and green coverage outside the windows of the block.

[0003] However, in the prior art, various systems such as hotels, real estate, and urban landscapes are independent, and it is difficult for ordinary users to search for the required outdoor landscape in multiple independent systems. For example, if a guest wants to book a hotel room with a view of a specific landmark, they need to first determine the hotel by themselves and then determine the location of the room through the hotel staff. In this way, the efficiency of users searching for windows that meet the outdoor landscape is very low. Another example is that it will be even more difficult for a user to rent a house where the green view rate and water view rate outside the window both reach 40%. Summary of the Invention

[0004] The present application provides a method and device for finding windows for a specific landscape to solve the problem of low efficiency in finding windows with a specific outdoor landscape in the prior art.

[0005] In a first aspect, the present application provides a method for finding windows for a specific landscape, the method including:

[0006] Obtaining building information and a multi-perspective image set in a target area, and constructing a three-dimensional model of the target area according to the building information and the image set;

[0007] Determining a window data set by analyzing the frame images in the image set, where the window data set includes geometric spatial characterization data of each window in the target area;

[0008] Determining a viewing frustum of the window according to a preset field of view angle and an outward normal vector in the geometric spatial characterization data, and determining visible area data of the window generated by projecting the viewing frustum onto the three-dimensional model, where the visible area data is used to indicate the outdoor landscape;

[0009] Construct a visibility database for all windows in the target area according to the building information, the geometric space characterization data of each window, and the visibility data.

[0010] Search for windows that match the multimodal query information in the visibility database according to the multimodal query information uploaded by the user, where the multimodal query information is used to query windows with specific outdoor landscapes.

[0011] Optionally, by analyzing the frame images in the image set, it is determined that the window data set includes:

[0012] By back-projecting the two-dimensional window frame corner points in the frame image to the three-dimensional model, a three-dimensional window frame contour under the image path is obtained, where the three-dimensional window frame contour is used to indicate the shape contour of the window in three-dimensional space.

[0013] Determine the window point clusters under the point cloud path according to the dense point cloud of the target area, and determine the geometric space characterization data of the window according to the window point clusters, where the window point clusters are used to indicate the point cloud data of multiple windows.

[0014] Determine the overlap degree of the three-dimensional window frame contour under the image path and the window point clusters under the point cloud path on the two-dimensional plane.

[0015] If the overlap degree is greater than the preset overlap degree threshold, determine that the three-dimensional window frame contour and the window point clusters are the same window, and use the geometric space characterization data under the point cloud path as the geometric space characterization data of the window.

[0016] Construct the window data set according to the geometric space characterization data of multiple windows.

[0017] Optionally, by back-projecting the two-dimensional window frame corner points in the frame image to the three-dimensional model, obtaining the three-dimensional window frame contour under the image path includes:

[0018] Perform semantic segmentation on the frame image through a semantic segmentation model to determine the initial window frame mask in the frame image.

[0019] Determine the final window frame mask of the window by tracking the same window in multiple consecutive frame images.

[0020] Extract the two-dimensional window frame corner points in the final window frame mask.

[0021] Back-project the two-dimensional window frame corner points to the three-dimensional model according to the camera parameters corresponding to the frame image to obtain the three-dimensional window frame contour of the window under the image path, where the camera parameters include shooting parameters and camera pose parameters.

[0022] Optionally, determining the geometric space representation data of the window based on the window point clusters includes:

[0023] Repeat the following operations: Fit an initial plane based on a randomly selected part of the points from the window point clusters, and determine the distance from each point in the window point clusters to the initial plane; Determine the number of points with a distance less than a preset distance threshold, where the initial plane is used to indicate a wall surface with multiple windows.

[0024] After reaching the set number of repeated operations, use the initial plane with the largest number of points as the target plane.

[0025] Determine the geometric space representation data of each window according to the distribution characteristics of the window point clusters in the target plane.

[0026] Optionally, determining the overlap degree between the three-dimensional window frame contour under the image path and the window point clusters under the point cloud path in a two-dimensional plane includes:

[0027] Project the three-dimensional window frame contour under the image path onto a two-dimensional plane to obtain a polygon contour.

[0028] Project the window point clusters under the point cloud path onto the two-dimensional plane to obtain a projected point cluster.

[0029] Select the internal point cluster that falls within the polygon contour from the projected point cluster.

[0030] Calculate the overlap degree according to the area of the region formed by the internal point cluster and the area of the region formed by the polygon contour.

[0031] Optionally, constructing a visible field database for all windows in the target area according to the building information, the geometric space representation data of each window, and the visible field data includes:

[0032] Determine the frustum image of the visible field data, and construct a frustum database according to the frustum images of multiple windows.

[0033] Generate a visual vector according to the frustum image of the window, and construct a vector database according to the visual vectors of multiple windows.

[0034] Determine the building floor data where the window is located according to the preset building information, determine the visible environment index according to the visible field data, and construct an attribute database by combining the building floor data where multiple windows are located, the geometric space representation data, and the visible environment index.

[0035] Construct a knowledge graph database, where the knowledge graph database uses entities, buildings, blocks, and windows as nodes, and uses the visible entities, the buildings where the windows are located, and the blocks where the windows are located as edges.

[0036] Among them, the cone database, the vector database, the attribute database, and the knowledge graph database all include window identifiers.

[0037] Optionally, searching for windows that match the multimodal query information in the visual field database according to the multimodal query information uploaded by the user includes:

[0038] Obtain the text to be recognized and the image to be recognized uploaded by the user, where the text to be recognized and the image to be recognized contain the outdoor scenery that the user expects to see;

[0039] Determine the image vector of the image to be recognized, and search for a first set of window identifiers that match the image vector in the vector database. If the image vector contains a landmark entity, add a landmark keyword to the text to be recognized;

[0040] Search for a second set of window identifiers that meet the text to be recognized in the first set of window identifiers in the attribute database;

[0041] Search for target nodes and target edges that meet the second set of window identifiers in the knowledge graph database, and construct a third set of window identifiers based on the target nodes and the target edges;

[0042] Determine the cone image to be recognized of the image to be recognized, and search for a final set of window identifiers that match the cone image to be recognized in the third set of window identifiers in the cone database;

[0043] Determine all the window identifiers in the final set of window identifiers.

[0044] In a second aspect, the present application provides a device for finding windows for a specific landscape, and the device includes:

[0045] An acquisition and construction module, configured to acquire building information and a multi-perspective image set in a target area, and construct a three-dimensional model of the target area according to the building information and the image set;

[0046] A first determination module, configured to determine a window data set by analyzing frame images in the image set, where the window data set includes geometric space representation data of each window in the target area;

[0047] A second determination module, configured to determine a viewing frustum of the window according to a preset field of view angle and an outward normal vector in the geometric space representation data, and determine visual field data of the window generated by projecting the viewing frustum onto the three-dimensional model, where the visual field data is used to indicate the outdoor scenery;

[0048] A construction module for constructing a visible field database of all windows in the target area according to the building information, the geometric space representation data of each window, and the visible field data;

[0049] A search module for searching for windows that meet the multimodal query information in the visible field database according to the multimodal query information uploaded by the user, where the multimodal query information is used to query windows with a specific outdoor view.

[0050] In a third aspect, the present application provides an electronic device, including: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor connected to the at least one bus; and at least one memory connected to the at least one bus.

[0051] In a fourth aspect, the present application further provides a computer storage medium storing computer-executable instructions for executing the method for searching for windows for a specific landscape according to any one of the above in the present application.

[0052] The above technical solutions provided by the embodiments of the present application have the following advantages compared with the prior art: First, a three-dimensional model is constructed through building information and a multi-perspective image set, and the geometric space representation data of the window is determined by combining image analysis. Then, a viewing frustum is generated based on a preset field of view angle and an outward normal vector, and projected onto the three-dimensional model to obtain visible field data, converting the abstract field of view into structured and quantifiable information; then, the building information, the geometric space representation data of the window, and the visible field data are integrated to construct a visible field database, and combined with multimodal query technology, to achieve a quick response to the complex query requirements of users. The present application realizes the interconnection and sharing of cross-industry data by integrating multi-source heterogeneous data. With the multimodal query function, users can break through the query limitations of traditional systems that are independently closed, and without relying on multi-platform switching and manual consultation, independently and accurately retrieve windows that meet specific outdoor view requirements, improving the user query efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The drawings here are incorporated into the specification and form a part of this specification, showing embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application.

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, other drawings can also be obtained based on these drawings without creative efforts.

[0055] One or more embodiments are exemplarily illustrated by the pictures in the corresponding drawings. These exemplary illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, unless otherwise stated. The drawings in the figures do not constitute a scale limitation.

[0056] Figure 1 System schematic diagram for finding windows for a specific landscape provided by an embodiment of the present application;

[0057] Figure 2 Method flowchart for finding windows for a specific landscape provided by an embodiment of the present application;

[0058] Figure 3 Window field of view rendering diagram of a frustum image provided by an embodiment of the present application;

[0059] Figure 4 Window field of view semantic map provided by an embodiment of the present application;

[0060] Figure 5 Window field of view depth of field diagram provided by an embodiment of the present application;

[0061] Figure 6 Knowledge graph schematic diagram provided by an embodiment of the present application;

[0062] Figure 7 Schematic diagram of the device structure for finding windows for a specific landscape provided by an embodiment of the present application;

[0063] Figure 8 Schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0064] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the scope of protection of the present application.

[0065] The following disclosure provides many different embodiments or examples for implementing different structures of the present application. To simplify the disclosure of the present application, the components and settings of specific examples are described below. Of course, they are only examples and are not intended to limit the present application. In addition, the present application may repeat reference numerals and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0066] To solve the problem of low efficiency in finding windows with specific outdoor landscapes mentioned in the background art, the embodiments of the present application infer the visible field data of windows based on the three-dimensional model of the target area, and then search for windows that meet the specific landscapes according to the user input, thereby improving the user's search efficiency.

[0067] The application fields of the present application include but are not limited to: hotel reservation, real estate transaction, urban planning and environmental assessment, tourism and cultural guide, and building design and renovation, etc.

[0068] Optionally, in the embodiments of the present application, the above method for finding windows for specific landscapes can be applied to Figure 1 the hardware environment composed of the terminal 101 and the server 103 as shown. As Figure 1 shown, the server 103 is connected to the terminal 101 through the network. The server constructs a visible field database based on the visible field data of each window in the target area. The user inputs the text and pictures to be recognized through the terminal to query the windows with specific outdoor landscapes. The server searches for the windows that meet the conditions in the visible field database according to the user input and feeds them back to the user. The database 105 can be set on the server or independently of the server to provide data storage services for the server 103. The above network includes but is not limited to: wide area network, metropolitan area network or local area network. The terminal 101 includes but is not limited to PC, mobile phone, tablet computer, etc.

[0069] Next, in combination with the specific implementation manners, a method for finding windows for specific landscapes provided by the embodiments of the present application will be described in detail. Taking the application to the server as an example, as Figure 2 shown, the specific steps are as follows:

[0070] Step 201: Obtain the building information and multi-perspective image sets in the target area, and construct a three-dimensional model of the target area according to the building information and the image sets;

[0071] Step 202: Determine the window data set by analyzing the frame images in the image set, where the window data set includes the geometric space representation data of each window in the target area;

[0072] Step 203: Determine the frustum of the window according to the preset field of view angle and the outer normal vector in the geometric space representation data, and determine the visible field data of the window generated by projecting the frustum onto the three-dimensional model, where the visible field data is used to indicate the outdoor landscape;

[0073] Step 204: Construct a visible field database of all windows in the target area according to the building information, the geometric space representation data of each window, and the visible field data;

[0074] Step 205: Search for windows that match the multimodal query information in the visibility database according to the multimodal query information uploaded by the user. The multimodal query information is used to query windows with specific outdoor views.

[0075] First, some terms mentioned in the embodiments of the present application are explained.

[0076] Multi-view image set: A series of image collections obtained by a user using devices such as mobile phones, drones, or panoramic cameras to capture a target area from different angles and positions. These images cover information about all aspects of the target area and are the basic data for constructing a 3D model.

[0077] 3D model: A 3D digital representation of the target area generated based on the multi-view image set and building information using specific algorithms and technologies, accurately showing the structural relationships and respective spatial positions among buildings, water surfaces, green plants, etc. within the target area.

[0078] Window dataset: A collection containing detailed information about each window in the target area. The geometric space representation data therein specifically describes geometric features such as the position, size, and shape of the window in 3D space.

[0079] Geometric space representation data: Data used to accurately describe the geometric features of a window in 3D space, including window frame vertex coordinates, window center point coordinates, outer normal vector (representing the orientation of the window), and window area, etc. These data can accurately determine the position and form of the window in space.

[0080] View frustum: A conical space range determined with the window center point as the viewing point and the outer normal vector of the window as the main axis direction, based on a preset horizontal field of view angle and vertical field of view angle. The light projection situation within this space range is used to determine the visible range and visibility data of the window.

[0081] Visibility data: The data result generated by projecting the view frustum onto the 3D model, containing various information about the outdoor view, such as the proportion of different objects (vegetation, sky, buildings, etc.) outside the window, whether a specific landmark (such as the Eiffel Tower) is visible and the visibility probability, etc., intuitively reflecting the external scene characteristics that the window can see.

[0082] Visibility database: Integrates the visibility data and geometric space representation data of all windows in the target area, as well as the building information in the target area. Through the centralized storage and management of these data, it is convenient to quickly query and obtain information related to the visibility of specific windows, supporting various visibility-based analysis and query operations.

[0083] Multi-modal query information: When a user searches for a window with a specific view outside the window and inputs query instructions in various ways such as text description, voice question, and uploading pictures, the system will perform corresponding processing and queries based on this information in different forms.

[0084] In step 201, the user collects a multi-view image set of the target area through devices such as mobile phones, drones, and panoramic cameras, and then uploads the uploaded image set to the server. In addition, the server also supports importing building model files, parsing and extracting key information such as window geometric dimensions, floor height, and building number. The image set can be collected in the following ways.

[0085] 1. Mobile device collection: The user quickly uploads a single photo or short video using a mobile terminal. When the data is uploaded, the system synchronously records metadata such as the shooting time, GPS (Global Positioning System) positioning coordinates, and the attitude of the mobile phone (pitch angle, roll angle, yaw angle), which is suitable for quickly capturing local scenes and instant queries, meeting the user's fragmented data collection needs.

[0086] 2. Drone aerial survey data collection: For large-scale scene requirements, a drone is used to periodically collect image data according to a preset route. During the collection process, the real-time trajectory information recorded by the inertial measurement unit of the aircraft is archived synchronously with attitude parameters such as pitch angle, roll angle, and yaw angle to ensure the acquisition of high-precision spatial data, providing a basis for 3D modeling and visibility analysis of large areas.

[0087] 3. Panoramic and indoor image supplementary collection: For areas that are difficult to observe on the building facade or indoor spaces, a 360-degree panoramic camera is used for supplementary collection. By shooting in a circular motion, a complete spatial image is obtained, filling the blind spots of traditional perspectives and constructing a more comprehensive building visibility data set, especially suitable for capturing details of buildings with complex structures.

[0088] The system constructs a 3D model of the target area using the frame images in the image set and the building information. For example, when analyzing a certain urban block, images are collected through various devices and imported into a CAD (Computer Aided Design) file, and then a 3D model of the block is constructed. The 3D model can clearly present the layout and structure of the buildings in the block.

[0089] In step 202, the system analyzes the frame images in the image set, identifies the relevant information of the windows from the frame images, and then obtains the geometric space representation data of each window, such as the coordinates of the window frame vertices, the coordinates of the center point, the outer normal vector, and the area, etc., to form a window data set containing these detailed information. The geometric space representation data provides an accurate data basis for the generation of the frustum in step 203 on the one hand, and also constructs an accurate data basis for building the attribute database on the other hand.

[0090] In step 203, based on the three-dimensional coordinates of the center point and the outer normal vector of each window in the window data set, combined with the horizontally defined or scene-adapted horizontal field of view and vertical field of view by the user, the system constructs a set of light ray sets with direction attributes through the ray tracing algorithm. These light rays start from the center point of the window and diverge evenly at a specific angle, jointly forming a frustum with a spatial geometric shape. In the three-dimensional model space, the system performs projection operations on the frustum, and uses spatial collision detection technology to track the propagation path of each light ray in real time, and accurately collects multi-dimensional data such as the intersection coordinates of the light ray and the objects in the three-dimensional model, the category labels of the hit objects (such as vegetation, sky, building, water body, landmark building, etc.), and the degree of occlusion. Through the integrated analysis of these data, detailed window visibility data is finally generated. The visibility data can not only intuitively present the specific composition of the view outside the window, but also accurately quantify the proportion and distribution of various landscape elements. For example, when analyzing the windows of a hotel, information such as the visible probability of each window looking at the landmark building and the greening rate outside the window can be clearly obtained.

[0091] The embodiment of the present application transforms the abstract and fuzzy concept of the field of view into structured and quantifiable visibility data. Through ray tracing and spatial collision detection technologies, the projection situation of the frustum of each window is accurately calculated, and multi-dimensional information such as the intersection coordinates of the light ray and various objects in the three-dimensional model, the category of the hit object, and the landmark visibility probability is detailedly recorded. After being standardized, these quantitative data can be efficiently integrated into the visibility database to form a data storage system with clear logic and distinct levels. This data storage system improves the data retrieval and analysis efficiency. Even in the face of extremely detailed query requirements from users (such as the visibility of specific landmarks, the exact value of the vegetation coverage rate outside the window, and the change of the field of view at different time periods), the system can rely on the structured visibility data to quickly locate and match relevant information and respond to the requirements with high precision.

[0092] In step 204, the system constructs a visibility database for all the windows in the target area based on the building information of the target area (including building numbers, floor heights where each window is located, etc.), the high-precision geometric spatial representation data of each window (coordinates of window frame vertices, center point coordinates, outer normal vectors, areas, etc.), and the quantified visibility data (intersection points of light and objects, hit object categories, landmark visibility probabilities, etc.). The visibility database facilitates the management and query of a large number of windows and provides data support for multimodal queries.

[0093] In step 205, the user inputs query information in multimodal ways such as text, voice, and pictures. After receiving it, the system converts the voice to text, analyzes the query intention by combining technologies such as image feature extraction, transforms it into query conditions, then searches for windows that meet the conditions in the visibility database according to specific query algorithms and logics, and finally filters and sorts the found windows and feeds back the results to the user. The multimodal query function meets the diverse needs of users and improves the query efficiency and accuracy. For example, when the user is in a certain scenic area, uploads a photo containing the scenic area landmark and inputs the occlusion rate condition, the system can quickly screen out the windows that meet the conditions and display their positions on the mobile terminal.

[0094] In this application, first, a three-dimensional model of the target area is constructed through building information and a multi-view image set. The geometric spatial representation data of the windows is determined by combining image analysis. Then, a viewing frustum is generated based on a preset field of view angle and outer normal vector and projected onto the three-dimensional model to obtain visibility data, converting the abstract field of view into structured and quantifiable information. Then, the building information, the geometric spatial representation data of the windows, and the visibility data are integrated to construct a visibility database. Combining multimodal query technology, it realizes a quick response to the complex query needs of users. This application realizes the interconnection and sharing of cross-industry data by integrating multi-source heterogeneous data. With the help of the multimodal query function, users can break through the query limitations of traditional systems that are independent and closed, and without relying on multi-platform switching and manual consultation, independently and accurately retrieve the windows that meet specific outdoor landscape requirements, improving the user query efficiency.

[0095] As an optional implementation manner, in step 201, constructing a three-dimensional model of the target area according to the image set includes the following steps.

[0096] Step S11: Perform feature point matching on the frame images in the image set and recover the camera parameters used when the frame images were taken in a preset manner, where the camera parameters include shooting parameters and camera pose parameters;

[0097] Step S12: Generate a sparse three-dimensional point cloud of the target area according to the matched feature point pairs, camera parameters, and building information;

[0098] Step S13: Process the sparse three-dimensional point cloud through density reconstruction to generate a dense point cloud, where the density of the dense point cloud is higher than that of the sparse three-dimensional point cloud;

[0099] Step S14: Use the Poisson reconstruction algorithm to convert the dense point cloud into a three-dimensional model in a unified coordinate system.

[0100] The system extracts feature points from the frame images in the image set, identifies key points with uniqueness in each frame image, and then uses the method based on descriptor matching to complete the feature point matching of cross-frame images, generating matching feature point pairs. At the same time, the system uses preset algorithms such as bundle adjustment to solve and recover the camera parameters when each frame of image is taken, including shooting parameters such as focal length and distortion coefficient, as well as camera pose parameters such as position and rotation angle.

[0101] Based on the matching feature point pairs and accurate camera parameters, and combined with data such as geographical coordinates and structural layout in the building information, the system uses the principle of triangulation to calculate the three-dimensional spatial coordinates of the feature points, generating a sparse three-dimensional point cloud of the target area. The geographical coordinates in the building information are used to determine the position of the point cloud in the real-world coordinate system, and the structural layout information assists in verifying the rationality of the point cloud distribution, ensuring that the point cloud preliminarily outlines the spatial framework of the target area.

[0102] To improve the detail expression ability of the point cloud, the system uses density reconstruction technology (such as multi-view stereo algorithm) to process the sparse three-dimensional point cloud. By analyzing the perspective differences between adjacent images and using the texture and depth information of the images, more points are interpolated on the basis of the sparse point cloud to increase the point cloud density and form a dense point cloud, which can present the surface details of the target area more finely.

[0103] For the dense point cloud, the system uses the Poisson reconstruction algorithm to convert the discrete point cloud data into a continuous triangular mesh model through implicit surface fitting. During the reconstruction process, if the building information contains global coordinates, the iterative closest point algorithm is used to align the model to the world coordinate system; if the global coordinates are missing, coordinate calibration is performed through a small number of ground control points, and finally a three-dimensional model with clear texture and accurate geometry in a unified coordinate system is generated.

[0104] In this application, feature point matching and camera parameter recovery ensure the accuracy of three-dimensional reconstruction; generating a sparse point cloud in combination with building information improves the geographical positioning accuracy and spatial structure rationality of the model; density reconstruction enhances the model detail expressiveness; Poisson reconstruction and coordinate calibration ensure the geometric integrity and coordinate system consistency of the model. The generated high-precision three-dimensional model can truly restore the spatial form of the target area.

[0105] As an alternative implementation, in step 202, by analyzing the frame images in the image set, it is determined that the window data set includes the following content.

[0106] Step S21: By back-projecting the two-dimensional window frame corner points in the frame image onto the three-dimensional model, a three-dimensional window frame contour under the image path is obtained, where the three-dimensional window frame contour is used to indicate the shape contour of the window in the three-dimensional space;

[0107] Step S22: Determine the window point clusters under the point cloud path according to the dense point cloud of the target area, and determine the geometric space representation data of the window according to the window point clusters, where the window point clusters are used to indicate the point cloud data of multiple windows;

[0108] Step S23: Determine the overlap degree of the three-dimensional window frame contour under the image path and the window point clusters under the point cloud path on the two-dimensional plane;

[0109] Step S24: If the overlap degree is greater than the preset overlap degree threshold, determine that the three-dimensional window frame contour and the window point clusters are the same window, and use the geometric space representation data under the point cloud path as the geometric space representation data of the window;

[0110] Step S25: Construct a window data set according to the geometric space representation data of multiple windows.

[0111] In step S21, on the image path, the system uses a pre-trained semantic segmentation model (such as the DeepLabv3+ model) to perform pixel-level analysis on the frame image, quickly identify the window area and generate a window frame mask. Subsequently, the two-dimensional window frame corner points are extracted through a corner detection algorithm, and combined with accurate camera parameters (such as pose, internal parameter matrix), and they are back-projected onto the three-dimensional model using the principle of triangulation, thereby constructing a high-precision three-dimensional window frame contour. The advantage of this path is that it can utilize the rich texture and semantic information of the image to accurately capture the boundary features of the window, especially suitable for the contour extraction of windows with complex appearances. Even in scenes with light and shadow changes or local occlusion, it can quickly locate the window position.

[0112] In step S22, in terms of the point cloud path, the system uses a density-based clustering algorithm to deeply mine the dense point cloud data of the target area. Since the window structure presents a unique point cloud aggregation form in space, by setting optimized density thresholds and neighborhood parameters, the window point cloud can be efficiently separated from the overall point cloud to form independent window point clusters. For each window point cluster, the system further calculates geometric parameters such as the window frame vertex coordinates, center point coordinates, and outer normal vector to generate accurate geometric space representation data. The core advantage of this path lies in relying on the three-dimensional space characteristics of the point cloud data, which can accurately describe the three-dimensional geometric shape of the window, has strong ability to capture details such as window depth and concave-convex structure, and is especially suitable for scenes where the building facade is complex and it is difficult to directly judge the structure through images.

[0113] In step S23, in order to verify whether the three-dimensional window frame contour in the image path and the window point cluster in the point cloud path correspond to the same window, the system projects the three-dimensional window frame contour and the window point cluster onto a two-dimensional plane respectively. By calculating the overlapping area of the two on the two-dimensional plane and comparing it with the total area of the two, an overlapping degree value is obtained, and this overlapping degree reflects the degree of coincidence of the window information determined by the two methods.

[0114] Although the image path can accurately identify the window contour, it is difficult to accurately obtain the spatial depth information of the window due to the limitations of the image perspective and occlusion problems; although the point cloud path can provide reliable three-dimensional geometric space representation data, in a complex texture background, point cloud clustering errors may occur. In the embodiment of the present application, by projecting the three-dimensional window frame contour and the window point cluster onto a two-dimensional plane, calculating the overlapping degree and comparing it with a preset threshold, the advantages of the two paths can be effectively integrated, and the errors of a single data source can be eliminated. Only when the overlapping degree meets the standard can it be ensured that the obtained window geometric space representation data has both the contour accuracy of the image path and the spatial accuracy of the point cloud path, improving the accuracy of the geometric space representation data.

[0115] In step S24, the system pre-sets an overlapping degree threshold. If the calculated overlapping degree is greater than this threshold, it indicates that the window information obtained from the two different paths of the image and the point cloud has a high degree of consistency. From this, it is determined that the three-dimensional window frame contour and the window point cluster represent the same window. At this time, the geometric space representation data determined in the point cloud path is used as the final geometric space representation data of the window, because the point cloud data has higher accuracy and reliability in describing spatial geometric features.

[0116] In step S25, after the system performs the above processing on all windows in the target area, it integrates the geometric space representation data of each window and constructs a complete window data set according to a certain data structure and storage format. This data set contains detailed and accurate geometric information of all windows in the target area, providing a data basis for subsequent window visibility analysis, database construction, and other operations.

[0117] In the present application, the image path relies on semantic segmentation and camera parameter inversion, and uses the image texture details to accurately outline the window contour for processing boundary recognition in complex appearance and occlusion scenarios; the point cloud path relies on the density clustering algorithm to accurately capture the three-dimensional structure of the window based on the three-dimensional space distribution characteristics. Through the collaborative operation of the image and point cloud dual paths and cross-verification, the image path can accurately identify the appearance characteristics of the window, and the point cloud path can accurately describe the spatial geometric structure of the window. Window information with accurate contours and accurate geometry can be selected, so that the verified geometric space representation data has both the visual accuracy of the image and the spatial depth of the point cloud, improving the reliability of the data.

[0118] As an alternative implementation, in step S21, by back-projecting the two-dimensional window frame corner points in the frame image onto the three-dimensional model, the three-dimensional window frame contour in the image path is obtained, including the following content.

[0119] Step S211: Perform semantic segmentation on the frame image through a semantic segmentation model to determine the initial window frame mask in the frame image;

[0120] Step S212: Determine the final window frame mask of the window by tracking the same window in multiple consecutive frame images;

[0121] Step S213: Extract the two-dimensional window frame corner points in the final window frame mask;

[0122] Step S214: Back-project the two-dimensional window frame corner points onto the three-dimensional model according to the camera parameters corresponding to the frame image to obtain the three-dimensional window frame contour of the window in the image path, where the camera parameters include shooting parameters and camera pose parameters.

[0123] In step S211, the system calls a semantic segmentation model (such as DeepLabv3+, U-Net, etc.) trained with a large amount of window image data to perform pixel-level semantic segmentation on the input frame image. Based on deep learning algorithms, this model can learn the texture, shape and other feature patterns of windows in the image. Through forward propagation calculation, it assigns class labels to each pixel in the image, separates the window area from the complex background environment, and generates the initial window frame mask. This mask is presented in the form of a binary image, where the white area represents the window and the black area is the background, initially delineating the position and approximate contour of the window in the image.

[0124] In step S212, to improve the accuracy of the window frame mask, the system uses a multi-frame object tracking algorithm to dynamically track the same window in multiple consecutive frame images. The algorithm analyzes the motion trajectory and appearance feature changes of the window between adjacent frames, establishes the correspondence of the window between frames, and excludes misidentifications or missed detections caused by factors such as occlusion and lighting changes. After comprehensive processing and optimization of multiple frames of data, the final window frame mask of the window is finally determined, which can more accurately depict the true contour and boundary of the window.

[0125] In step S213, after obtaining the final window frame mask, the system uses an edge detection algorithm to extract the edges of the mask image to obtain the pixel-level contour of the window edge. Subsequently, a corner point detection algorithm is used to locate the two-dimensional window frame corner points with obvious features on the contour.

[0126] In step S214, based on the accurate camera parameters corresponding to each frame of image (including shooting parameters such as focal length and distortion coefficient, as well as camera pose parameters such as position and rotation angle), the system uses the principle of triangulation to back-project the two-dimensional window frame corner points from the image plane into the pre-constructed three-dimensional model space. Specifically, the system converts the image coordinates into coordinates in the camera coordinate system through the internal parameter matrix of the camera, and then combines the external parameter matrix (i.e., the camera pose parameters) to convert the coordinates in the camera coordinate system to the world coordinate system, so as to calculate the positions of the two-dimensional window frame corner points in the three-dimensional space, and finally constructs the three-dimensional window frame contour of the window under the image path, intuitively presenting the shape, size and position information of the window in the three-dimensional space.

[0127] In this application, the semantic segmentation model realizes the automatic and pixel-level accurate recognition of the window area, improving the accuracy and stability of window recognition; the multi-frame tracking algorithm further eliminates the interference of environmental factors on the recognition results, ensuring that the window frame mask can accurately fit the actual contour of the window. The system accurately extracts the two-dimensional window frame corner points, and then uses the accurate camera parameters and the principle of triangulation for two-dimensional to three-dimensional conversion, so that the three-dimensional window frame contour obtained from the image path can highly restore the shape and position of the window in the real space.

[0128] As an optional implementation manner, in step S22, determining the geometric space representation data of the window according to the window point cluster includes the following contents.

[0129] Step S221: Repeat the following operations: fit an initial plane according to some randomly selected points from the window point cluster, and determine the distance from each point in the window point cluster to the initial plane; determine the number of points whose distance is less than the preset distance threshold, where the initial plane is used to indicate a wall surface with multiple windows;

[0130] Step S222: After reaching the set number of repeated operations, take the initial plane with the largest number of points as the target plane;

[0131] Step S223: Determine the geometric space representation data of each window according to the distribution characteristics of the window point cluster in the target plane.

[0132] In step S221, the system adopts an iterative random sampling and plane fitting strategy for each window point cluster. First, 20%-30% of the points are randomly selected from the window point cluster, and an initial plane is fitted using the least squares method. This plane can approximately represent the wall containing multiple windows. Subsequently, based on a preset distance threshold (such as 5 cm), the system calculates the perpendicular distance from each point in the point cluster to the initial plane through vector operations. Points with a distance less than the threshold are determined as "inliers", representing valid data that closely adheres to the wall plane; points with a distance greater than the threshold are "outliers", representing invalid data caused by uneven wall surfaces, noise interference, etc. The system counts the number of inliers under each fitted plane, and generates multiple groups of different initial planes and their corresponding inlier distribution results through multiple random samplings.

[0133] In step S222, when the iterative operation reaches the set number of 50-100 times, the system comprehensively evaluates all the generated initial planes. Due to the complex shapes of building walls such as convex decorations and concave window frames, the fitting degrees of different initial planes to the real wall vary due to sampling randomness. The system selects the plane with the largest number of inliers as the target plane by comparing the number of inliers of each initial plane. This is because the more inliers there are, the higher the degree of fit of the plane to the real shape of the wall, which can filter out interference points caused by wall surface unevenness and data noise to the greatest extent, accurately restore the spatial relationship between the wall and the window, and provide a reliable basis for subsequent geometric space characterization data calculation. This method is especially suitable for processing buildings with complex facades such as relief decorations, curved window frames, and special-shaped curtain walls, effectively avoiding the problem of plane misjudgment in the case of irregular wall scenes by the traditional fixed threshold method, and improving the accuracy and robustness of geometric space characterization data extraction.

[0134] In step S223, after determining the target plane, the system further analyzes the distribution characteristics of the window point cluster in the target plane. By calculating the convex hull boundary of the point cluster, the vertex coordinates of the window frame are identified; the centroid algorithm is used to determine the center point coordinates of the window; by calculating the normal vector of the point cluster, the external normal vector of the window is obtained to determine the orientation of the window; finally, according to the area of the region covered by the point cluster, the actual area of the window is calculated. These operations comprehensively use point cloud processing algorithms to accurately extract the geometric space characterization data of each window from the target plane, and completely depict the shape, position, and orientation information of the window in three-dimensional space.

[0135] In this application, through the method of random sampling and iterative fitting, the system can adaptively process complex and variable wall shapes. Whether it is a regular planar wall or an irregular wall with local unevenness, it can accurately fit the target plane representing the wall. Compared with the traditional plane fitting method with fixed parameters, this method improves the adaptability to different building structures and reduces the calculation error of geometric space characterization data caused by wall shape differences.

[0136] Based on a reliable target plane, subsequent analysis of the distribution characteristics of the window point clusters can more accurately determine the geometric space representation data of the window. The screening mechanism of the preset distance threshold can effectively filter out the noise points and outliers in the point cloud data, avoiding the influence of these interfering data on plane fitting and parameter calculation. At the same time, the strategy of multiple iterative samplings enhances the robustness of the algorithm. Even when there are local missing or error in the point cloud data, it can still stably output reliable target plane and geometric space representation data.

[0137] As an alternative implementation, in step S23, determining the overlap degree between the three-dimensional window frame contour in the image path and the window point clusters in the point cloud path on a two-dimensional plane includes the following content.

[0138] In step S231, project the three-dimensional window frame contour in the image path onto a two-dimensional plane to obtain a polygonal contour;

[0139] In step S232, project the window point clusters in the point cloud path onto a two-dimensional plane to obtain a projected point cluster;

[0140] In step S233, select the internal point clusters that fall within the polygonal contour from the projected point cluster;

[0141] In step S234, calculate the overlap degree according to the area of the region formed by the internal point clusters and the area of the region formed by the polygonal contour.

[0142] In step S231, the system projects the three-dimensional window frame contour constructed in the image path onto a two-dimensional plane by means of orthogonal projection or perspective projection. During the projection process, according to the coordinate system of the three-dimensional model and the preset projection rules, the vertex coordinates of the window frame contour in three-dimensional space are converted, and finally a two-dimensional polygonal contour is formed. This polygonal contour completely retains the shape and boundary information of the window in the image path, the number of its vertices corresponds to the number of corner points of the three-dimensional window frame contour, and the connection relationship of the edges also matches the window frame structure in three-dimensional space.

[0143] In step S232, for the window point clusters in the point cloud path, the system also performs a projection operation. The points in each window point cluster in three-dimensional space are mapped onto a two-dimensional plane one by one according to the same projection rules and coordinate system conversion methods as the three-dimensional window frame contour, forming a projected point cluster. The distribution of these projected points on the two-dimensional plane reflects the morphological characteristics and position relationships of the window point cloud in three-dimensional space, and retains the discrete characteristics of the point cloud data.

[0144] In step S233, the system detects each point in the projected point cluster through the inclusion relationship between the point and the polygon. Specifically, each point is traversed from the projected point cluster to determine whether it falls inside the polygon contour generated in step S231. If the point is inside the polygon, it is included in the internal point cluster; otherwise, it is excluded. Through this operation, the point cloud data with a spatial overlap relationship with the window frame contour in the image path can be accurately screened out.

[0145] Step S234: The system calculates the area of the region formed by the internal point cluster and the area of the region formed by the polygon contour respectively. For the internal point cluster, the convex hull algorithm or the area calculation method based on the grid is used to obtain the area of the region it covers on the two-dimensional plane; for the polygon contour, the area is directly obtained using the polygon area calculation formula. Finally, the area of the internal point cluster region is divided by the area of the polygon contour region to obtain the overlap degree value. This value intuitively reflects the spatial coincidence degree of the window information obtained from the image path and the point cloud path in the form of a percentage. When the calculated overlap degree value reaches the preset threshold (such as 80%), it means that the three-dimensional window frame contour in the image path and the window point cluster in the point cloud path are highly coincident in spatial form and position, and the verification process passes smoothly.

[0146] In this application, by uniformly projecting the three-dimensional window frame contour and the window point cluster onto the two-dimensional plane, the intuitive comparison and quantitative analysis of the image data and the point cloud data in the same dimension are realized. In the building scene, the image data may have contour deviations due to occlusion and perspective problems, and the point cloud data may have errors due to noise and sampling density. By calculating the overlap degree, the system can quantitatively evaluate the consistency of the information obtained from the two paths, effectively eliminate the false matches caused by data errors or ambiguities, improve the reliability of window recognition, and avoid misjudgment caused by the limitations of a single data source. Only when the overlap degree meets the preset threshold, the geometric space representation data in the point cloud path is used as the final information of the window, ensuring the high quality of the window data set. This method has strong adaptability to complex building scenes. Whether it is a regular or irregular shaped window, whether it is image data with light changes and occlusion, or point cloud data with noise and holes, it can be effectively screened and verified through two-dimensional projection and overlap degree calculation, enhancing the stability and reliability of the system under different data qualities and complex environments.

[0147] As an optional implementation manner, in step 204, according to the building information, the geometric space representation data of each window, and the visible field data, constructing a visible field database for all windows in the target area includes the following content.

[0148] Step S31: Determine the frustum images of the visible field data, and construct a frustum database based on the frustum images of multiple windows;

[0149] Step S32: Generate visual vectors based on the frustum images of the windows, and construct a vector database based on the visual vectors of multiple windows;

[0150] Step S33: Determine the building floor data where the windows are located according to the preset building information, determine the visual environment indicators according to the visible field data, and construct an attribute database by combining the building floor data, geometric space representation data, and visual environment indicators of multiple windows;

[0151] Step S34: Construct a knowledge graph database, where the knowledge graph database takes entities, buildings, blocks, and windows as nodes, and takes the visible entities, the buildings where the windows are located, and the blocks where the windows are located as edges;

[0152] Among them, the frustum database, vector database, attribute database, and knowledge graph database all include window identifiers.

[0153] In step S31, in the frustum generation stage, the system traverses all the windows in the target area, collects and organizes the frustum images of each window, and constructs a frustum database. During the construction of the frustum database, to ensure the accuracy and integrity of the data, the frustum images are standardized, and parameters such as image resolution and color space are unified. At the same time, the indexing of the frustum images is optimized so that the corresponding frustum images can be quickly retrieved according to the window ID, improving the data query efficiency.

[0154] Figure 3 Is the window field of view rendering diagram of the frustum image. From Figure 3 It can be seen that the visible field of the window includes but is not limited to: the urban skyline, waterfront landscape, high-rise buildings, urban density, and visible corridors, etc. The scene description of this image is: Looking into the distance through the window, a high-density waterfront urban landscape can be seen. In the foreground are low-rise industrial or office buildings, and the line of sight gradually extends to the dense high-rise buildings on both sides of Victoria Harbour and the distant mountain background, forming a rich depth of scene. The mountains and the sea blend in the scene, forming a symbiotic visual corridor between nature and the city. The high-rise towers and the coastal skyline form the visual focus, with good openness and depth. In addition, this image also has corresponding final semantic vectors (visual vectors and text vectors).

[0155] In step S32, the system reprojects the frustum image into an image snapshot with a fixed resolution. This image snapshot, as the carrier of visual information, is input into the visual pre-training model for vectorization encoding. The model snapshot generates a set of floating-point vectors with the same length for the frustum image of each window through feature extraction and transformation of the image, that is, visual vectors.

[0156] Figure 4The window field of view semantic map is generated based on the image snapshot. Each color represents a semantic information. The window field of view semantic map uses different colors to distinguish semantic information such as buildings, corridors, sky and water, allowing users to intuitively know the category composition of the scenery outside the window.

[0157] Figure 5 The window field of view depth map is generated based on the image snapshot. The window field of view depth map is used to display the depth information of the landscape outside the window and present the distance and hierarchical relationship of each part of the landscape.

[0158] The window field of view semantic map and window field of view depth map can provide more image feature information for visual vector generation, so that the generated vector can more accurately and comprehensively describe the scene outside the window.

[0159] Preferably, the system can also obtain prompts input by the user, which are used to describe the scene outside the window of the image snapshot. The system feeds the constructed prompts to the large language model. The large language model, with its powerful language understanding and generation capabilities, conducts in-depth analysis and processing of the input information, and then generates a brief Chinese summary, and then converts the Chinese summary into a numerical vector representation, that is, a text vector.

[0160] At this point, the system already has the visual vector and text vector corresponding to each window. Because the visual vector and text vector are dimensionally compatible, they can be spliced together bit by bit. The final semantic vector formed combines information from both visual and textual modalities, which can more comprehensively and accurately describe the scene outside the window. Finally, the final semantic vector is written into the vector database. This application considers both visual and textual information, improving the accuracy of the vector database. The vector database is indexed by the window ID, and the corresponding final semantic vector can be found.

[0161] In step S33, the system determines the building floor data for each window based on pre-set building information, such as the building's floor structure and floor height. Simultaneously, based on the visual area data obtained from the visual area analysis, it calculates visual environment indicators such as greening ratio, occlusion ratio, and sky visibility. Combined with the window's geometric spatial representation data (including window frame vertex coordinates, center point coordinates, external normal vector, area, etc.), the system constructs an attribute database.

[0162] In the construction of the attribute database, the window field of view semantic map and the window field of view depth map can jointly assist in determining the visual environment indicators. The greening rate and occlusion rate can be analyzed through the window field of view semantic map, and the spatial depth relationship can be judged using the window field of view depth map to improve the attribute data.

[0163] When building the attribute database, a suitable spatial database management system is selected to structurally store the data. The various attribute data of the windows are classified and stored according to different fields, and a unique identifier (window ID) is established for each window. Association queries for the various attribute data are realized through the window ID.

[0164] The attribute database integrates the building floor data, geometric spatial representation data, and visual environment indicators of the windows, providing rich data support for in-depth analysis of the characteristics of the windows and their surrounding environment. For example, in the real estate field, these data can be used to evaluate value factors such as the lighting and view of a house; in urban planning, it can be used to analyze the landscape visibility and environmental quality of different regions.

[0165] In step S34, the system constructs a knowledge graph database, with entities (such as landmark buildings, green areas, etc.), buildings, blocks, and window IDs as nodes, and the visible entities, the buildings where the windows are located, and the blocks where the windows are located as edges, to construct a complex relationship network. For example, "window - visible - landmark" represents the relationship between the window and the visible landmark, "window - belongs to - building" represents the relationship between the window and the building to which it belongs, and "building - located in - block" represents the relationship between the building and the block where it is located.

[0166] This application can also enrich the semantic information of the knowledge graph by defining the attributes of nodes and edges. For example, adding attributes such as visibility probability and occlusion degree to the "window - visible - landmark" edge enables the knowledge graph to more accurately describe the relationship between the window and the surrounding environment. The embodiments of this application can periodically update and expand the knowledge graph. With the collection and analysis of new data, the structure and content of the knowledge graph are continuously improved to adapt to the changing actual situation.

[0167] Figure 6 It is a four - level knowledge graph structure of "window - entity - building - block". Each node and edge in the graph represents the complex semantic relationship between the window and the visible elements around it. For example, window 7 belongs to Jiangfeng Villa, and at the same time window 7 has the feature: medium green view rate, and Jiangfeng Villa is located in Dapu.

[0168] The knowledge graph database visually displays the relationships between the window and the surrounding entities, buildings, and blocks in a graphical way. This relationship model is not only easy to understand and visualize, but also supports complex relationship reasoning. For example, through the knowledge graph, it is possible to quickly query all the visible windows within a certain range around a landmark building, or the window information of specific types of buildings in a certain block.

[0169] Table 1 shows the multiple sub - databases included in the visibility field database and the main content of each sub - database.

[0170] Table 1

[0171]

[0172] In this application, by constructing a cone database, a vector database, an attribute database, and a knowledge graph database, multi-source heterogeneous data related to windows is integrated, and associations are established with the window identifier as the link to achieve unified management and efficient storage of the data. This structured data organization method enables quick positioning and acquisition of the required information when querying and analyzing data, reduces data processing time, and improves the system operation efficiency.

[0173] As an alternative implementation, in step 205, finding the windows that match the multi-modal query information in the visual field database according to the multi-modal query information uploaded by the user includes the following content.

[0174] Step S41: Obtain the text to be recognized and the image to be recognized uploaded by the user, where the text to be recognized and the image to be recognized contain the outdoor landscape that the user expects to see;

[0175] Step S42: Determine the image vector of the image to be recognized, and find the first set of window identifiers that match the image vector in the vector database. If the image vector contains landmark entities, add landmark keywords to the text to be recognized;

[0176] Step S43: Find the second set of window identifiers that meet the text to be recognized in the first set of window identifiers in the attribute database;

[0177] Step S44: Find the target nodes and target edges that meet the second set of window identifiers in the knowledge graph database, and construct the third set of window identifiers according to the target nodes and target edges;

[0178] Step S45: Determine the cone image to be recognized of the image to be recognized, and find the final set of window identifiers that match the cone image to be recognized in the third set of window identifiers in the cone database;

[0179] Step S46: Determine all the window identifiers in the final set of window identifiers.

[0180] In step S41, the system provides a convenient interaction interface to support the user in uploading the text to be recognized and the image to be recognized containing the information of the expected outdoor landscape. The user can upload the text description of the expected outdoor landscape (such as being able to see the sea and the beach) and the relevant pictures (such as pictures of the sea and the beach) through the mobile terminal or the web terminal. The system will perform format verification and preliminary preprocessing on the uploaded files to ensure the integrity and availability of the data.

[0181] In step S42, the system uses an image feature extraction algorithm to deeply analyze the image to be recognized and extract its image vector. This image vector is a numerical representation of the image content and contains key feature information such as the color, texture, and shape of the image. After the extraction is completed, the system will perform a search in the vector database. By calculating the similarity (such as cosine similarity) between the image vector and the visual vectors already stored in the database, it will find the window identifiers corresponding to the vectors with higher similarity, forming the first window identifier set.

[0182] If landmark entities (such as the Eiffel Tower, the Oriental Pearl Tower, etc.) are recognized in the extracted image vector through the object detection algorithm, the system will automatically extract keywords from the names of the landmark entities and add these keywords to the text to be recognized. For example, if the Eiffel Tower is recognized in the image, the system will add the keyword "Eiffel Tower" to the original text to be recognized, further enriching the query conditions and improving the accuracy of the search.

[0183] In step S43, for the first window identifier set, the system will perform a screening in the attribute database. The system uses natural language processing technology to parse the text to be recognized and extract key information such as the greening rate, the occlusion rate requirement, and the landmark name. Then, the system queries the corresponding window information in the attribute database according to these conditions. For numerical conditions such as the greening rate and the occlusion rate, the system will directly compare the visual environment indicators of the windows in the attribute database; for text conditions such as the landmark name, the system will perform a text matching operation. Only the identifiers corresponding to the windows that meet all the extracted conditions will be included in the second window identifier set, thus further narrowing the query scope.

[0184] In step S44, based on the second window identifier set, the system performs in-depth mining in the knowledge graph database. The knowledge graph database constructs a complex relationship network with entities, buildings, blocks, and windows as nodes and the visible entities, the buildings where the windows are located, and the blocks where the windows are located as edges. The system will find the relevant target nodes and target edges in the knowledge graph according to the window identifier. For example, it will find the "visible" relationship edge between the window and the landmark entity, as well as the relationships between the window and the building and block to which it belongs. Through the sorting and screening of these relationships, the system can obtain window information more relevant to the query conditions and construct the third window identifier set. This process makes full use of the rich semantic relationships in the knowledge graph database and further improves the accuracy and relevance of the query results.

[0185] In step S45, the system generates a cone image to be recognized by simulating based on information such as the shooting perspective and position of the image to be recognized (if there is relevant metadata). This cone image represents the range and content of the scene that can be observed from the perspective of the user taking the image. Then, in the cone database, for each window in the third window identification set, the system compares the similarity between the corresponding cone image and the cone image to be recognized. The similarity calculation can use methods such as the Hausdorff distance comparison. Only the identification corresponding to the window whose cone image similarity reaches a certain threshold will be included in the final window identification set, thus ensuring that the query result highly matches the user's expectation in terms of visual effect and visible range.

[0186] In step S46, the system obtains all the window identifications in the final window identification set. For these windows, a learning to rank model is used for comprehensive scoring. The learning to rank model comprehensively considers multiple features such as perspective similarity, landmark confidence, and the inverse of the occlusion rate. Among them, the perspective similarity is determined by comparing the similarity between the cone image to be recognized and the actual cone image of the window; the landmark confidence is evaluated based on the probability and accuracy of the visible landmark entities of the window; and the inverse of the occlusion rate takes the reciprocal of the occlusion rate of the window to ensure that the window with a lower occlusion rate has an advantage in the scoring.

[0187] Through the calculation of the learning to rank model, the system ranks these windows to generate a final result list. At the same time, for the convenience of the user's understanding and use, the system attaches a Chinese interpretable explanation to each result in the list, elaborating in detail the reasons why the window meets the query conditions, as well as a confidence value, which is used to indicate the matching degree between the result and the user's query conditions, enabling the user to intuitively understand the reliability and relevance of each query result.

[0188] Exemplarily, a user plans to rent a house near a tourist scenic area and expects to see green mountains and a landmark ancient temple outside the window, with a wide view and few obstructions. The user uploads a visible (image to be recognized) containing green mountains and the ancient temple to the system and enters "There are green mountains outside the window and the view is wide" (text to be recognized). The system first extracts the image vector, finds the first set of window identifiers corresponding to similar visual vectors in the vector database. Since the image contains the landmark of the ancient temple, the system adds "ancient temple" to the text to be recognized. Then, in the attribute database, it filters out the second set of window identifiers with a low obstruction rate and visible green mountains and the ancient temple according to the new text. Then, in the knowledge graph database, it searches and constructs the third set of window identifiers with the triples (window - visible - green mountain, window - visible - ancient temple). Then, it simulates and generates the cone of vision image to be recognized according to the photo, and finds the final set of window identifiers that match it in the cone of vision database. Finally, the system comprehensively scores and ranks these windows according to features such as view angle similarity, landmark confidence, and inverse of obstruction rate, and generates a result list with Chinese explanations and confidence values, such as "Window W1, confidence 85%, can clearly see green mountains and the ancient temple, wide view; Window W5, confidence 70%, can see part of the green mountains and the ancient temple, relatively wide view", etc., to help the user quickly understand the matching degree of each window with the requirements.

[0189] In this application, the system integrates information in two modalities of text and image for retrieval. When the user holds a mobile device and aims it at the target building, the system quickly analyzes the captured image, accurately identifies the candidate windows. When the user inputs the desired window scene by voice, the system can respond quickly and accurately, making full use of the semantic description of the text to be recognized and the intuitive visual information of the image to be recognized. Through hierarchical screening and matching, it obtains relevant information from different types of databases, achieving the instant interaction effect of "what you shoot is what you get". The user does not need to wait for a long time and can obtain the required window information instantly when raising the device, improving the accuracy and efficiency of retrieval.

[0190] This application provides an overall process for finding windows for a specific landscape, including the following steps.

[0191] 1. Data preparation: Collect building information in the target area, and at the same time obtain a multi - perspective image set, such as from drone aerial photography, ground shooting, etc. Organize this data to prepare for building a 3D model.

[0192] 2. Build a 3D model: Extract feature points from the multi - perspective images and match them. Use the "structure from motion" method to recover the camera parameters to obtain a sparse 3D point cloud. Then use the multi - view density recovery algorithm to encrypt the point cloud, and generate a textured 3D model through Poisson reconstruction, and integrate it with the building information.

[0193] 3. Determine the window dataset (image path): Process the frame image with a semantic segmentation model to obtain an initial window frame mask, and track consecutive frames to determine the final window frame mask. Extract the two-dimensional window frame corner points, and back-project them into the three-dimensional model in combination with the camera parameters to obtain the three-dimensional window frame contour under the image path.

[0194] 4. Determine the window dataset (point cloud path): Input the dense point cloud of the target area into a point cloud instance segmentation network to obtain window point clusters. Randomly select points to fit an initial plane, and after repeating the operation multiple times, determine the target plane. Determine the window geometric space representation data based on the distribution of window point clusters on the target plane.

[0195] 5. Fuse and determine the window dataset: Project the three-dimensional window frame contour of the image path and the window point clusters of the point cloud path onto a two-dimensional plane, and calculate the overlap degree. If the overlap degree exceeds the preset threshold, use the geometric space representation data of the point cloud path as window data to construct a window dataset.

[0196] 6. Determine the visible field data and construct a database: Determine the viewing frustum according to the outer normal vector of the window and the preset field of view angle, and project it onto the three-dimensional model to obtain the visible field data. Generate a viewing frustum image based on the visible field data and construct a viewing frustum database; extract visual vectors to construct a vector database; integrate various data to construct an attribute database and a knowledge graph database.

[0197] 7. Multimodal query processing: Obtain the text to be recognized and the image to be recognized uploaded by the user, extract the image vector of the image to be recognized and find the first set of window identifiers that match in the vector database, then filter in the attribute database to obtain the second set of window identifiers, then process in the knowledge graph database to obtain the third set of window identifiers, and finally find the final set of window identifiers in the viewing frustum database to complete the processing through hierarchical query.

[0198] Based on the same technical concept, the present application provides a device for finding windows for a specific landscape, as Figure 7 shown, the device includes:

[0199] An acquisition and construction module 701, configured to acquire building information and a multi-view image set in a target area, and construct a three-dimensional model of the target area according to the building information and the image set;

[0200] A first determination module 702, configured to determine a window dataset by analyzing the frame images in the image set, where the window dataset includes geometric space representation data of each window in the target area;

[0201] A second determination module 703, configured to determine the viewing frustum of the window according to the preset field of view angle and the outer normal vector in the geometric space representation data, and determine the visible field data of the window generated by projecting the viewing frustum onto the three-dimensional model, where the visible field data is used to indicate the landscape outside the window;

[0202] A construction module 704, configured to construct a visible field database of all windows in a target area according to building information, geometric space characterization data of each window, and visible field data.

[0203] A search module 705, configured to search for windows that match the multi-modal query information in the visible field database according to the multi-modal query information uploaded by a user, where the multi-modal query information is used to query windows with a specific outdoor view.

[0204] Optionally, the first determination module 702 is configured to:

[0205] Back-project the two-dimensional window frame corner points in the frame image onto a three-dimensional model to obtain a three-dimensional window frame contour in the image path, where the three-dimensional window frame contour is used to indicate the shape contour of the window in three-dimensional space;

[0206] Determine a window point cluster of a window in the point cloud path according to the dense point cloud of the target area, and determine geometric space characterization data of the window according to the window point cluster, where the window point cluster is used to indicate the point cloud data of multiple windows;

[0207] Determine the overlap degree of the three-dimensional window frame contour in the image path and the window point cluster in the point cloud path on a two-dimensional plane;

[0208] If the overlap degree is greater than a preset overlap degree threshold, determine that the three-dimensional window frame contour and the window point cluster are the same window, and use the geometric space characterization data in the point cloud path as the geometric space characterization data of the window;

[0209] Construct a window data set according to the geometric space characterization data of multiple windows.

[0210] Optionally, the first determination module 702 is configured to:

[0211] Perform semantic segmentation on the frame image through a semantic segmentation model to determine an initial window frame mask in the frame image;

[0212] Determine a final window frame mask of the window by tracking the same window in multiple consecutive frame images;

[0213] Extract two-dimensional window frame corner points from the final window frame mask;

[0214] Back-project the two-dimensional window frame corner points onto a three-dimensional model according to the camera parameters corresponding to the frame image to obtain a three-dimensional window frame contour of the window in the image path, where the camera parameters include shooting parameters and camera pose parameters.

[0215] Optionally, the first determination module 702 is configured to:

[0216] Repeat the following operations: Fit an initial plane based on a randomly selected part of the points in the window point cluster, and determine the distance from each point in the window point cluster to the initial plane; Determine the number of points whose distance is less than a preset distance threshold, where the initial plane is used to indicate a wall surface with multiple windows;

[0217] After reaching the set number of repeated operations, take the initial plane with the largest number of points as the target plane;

[0218] According to the distribution characteristics of the window point cluster in the target plane, determine the geometric space characterization data of each window.

[0219] Optionally, the first determination module 702 is used for:

[0220] Project the three-dimensional window frame contour under the image path onto a two-dimensional plane to obtain a polygon contour;

[0221] Project the window point cluster under the point cloud path onto a two-dimensional plane to obtain a projected point cluster;

[0222] Select the internal point cluster that falls within the polygon contour from the projected point cluster;

[0223] Calculate the overlap degree according to the area of the region formed by the internal point cluster and the area of the region formed by the polygon contour.

[0224] Optionally, the construction module 704 is used for:

[0225] Determine the frustum image of the visible field data, and construct a frustum database according to the frustum images of multiple windows;

[0226] Generate a visual vector according to the frustum image of the window, and construct a vector database according to the visual vectors of multiple windows;

[0227] Determine the building floor data where the window is located according to the preset building information, determine the visible environment index according to the visible field data, and construct an attribute database by combining the building floor data, geometric space characterization data, and visible environment index where multiple windows are located;

[0228] Construct a knowledge graph database, where the knowledge graph database takes entities, buildings, blocks, and windows as nodes, and takes the visible entities, the buildings where the windows are located, and the blocks where the windows are located as edges;

[0229] Among them, the frustum database, the vector database, the attribute database, and the knowledge graph database all include window identifiers.

[0230] Optionally, the search module 705 is used for:

[0231] Obtain the text to be recognized and the image to be recognized uploaded by the user, where the text to be recognized and the image to be recognized contain the outdoor landscape that the user expects to see;

[0232] Determine the image vector of the image to be recognized, and search for a first set of window identifiers that match the image vector in the vector database. Among them, if the image vector contains a landmark entity, add a landmark keyword to the text to be recognized.

[0233] Search for a second set of window identifiers that meet the text to be recognized in the first set of window identifiers in the attribute database.

[0234] Search for target nodes and target edges that meet the second set of window identifiers in the knowledge graph database, and construct a third set of window identifiers based on the target nodes and target edges.

[0235] Determine the cone image to be recognized of the image to be recognized, and search for a final set of window identifiers that match the cone image to be recognized in the third set of window identifiers in the cone database.

[0236] Determine all window identifiers in the final set of window identifiers.

[0237] As Figure 8 shown, an embodiment of the present application provides an electronic device, including a processor 801, a communication interface 802, a memory 803, and a communication bus 804. Among them, the processor 801, the communication interface 802, and the memory 803 complete mutual communication through the communication bus 804.

[0238] The memory 803 is used to store a computer program.

[0239] In an embodiment of the present application, when the processor 801 is used to execute the program stored on the memory 803, it implements the method for finding windows for a specific landscape provided by any one of the foregoing method embodiments.

[0240] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps of the method for finding windows for a specific landscape provided by any one of the foregoing method embodiments.

[0241] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0242] Through the descriptions of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the essence of the above technical solution, or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0243] It should be understood that the terms used herein are for the purpose of describing specific example embodiments only and are not intended to be limiting. Unless the context clearly indicates otherwise, as used herein, the singular forms "a", "an", and "the" may also include the plural forms. The terms "comprising", "including", "containing", and "having" are inclusive and thus specify the presence of the stated features, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or their combinations. The method steps, processes, and operations described herein are not to be construed as necessarily requiring them to be performed in the particular order described or illustrated, unless the order of execution is explicitly stated. It should also be understood that additional or alternative steps may be used.

[0244] The above are only specific embodiments of the present application, enabling those skilled in the art to understand or implement the present application. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to these embodiments shown herein, but rather will conform to the broadest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for finding windows for a specific landscape, characterized in that, The method includes: Obtaining building information and an image set with multiple perspectives in a target area, and constructing a 3D model of the target area based on the building information and the image set; Determining a window data set by analyzing frame images in the image set, where the window data set includes geometric space representation data of each window in the target area; Determining a viewing frustum of the window according to a preset field of view angle and an outward normal vector in the geometric space representation data, and determining visible area data of the window generated by projecting the viewing frustum onto the 3D model, where the visible area data is used to indicate the view outside the window; Constructing a visible area database of all windows in the target area according to the building information, the geometric space representation data of each window, and the visible area data; Searching for windows that match the multimodal query information in the visible area database according to the multimodal query information uploaded by the user, where the multimodal query information is used to query windows with a specific view outside the window; Among them, determining the window data set by analyzing frame images in the image set includes: Back-projecting two-dimensional window frame corner points in the frame image onto the 3D model to obtain a 3D window frame contour in the image path, where the 3D window frame contour is used to indicate the shape contour of the window in 3D space; Determining a window point cluster in the point cloud path according to the dense point cloud of the target area, and determining geometric space representation data of the window according to the window point cluster, where the window point cluster is used to indicate point cloud data of multiple windows; Determining the overlap degree between the 3D window frame contour in the image path and the window point cluster in the point cloud path on a two-dimensional plane; If the overlap degree is greater than a preset overlap degree threshold, determining that the 3D window frame contour and the window point cluster are the same window, and using the geometric space representation data in the point cloud path as the geometric space representation data of the window; Constructing the window data set according to the geometric space representation data of multiple windows; Among them, constructing the visible area database of all windows in the target area according to the building information, the geometric space representation data of each window, and the visible area data includes: Determining a viewing frustum image of the visible area data, and constructing a viewing frustum database according to the viewing frustum images of multiple windows; Generating a visual vector according to the viewing frustum image of the window, and constructing a vector database according to the visual vectors of multiple windows; Determining building floor data where the window is located according to preset building information, determining a visible environment index according to the visible area data, and constructing an attribute database by combining the building floor data where multiple windows are located, the geometric space representation data, and the visible environment index; Constructing a knowledge graph database, where the knowledge graph database uses entities, buildings, blocks, and windows as nodes, and uses visible entities, the buildings where the windows are located, and the blocks where the windows are located as edges; Among them, the viewing frustum database, the vector database, the attribute database, and the knowledge graph database all include window identifiers.

2. The method according to claim 1, characterized in that: Back-projecting the two-dimensional window frame corner points in the frame image onto the three-dimensional model to obtain the three-dimensional window frame contour in the image path includes: Performing semantic segmentation on the frame image through a semantic segmentation model to determine the initial window frame mask in the frame image; Determining the final window frame mask of the window by tracking the same window in multiple consecutive frame images; Extracting the two-dimensional window frame corner points from the final window frame mask; Back-projecting the two-dimensional window frame corner points onto the three-dimensional model according to the camera parameters corresponding to the frame image to obtain the three-dimensional window frame contour of the window in the image path, where the camera parameters include shooting parameters and camera pose parameters.

3. The method according to claim 1, wherein Determining the geometric spatial representation data of the window according to the window point cluster includes: Repeating the following operations: fitting an initial plane according to a part of the points randomly selected from the window point cluster, and determining the distance from each point in the window point cluster to the initial plane; determining the number of points with a distance less than a preset distance threshold, where the initial plane is used to indicate a wall surface with multiple windows; After reaching the set number of repeated operations, taking the initial plane with the largest number of points as the target plane; Determining the geometric spatial representation data of each window according to the distribution characteristics of the window point cluster in the target plane.

4. The method according to claim 1, characterized in that Determining the overlap degree between the three-dimensional window frame contour in the image path and the window point cluster in the point cloud path on a two-dimensional plane includes: Projecting the three-dimensional window frame contour in the image path onto a two-dimensional plane to obtain a polygon contour; Projecting the window point cluster in the point cloud path onto the two-dimensional plane to obtain a projected point cluster; Selecting the internal point cluster falling within the polygon contour from the projected point cluster; Calculating the overlap degree according to the area of the region formed by the internal point cluster and the area of the region formed by the polygon contour.

5. The method according to claim 1, characterized in that Searching for windows that match the multi-modal query information in the visual field database according to the multi-modal query information uploaded by the user includes: Obtaining the text to be recognized and the image to be recognized uploaded by the user, where the text to be recognized and the image to be recognized contain the outdoor landscape that the user expects to see; Determining the image vector of the image to be recognized and searching for the first set of window identifiers that match the image vector in the vector database, where if the image vector contains landmark entities, adding landmark keywords to the text to be recognized; Searching for the second set of window identifiers that meet the text to be recognized in the first set of window identifiers in the attribute database; Searching for the target nodes and target edges that meet the second set of window identifiers in the knowledge graph database, and constructing the third set of window identifiers according to the target nodes and the target edges; Determining the cone image to be recognized of the image to be recognized, and searching for the final set of window identifiers that match the cone image to be recognized in the third set of window identifiers in the cone database; Determining all the window identifiers in the final set of window identifiers.

6. A device for finding windows for a specific landscape, characterized in that, The device includes: An acquisition and construction module, configured to acquire the building information and the multi-view image set in the target area, and construct the three-dimensional model of the target area according to the building information and the image set; The first determination module is configured to determine a window data set by analyzing the frame images in the image set, where the window data set includes geometric space characterization data of each window in the target area; The second determination module is configured to determine the viewing frustum of the window according to a preset field of view angle and the normal vector in the geometric space characterization data, and determine the visible area data of the window generated by projecting the viewing frustum onto the three-dimensional model, where the visible area data is used to indicate the landscape outside the window; The construction module is configured to construct a visible area database of all the windows in the target area according to the building information, the geometric space characterization data of each window, and the visible area data; The search module is configured to search for windows that match the multimodal query information in the visible area database according to the multimodal query information uploaded by the user, where the multimodal query information is used to query windows with a specific landscape outside the window; Wherein, the first determination module is configured to: Back-project the two-dimensional window frame corner points in the frame image onto the three-dimensional model to obtain a three-dimensional window frame contour in the image path, where the three-dimensional window frame contour is used to indicate the shape contour of the window in three-dimensional space; Determine a window point cluster in the point cloud path according to the dense point cloud of the target area, and determine the geometric space characterization data of the window according to the window point cluster, where the window point cluster is used to indicate the point cloud data of multiple windows; Determine the overlap degree between the three-dimensional window frame contour in the image path and the window point cluster in the point cloud path on a two-dimensional plane; If the overlap degree is greater than a preset overlap degree threshold, determine that the three-dimensional window frame contour and the window point cluster are the same window, and use the geometric space characterization data in the point cloud path as the geometric space characterization data of the window; Construct the window data set according to the geometric space characterization data of multiple windows; Wherein, the second determination module is configured to: Determine the frustum image of the visible area data, and construct a frustum database according to the frustum images of multiple windows; Generate a visual vector according to the frustum image of the window, and construct a vector database according to the visual vectors of multiple windows; Determine the building floor data where the window is located according to the preset building information, determine the visible environment index according to the visible area data, and construct an attribute database by combining the building floor data where multiple windows are located, the geometric space characterization data, and the visible environment index; Construct a knowledge graph database, where the knowledge graph database uses entities, buildings, blocks, and windows as nodes, and uses the visible entities, the buildings where the windows are located, and the blocks where the windows are located as edges; Wherein, the frustum database, the vector database, the attribute database, and the knowledge graph database all include window identifiers.

7. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus. Among them, the processor, the communication interface, and the memory complete communication with each other through the communication bus; The memory is used to store a computer program; The processor is configured to implement the method according to any one of claims 1-5 when executing the program stored on the memory.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images