Searching method and device for oblique photography model, equipment, medium and program product

Abstract

The embodiment of the invention provides an oblique photography model searching method and device, equipment, a medium and a program product. The method comprises the steps of obtaining a search text, a first oblique photography model and a first point cloud model of the first oblique photography model; the first oblique photography model and the first point cloud model are monomerized; inputting the search text into an image-text conversion neural network for processing to obtain a search image; the image-text conversion neural network is obtained by training a preset first neural network based on a sample image in advance, and the sample image is generated by rendering at least one sample point cloud monomer model under at least one view angle; and matching at least one target point cloud monomer model matched with the search image from at least one point cloud monomer model in the first point cloud model, and determining the monomer model corresponding to the at least one target point cloud monomer model in the first oblique photography model as a target monomer model hit by the searched text.

Description

Technical Field

[0001] The present application relates to the field of computer graphics, more specifically to the field of computer three-dimensional model processing, and in particular to a method, device, equipment, medium and program product for searching oblique photography models. Background Art

[0002] With the development of science and technology, the concept of smart cities has emerged. Smart cities are modern urban forms that utilize next-generation information technologies such as the Internet of Things (IoT), big data, artificial intelligence (AI), 5G communications, and blockchain to digitally empower urban governance, public services, and industrial development. The goal is to optimize the allocation of urban resources, provide convenient and efficient living for residents, and achieve environmental sustainability.

[0003] In the construction of intelligent city system platforms based on digital twin technology, oblique photography is used to acquire multi-view images of the city. These images are then used to create oblique photogrammetry models. Oblique photogrammetry models recreate the real-world environment in the virtual world to a certain extent. Oblique photogrammetry models are three-dimensional geographic information models constructed using oblique photogrammetry. This technology utilizes platforms such as drones and aircraft equipped with multi-lens cameras to capture multi-angle images of surface objects from five directions: vertical, front, back, left, and right. Computer vision algorithms (such as Structure from Motion and Multi-View Stereo) are then used to convert the massive amount of image data into a three-dimensional model with realistic textures and geographic coordinates. The data structure of an oblique photogrammetry model is typically a multi-layered, continuous triangular facet structure. In other words, at a microscopic level, an oblique photogrammetry model is actually composed of a large number of interwoven triangular facets. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide a method, device, equipment, medium and program product for searching oblique photography models, which can, to a certain extent, realize text search for each individual model in the oblique photography model, such as searching for a building or a road in the oblique photography model of a city through text.

[0005] A first aspect of an embodiment of the present application provides a method for searching an oblique photography model, the method comprising: Acquire a search text, a first oblique photography model, and a first point cloud model of the first oblique photography model; the first oblique photography model and the first point cloud model are both singulated; Inputting the search text into a text-to-image conversion neural network for processing to obtain a search image; the text-to-image conversion neural network is obtained by pre-training a preset first neural network based on a sample image, and the sample image is generated by rendering at least one sample point cloud monomer model under at least one viewing angle; At least one target point cloud monomer model that matches the search image is matched from at least one point cloud monomer model in the first point cloud model, and the monomer model corresponding to each of the at least one target point cloud monomer model in the first oblique photography model is determined as the target monomer model hit by the search text.

[0006] A second aspect of an embodiment of the present application provides a device for searching an oblique photography model, the device comprising: A first acquisition module is configured to acquire a search text, a first oblique photography model, and a first point cloud model of the first oblique photography model; the first oblique photography model and the first point cloud model are both singulated; a first processing module configured to input the search text into a text-to-image conversion neural network for processing to obtain a search image; the text-to-image conversion neural network is obtained by pre-training a preset first neural network based on a sample image, wherein the sample image is generated by rendering at least one sample point cloud monomer model under at least one viewing angle; The first sampling module is used to match at least one target point cloud monomer model that matches the search image from at least one point cloud monomer model in the first point cloud model, and determine the monomer model corresponding to each of the at least one target point cloud monomer model in the first oblique photography model as the target monomer model hit by the search text.

[0007] A third aspect of an embodiment of the present application provides an electronic device, comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the search method for the oblique photography model as described in the first aspect are implemented.

[0008] A fourth aspect of an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method for searching the oblique photography model as described in the first aspect are implemented.

[0009] A fifth aspect of an embodiment of the present application provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the steps of the method for searching the oblique photography model as described in the first aspect.

[0010] A sixth aspect of the embodiments of the present application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the steps of the method for searching the oblique photography model as described in the first aspect.

[0011] In an embodiment of the present application, a search text is converted into a search image through a pre-trained image-text neural conversion network, and then a match is performed based on the search image in the individualized first point cloud model corresponding to the individualized first oblique photography model to be searched, and a target point cloud monomer model that matches the search image is matched from the first point cloud model, and then the monomer model corresponding to the target point cloud monomer model in the first oblique photography model is determined as the target monomer model hit by the search text, thereby realizing the search for monomer models in the oblique photography model through text. In other words, the embodiment of the present application provides a new interactive method for interacting with the oblique photography model, so that the user can search for monomer models in the oblique photography model by inputting text, which can enable the user to find the monomer model of the desired location from the oblique photography model more quickly and conveniently. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 1 is a flow chart of a method for searching an oblique photography model provided in an embodiment of the present application; Figure 2 This is a flowchart of some steps in the method for searching for an oblique photography model provided in an embodiment of the present application; Figure 3 This is a flowchart of some steps in the method for searching for an oblique photography model provided in an embodiment of the present application; Figure 4 This is a flowchart of some steps in the method for searching for an oblique photography model provided in an embodiment of the present application; Figure 5 This is a flowchart of some steps in the method for searching for an oblique photography model provided in an embodiment of the present application; Figure 6 This is a flowchart of some steps in the method for searching for an oblique photography model provided in an embodiment of the present application; Figure 7 This is a flowchart of some steps in the method for searching for an oblique photography model provided in an embodiment of the present application; Figure 8 Schematic diagram of the steps involved in training the individualized neural network and the image-text conversion neural network in the oblique photography model search method provided in an embodiment of the present application; Figure 9 This is a schematic structural diagram of a search device for an oblique photography model provided in an embodiment of the present application; Figure 10This is a schematic diagram of the hardware structure of an electronic device implementing an embodiment of the present application; Figure 11 This is another hardware structure diagram of an electronic device implementing an embodiment of the present application. DETAILED DESCRIPTION

[0013] The following will be combined with the accompanying drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this application.

[0014] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. In addition, the term "and / or" in the specification and claims refers to at least one of the connected objects, and the character " / " generally indicates that the objects connected are in an "or" relationship.

[0015] In the construction of digital twin-based smart city platforms, oblique photography technology is extensively used to capture urban landscapes and construct oblique photography models based on these multi-view images. Oblique photography models are typically constructed from multiple layers of continuous triangular faces. In oblique photography models, triangle meshes are the basic geometric units that form three-dimensional models. They connect three vertices in space to form triangles, and then stitch countless triangles together to form a continuous mesh surface, thereby achieving a three-dimensional geometric representation of real-world objects (such as buildings, terrain, and vegetation).

[0016] In the related art, if a user wants to find a certain single model in the oblique photography model, there is no related technology to help the user quickly locate the single model they want to find. Therefore, the user has to directly identify and judge with the naked eye whether the oblique photography model presented in the current window screen contains the single model they want to find. If not, the user needs to move or switch the range of the oblique photography model presented in the window screen so that the other parts of the oblique photography model are presented in the current window screen. The user again judges whether the oblique photography model presented in the current window screen contains the single model they want to find. If not, the user needs to move or switch the range of the oblique photography model presented in the window screen again. This cycle repeats until the single model the user wants to find is found. This process is time-consuming and labor-intensive, and is not convenient for the user to quickly find the single model they want to find.

[0017] The present invention provides a method, device, equipment, storage medium, chip, and computer program product for searching an oblique photography model, which can effectively solve the above-mentioned technical problems. The following, in conjunction with the accompanying drawings, describes in detail the method, device, equipment, storage medium, chip, and computer program product for searching an oblique photography model provided by the present invention through specific embodiments and their application scenarios.

[0018] like Figure 1 As shown, Figure 1 This is a flow chart of the method for searching for oblique photography models provided in an embodiment of the present application. The method for searching for oblique photography models can be applied to electronic devices, which can be computers, smart phones, tablet computers, wearable smart devices, etc., or servers in distributed systems, cloud servers, intelligent cloud computing servers with artificial intelligence technology, or intelligent cloud hosts. It can also be applied to Figure 9 The search device of the oblique photography model shown, Figure 10 or Figure 11 For details about the electronic equipment shown, please refer to the relevant instructions in the following text. Figure 1 The method for searching the oblique photography model includes the following steps S11 to S13: S11 , obtaining a search text, a first oblique photography model, and a first point cloud model of the first oblique photography model.

[0019] The search text may be text used to search for the target single model in the first oblique photography model. For example, it may be the name of a street or a building manually input by a user and received or detected by the device executing the oblique photography model method from another electronic device to which the device is communicating.

[0020] The first oblique photography model is the oblique photography model to be searched, that is, the oblique photography model in which the target single model matching the search text is expected to be searched through the search text. In other words, the first oblique photography model has been singulated. Optionally, the first oblique photography model can be obtained by singulating the second oblique photography model in advance (for details, see Figure 5 、 Figure 6 and Figure 7 In the relevant description of the embodiment shown in the figure), the second oblique photography model can be obtained by pre-fusion processing based on multi-perspective (vertical perspective, front perspective, rear perspective, left perspective, right perspective) images of a city, and the multi-perspective images are obtained by pre-oblique photography of the city's terrain environment.

[0021] The first point cloud model is the point cloud model corresponding to the first oblique photography model. That is, the first point cloud model and the first oblique photography model reflect the topography and environment of the same area. The difference is that the first point cloud model is presented in the form of a point cloud, while the first oblique photography model is presented in the form of a mesh surface model in the form of triangular facets. The first point cloud model has also been singulated. Optionally, the first point cloud model can be obtained by singulation based on the second point cloud model in advance (see Figure 5 ), further optionally, the second point cloud model is obtained by back-calculating the second oblique photography model (see Figure 6 1 and 2).

[0022] In some optional embodiments, the first oblique photography model and the first point cloud model are pre-stored in a preset storage medium or storage device, and the device that executes the method of the oblique photography model can read the first oblique photography model and the first point cloud model from the preset storage medium or storage device to obtain the first oblique photography model.

[0023] S12, inputting the search text into a picture-text conversion neural network for processing to obtain a search image.

[0024] The image-text conversion neural network is pre-trained on sample images using a preset first neural network. The first neural network is a multimodal neural network (a term used to describe artificial intelligence models that can simultaneously process two or more different types of data, such as images, text, voice, video, and sensor signals, and achieve unified semantic understanding by learning cross-modal associations. Its core goal is to mimic the human brain's ability to integrate multi-sensory information, such as auditory, visual, and language, to solve complex real-world problems that cannot be addressed by single-modal data). For example, it can be a CLIP (Contrastive Language-Image Pre-training) model, which uses contrastive learning to perform self-supervised training on massive image-text pairs. This maps images and text into the same embedding space, ensuring that semantically related images and text generate similar vector representations, while unrelated images and text are further apart. This enables the model to understand semantics across modalities. Therefore, after the first neural network is trained as an image-text conversion neural network, the image-text conversion neural network can convert input images into corresponding text, and vice versa.

[0025] The sample image used to train the first neural network is generated by rendering at least one sample point cloud monomer model under at least one viewing angle (optionally, the at least one viewing angle includes at least one of a vertical viewing angle, a front viewing angle, a rear viewing angle, a left viewing angle, and a right viewing angle). Optionally, the sample point cloud monomer model can be extracted from the individualized sample point cloud model. Further, optionally, the sample point cloud model can be obtained by inverse calculation of the sample oblique photography model (see Figure 6 1 and 2).

[0026] After the search text is input into the pre-trained image-text conversion neural network for processing, the search image is obtained.

[0027] S13, matching at least one target point cloud monomer model that matches the search image from at least one point cloud monomer model in the first point cloud model, and determining the monomer model corresponding to each of the at least one target point cloud monomer model in the first oblique photography model as the target monomer model hit by the search text.

[0028] As previously described, the first point cloud model has been individualized, that is, it includes at least one individual point cloud model. After obtaining a search image, the search image is matched with the individual point cloud models in the first point cloud model to identify at least one target point cloud model from the first point cloud model that matches the search image.

[0029] In some optional embodiments, matching at least one target point cloud monomer model that matches the search image from at least one point cloud monomer model in the first point cloud model includes the following steps S131 to S132: Figure 2 As shown, a schematic diagram of the process of steps S131 to S132 is shown: S131, determining a second similarity between the search image and each point cloud monomer model in the at least one point cloud monomer model; The second similarity can measure the semantic similarity between the search image and the semantics of each point cloud model to a certain extent.

[0030] The search image and the at least one point cloud monomer model can be converted into spatial vectors in the same space, and the similarity between the spatial vector of the search image and the spatial vector corresponding to each point cloud monomer model is calculated to obtain a second similarity between the search image and each point cloud monomer model.

[0031] S132: Determine the point cloud monomer model corresponding to the second similarity whose second similarity is less than a preset second threshold as the target point cloud monomer model matching the search image.

[0032] The second threshold may be a preset hit criterion. For example, the point cloud monomer model corresponding to the second similarity less than the second threshold among all the obtained second similarities is determined as the target point cloud model matching the search image, that is, the target point cloud model hit by the search text.

[0033] In some optional embodiments, matching at least one target point cloud monomer model that matches the search image from at least one point cloud monomer model in the first point cloud model includes the following steps S133 to S135: Figure 3 As shown, a schematic flow chart of steps S133 to S135 is shown: S133: Determine a first similarity between the search image and the search text and a second similarity between the search image and each point cloud monomer model in the at least one point cloud monomer model.

[0034] The first similarity can measure the semantic similarity between the search image and the search text to a certain extent.

[0035] The search image and the search text may be converted into space vectors in the same space, and the similarity between the space vector of the search image and the space vector of the search text may be calculated to obtain a second similarity between the search images.

[0036] Regarding the process of determining the second similarity between the search image and the point cloud monomer model, please refer to the relevant description of step S131 above, which will not be repeated here.

[0037] S134: Determine the difference between the first similarity and each of the second similarities.

[0038] The difference can be used to measure the gap between the first similarity and the second similarity.

[0039] First, a difference operation is performed on the first similarity and the second similarity, and then an absolute value of the difference obtained by the difference operation is calculated to obtain the difference between the first similarity and the second similarity.

[0040] S135 , determining the point cloud monomer model corresponding to the second similarity whose difference is less than the preset first threshold as the target point cloud monomer model matching the search image.

[0041] The second threshold may be a preset hit criterion. For example, the point cloud monomer model corresponding to the second similarity less than the second threshold among all the obtained differences is determined as the target point cloud model matching the search image, that is, the target point cloud model hit by the search text.

[0042] After determining the target point cloud model hit by the search text from the first point cloud model, a target monomer model corresponding to the target point cloud model can be found in the first oblique photography model, thereby determining the target monomer model hit by the search text.

[0043] In some optional embodiments, determining the monomer model corresponding to each of the at least one target point cloud monomer model in the first oblique photography model as the target monomer model hit by the search text includes the following steps S136 to S137, such as Figure 4 As shown, a flow chart of steps S136 to S137 is shown: S136: Determine a first spatial position of each target point cloud monomer model in the first point cloud model.

[0044] S137: Determine the monomer models located at the respective first spatial positions in the first oblique photography model as target monomer models hit by the search text.

[0045] Based on the above, it can be known that each point cloud monomer model in the first point cloud model corresponds one-to-one to each monomer model in the first oblique photography model, and the spatial positions of the corresponding point cloud monomer models in the first point cloud model are the same as the spatial positions of the corresponding monomer models in the first oblique photography model. Therefore, after determining the target point cloud monomer model hit by the search text from the first point cloud model, the first spatial position of each target point cloud monomer model in the first point cloud model can be determined, and then the monomer model located at each first spatial position in the first oblique photography model is determined as the target monomer model hit by the search text. In this way, the monomer model searched by the search text is found in the first oblique photography model.

[0046] In an embodiment of the present application, a search text is converted into a search image through a pre-trained image-text neural conversion network, and then a match is performed based on the search image in the individualized first point cloud model corresponding to the individualized first oblique photography model to be searched, and a target point cloud monomer model that matches the search image is matched from the first point cloud model, and then the monomer model corresponding to the target point cloud monomer model in the first oblique photography model is determined as the target monomer model hit by the search text, thereby realizing the search for monomer models in the oblique photography model through text. In other words, the embodiment of the present application provides a new interactive method for interacting with the oblique photography model, so that the user can search for monomer models in the oblique photography model by inputting text, which can enable the user to find the monomer model of the desired location from the oblique photography model more quickly and conveniently.

[0047] In the related art, the acquired oblique photography model is usually not yet individualized and lacks the corresponding original multi-view images, which makes individualization processing very difficult. In traditional methods, the three-dimensional building base surface in the oblique photography model is usually manually determined, and then a local space surrounding the three-dimensional building is constructed based on the three-dimensional building base surface (similar to a bounding box). The oblique photography model is then segmented based on the local space to segment the three-dimensional building from the oblique photography model, that is, the three-dimensional building is segmented individually. However, this method of purely manual individualization not only consumes a lot of manpower costs, but also takes up a lot of time costs. In addition, during the segmentation process, the CPU is required to perform intensive and inefficient cropping, which consumes a lot of time and hardware overhead.

[0048] The embodiment of the present application proposes a new individual segmentation method for the above-mentioned oblique photography model that lacks original multi-view images, which is described below.

[0049] In some optional implementations, the step S11 of obtaining the first point cloud model of the first oblique photography model includes the following steps S111 to S113: Figure 5 As shown, a schematic diagram of the process of steps S111 to S113 is shown: S111: Acquire a second oblique photography model.

[0050] The second oblique photography model is a model of the first oblique photography model that has not been individualized. That is, the second oblique photography model and the first oblique photography model present the topographic environment of the same area, but the difference is that the second oblique photography model has not been individualized, while the first oblique photography model has been individualized.

[0051] S112, performing back calculation processing on the second oblique photography model to obtain a second point cloud model; Back-calculating the second oblique photography model, which has not yet been individualized, yields a corresponding second point cloud model. This means the second point cloud model and the second oblique photography model represent the same geomorphic environment. The difference is that the second point cloud model is presented as a point cloud, while the second oblique photography model is presented as a mesh surface model in the form of triangular facets. The second point cloud model has also not yet been individualized.

[0052] Optionally, step S112 includes the following steps S1121 to S1123, such as Figure 6 As shown, a schematic diagram of the process of steps S1121 to S1123 is shown: S1121, determining an average gradient value of each patch in the second oblique photography model and a geometric complexity corresponding to each patch; wherein the geometric complexity corresponding to any patch is the arithmetic mean of the angles between the normals of adjacent patches adjacent to the patch and the normal of the patch itself; After obtaining the second oblique photography model, first, the three-dimensional spatial coordinates (X, Y, Z), normal vector, UV coordinates (UV coordinates belong to the two-dimensional texture coordinate system, and their main function is to correspond the three-dimensional model surface and the two-dimensional texture image. The UV coordinates have nothing to do with the three-dimensional spatial coordinates of the model. It is an independent coordinate system set specifically for texture mapping.), texture resources, etc. of the second oblique photography model are extracted according to the geometric information of the second oblique photography model.

[0053] Then, gradient calculation is performed on the texture image of each (triangular) patch in the second oblique photography model to obtain a gradient distribution map corresponding to each patch in the second oblique photography model. Optionally, the OpenCV library can be used to perform gradient calculation on the texture image. Of course, other methods can also be used to perform gradient calculation on the texture image, not limited to the OpenCV library, such as the Pillow library, the Imageio library, etc.

[0054] Next, according to the UV coordinates of the three vertices of each (triangular) patch, the gradient value on the gradient distribution map corresponding to each patch is obtained, and the average gradient value corresponding to each patch is calculated. .

[0055] Next, we need to calculate the geometric complexity of each facet in the second oblique photography model. As mentioned above, the oblique photography model is a mesh surface model in the form of triangular facets, and of course the second oblique photography model is no exception. Therefore, each facet in the second oblique photography model has adjacent faces, and each facet has its own normal (a line perpendicular to the triangular facet). For each facet in the second oblique photography model , determine the patch Adjacent adjacent patches The normal of the patch The arithmetic mean of the angles between its own normals , the value That means the dough The corresponding geometric complexity. The larger the value, the higher the geometric complexity. The smaller it is, the lower the geometric complexity.

[0056] S1122 : Determine a sampling weight for each facet in the second oblique photography model according to an average gradient value of each facet, the geometric complexity corresponding to each facet, and a preset weight determination rule.

[0057] The weight determination rule is determined by the following formula:

[0058] in, Represents a normalization operation, using the maximum and minimum ranges for normalization. , Converted to the range of 0 to 1. 、 is the adjustment coefficient (default 、 Of course, you can also adjust it according to actual needs 、 The value of can be flexibly adjusted).

[0059] The above formula is used to determine the sampling weight of each patch in the second oblique photography model .

[0060] S1123: Sample each facet in the second oblique photography model according to the sampling weight of each facet and a preset sampling rule, and determine the vertices corresponding to each facet based on the sampling points sampled from each facet, to obtain a second point cloud model composed of the sampling points and vertices.

[0061] The sampling rule is determined by the following formula:

[0062] in, Indicates rounding of floating point numbers. The default is 50, of course The specific value can also be flexibly adjusted according to actual needs.

[0063] The sampling weights corresponding to each patch determined in step S1122 are Substitute the sampling rule formula to obtain the number of sampling points corresponding to each patch.

[0064] Then, the blue noise sampling point method is used to sample each patch in the second oblique photography model, and the UV coordinates of each sampling point in the patch to which it belongs are determined. , based on the patch The sampling points sampled internally and the barycentric coordinate interpolation method are used to determine the patch. The UV coordinates of each vertex are obtained to determine the color value of each vertex. The second point cloud model is composed of these sampling points sampled from the surface and the vertices of each surface.

[0065] S113: Input the second point cloud model into the monomeric neural network for processing to obtain a first point cloud model.

[0066] The monomerized neural network is obtained by pre-training a preset second neural network based on a sample point cloud model and the monomer model label corresponding to the sample point cloud model (please refer to the relevant description below for details). The second neural network can be any one of the neural networks that can be used to process three-dimensional point cloud data, such as Mask3D, SoftGroup, PointGroup, etc. For the second neural network, a large number of sample point cloud models are used to train the second neural network, and the second neural network is supervised based on the monomer model labels pre-labeled in the sample point cloud model, so that it acquires the ability to perform monomer segmentation on the point cloud model, that is, to obtain a monomerized neural network. After obtaining the monomerized neural network, the second point cloud model is input into the monomerized neural network model for processing to obtain the first point cloud model.

[0067] In some optional implementations, after obtaining the individualized first point cloud model, the second oblique photography model may be individualized according to the first point cloud model.

[0068] In some optional implementations, the step S11 of obtaining the first oblique photography model includes the following steps S114 to S115: Figure 7 As shown, a schematic flow chart of steps S114 to S115 is shown: S114: Determine spatial position information of each point cloud monomer model in the first point cloud model.

[0069] S115 , segmenting the monomer models located at the same position in the second oblique photography model according to the spatial position information of each point cloud monomer model in the at least one point cloud monomer model in the first point cloud model to obtain the first oblique photography model.

[0070] As mentioned above, the first point cloud model, the second point cloud model, and the second oblique photography model all reflect the topographic environment of the same area. That is to say, the same single object (such as a building, a road) is in the same spatial position in the first point cloud model, the second point cloud model, and the second oblique photography model. Therefore, after obtaining the first point cloud model that has been individualized, the spatial position information of each point cloud single model in the first point cloud model (the so-called point cloud single model is the local point cloud model corresponding to the physical entity in the overall point cloud model) can be determined. Then, based on the spatial position information of each point cloud single model in the first point cloud model, it is positioned in the second oblique photography model, and the single models at the same position in the second oblique photography model are segmented to obtain the first oblique photography model. For example, it can be based on the point cloud single model Spatial position information in the first point cloud model , the spatial position information in the first oblique photography model Monolithic model Cutting is performed to achieve the monomer model Monomer cutting.

[0071] In some optional implementations, the method for searching for an oblique photography model provided in the embodiments of the present application further includes the following steps: Attributes are linked to the individual models in the first oblique photography model to inject semantics into the individual models in the first oblique photography model.

[0072] Attribute attachment refers to the process of associating attributes (data, parameters, features) with specific objects, components, or entities. "Attributes" here can be understood as information describing the characteristics of a single model, such as name, color, size, and functional parameters. In the digital twin city platform, this can involve associating the spatial location information of geographic entities (such as roads, buildings, and rivers) with attribute data (such as name, length, material, and population data).

[0073] After the second oblique photography model is segmented into individual units to obtain the first oblique photography model, attributes of the individual units in the first oblique photography model can be linked, thereby achieving semantic injection of the individual units in the first oblique photography model.

[0074] As mentioned above, two neural networks are involved in the embodiment of this application: a single neural network and a graph-text conversion neural network. The following describes their training process: like Figure 8 As shown, Figure 8 Schematic diagram of the steps involved in training the individual neural network and the image-text conversion neural network in the tilt photography model search method provided in the embodiment of the present application. Figure 8 As shown, the following steps S21 to S28 are included: S21, obtaining a sample oblique photography model.

[0075] The sample oblique photography model is a pre-prepared oblique photography model for training the first neural network and the second neural network. Of course, in order to ensure that the trained image-text conversion neural network and the individualized neural network meet the requirements, a sufficient number of sample oblique photography models should be selected.

[0076] The sample oblique photography model can be stored in a preset storage medium or storage device. When training the first neural network and the second neural network, the device that executes the oblique photography model search method provided in the embodiment of the present application can read the sample oblique photography model from the storage medium or storage device to obtain the sample oblique photography model.

[0077] S22, performing back calculation processing on the sample oblique photography model to obtain a sample point cloud model.

[0078] The backcalculation process for the sample oblique photography model is similar to that for the second oblique photography model. Based on the aforementioned description of the backcalculation of the second oblique photography model, those skilled in the art will readily understand the backcalculation of the sample oblique photography model. Therefore, this backcalculation will not be specifically explained here. Accordingly, after backcalculating the sample oblique photography model, a sample point cloud model is obtained.

[0079] S23, obtaining a single model label corresponding to the sample point cloud model.

[0080] Individual model labels are pre-labeled individual point cloud models from the sample point cloud model. They are used as a reference for adjusting the parameters of the second neural network (such as Mask3D) during the subsequent supervised training of the second neural network.

[0081] S24, training the second neural network based on the sample point cloud model and the monomer model label until a preset number of rounds is reached or the second neural network meets preset requirements, thereby obtaining the monomerized neural network.

[0082] During the training process, in each round, the sample point cloud model is input into the second neural network for processing. The processing results output by the second neural network are compared with the corresponding individual model labels, and the parameters of the second neural network are adjusted based on the comparison results. This cycle is repeated until the second neural network iterates to a preset number of rounds or the second neural network meets the preset requirements (for example, the difference between the processing results of the second neural network for the sample point cloud model and the corresponding individual model labels is less than a preset threshold). This results in a single neural network, that is, a neural network capable of point cloud data recognition and individual segmentation.

[0083] S25, obtaining a monomerized sample point cloud model.

[0084] The sample point cloud model is a point cloud model used to train the first neural network.

[0085] In some optional implementations, after the individualized neural network is obtained, the sample point cloud model can be input into the individualized neural network for processing to obtain a individualized sample point cloud model.

[0086] S26 , extracting at least one sample point cloud monomer model from the monomerized sample point cloud models.

[0087] After obtaining the monomerized sample point cloud model, the sample point cloud monomer model can be extracted therefrom.

[0088] S27: Render the at least one sample point cloud model under at least one viewing angle to obtain a sample image.

[0089] The extracted sample point cloud monomer model is rendered at multiple perspectives (i.e., vertical perspective, front perspective, rear perspective, left perspective, and right perspective) to obtain image patches at multiple perspectives, i.e., sample images.

[0090] S28, training the first neural network based on the sample image until a preset number of iterations is completed or the first neural network meets preset requirements, thereby obtaining the image-text conversion neural network.

[0091] During each training round, a sample image is fed into a first neural network (e.g., a CLIP model) for processing, and the parameters of the first neural network are adjusted based on the processing results output by the first neural network. This cycle repeats until the first neural network reaches a preset number of rounds or meets preset requirements (e.g., the matching accuracy between the processing results of the first neural network on the sample image and the semantics of the sample image reaches a preset threshold), resulting in an image-to-text conversion neural network.

[0092] The method for searching for oblique photography models provided in the embodiment of the present application can be performed by a device for searching for oblique photography models. The method for searching for oblique photography models by the device for searching for oblique photography models is used as an example to illustrate the device for searching for oblique photography models provided in the embodiment of the present application.

[0093] like Figure 9 FIGURE 1 shows a schematic diagram of the structure of a search device for an oblique photography model provided by an embodiment of the present application. Figure 9 The tilt photography model search device 10 comprises: A first acquisition module 100 is configured to acquire a search text, a first oblique photography model, and a first point cloud model of the first oblique photography model; the first oblique photography model and the first point cloud model are both singulated; A first processing module 101 is configured to input the search text into a text-to-image conversion neural network for processing to obtain a search image; the text-to-image conversion neural network is obtained by pre-training a preset first neural network based on a sample image, wherein the sample image is generated by rendering at least one sample point cloud model at at least one viewing angle; The first sampling module 102 is used to match at least one target point cloud monomer model that matches the search image from at least one point cloud monomer model in the first point cloud model, and determine the monomer model corresponding to each of the at least one target point cloud monomer model in the first oblique photography model as the target monomer model hit by the search text.

[0094] Optionally, the first sampling module 102 includes: a first determining submodule, configured to determine a first similarity between the search image and the search text and a second similarity between the search image and each point cloud monomer model in the at least one point cloud monomer model; a second determining submodule, configured to determine a difference between the first similarity and each of the second similarities; The third determining submodule is configured to determine a point cloud monomer model corresponding to a second similarity whose difference is less than a preset first threshold as a target point cloud monomer model matching the search image.

[0095] Optionally, the first acquisition module 100 includes: A first acquisition submodule acquires a second oblique photography model; the second oblique photography model is an oblique photography model that has not been individualized from the first oblique photography model; A first inverse calculation submodule is configured to perform inverse calculation on the second oblique photography model to obtain a second point cloud model; The first processing submodule is used to input the second point cloud model into the individualized neural network for processing to obtain a first point cloud model; the individualized neural network is obtained by pre-training a preset second neural network based on a sample point cloud model and the individual model label corresponding to the sample point cloud model.

[0096] Optionally, the first inverse operator module includes: A first determining unit is configured to determine an average gradient value of each patch in the second oblique photography model and a geometric complexity corresponding to each patch; wherein the geometric complexity corresponding to any patch is the arithmetic mean of angles between normals of adjacent patches adjacent to the patch and the normal of the patch itself; a second determining unit, configured to determine a sampling weight for each patch in the second oblique photography model according to an average gradient value of each patch, a geometric complexity corresponding to each patch, and a preset weight determination rule; The first sampling unit is used to sample each facet in the second oblique photography model according to the sampling weight of each facet and a preset sampling rule, and determine the vertices corresponding to each facet based on the sampling points sampled from each facet, to obtain a second point cloud model composed of the sampling points and the vertices.

[0097] Optionally, the first acquisition module 100 includes: A first determining submodule, configured to determine spatial position information of each point cloud monomer model in the first point cloud model in the first point cloud model; The first segmentation submodule is used to segment the monomer models located at the same position in the second oblique photography model according to the spatial position information of each point cloud monomer model in the at least one point cloud monomer model in the first point cloud model to obtain the first oblique photography model.

[0098] Optionally, the image-text conversion neural network is trained through the following steps: Obtain the individualized sample point cloud model; Extracting at least one sample point cloud monomer model from the monomerized sample point cloud models; Rendering the at least one sample point cloud monomer model under at least one viewing angle to obtain a sample image; The first neural network is trained based on the sample image until a preset number of iterations are performed or the first neural network meets preset requirements, thereby obtaining the image-text conversion neural network.

[0099] Optionally, obtaining a singulated sample point cloud model includes: Obtaining a sample oblique photography model; Performing back-calculation processing on the sample oblique photography model to obtain a sample point cloud model; The sample point cloud model is input into the individualized neural network for processing to obtain a individualized sample point cloud model.

[0100] Optionally, the individualized neural network is trained by the following steps: Obtaining a single model label corresponding to the sample point cloud model; The second neural network is trained based on the sample point cloud model and the monomer model label until a preset number of rounds is reached or the second neural network meets preset requirements, thereby obtaining the monomerized neural network.

[0101] Optionally, the device 10 further includes: The first attachment module is used to attach attributes to each monomer model in the first oblique photography model, so as to perform semantic injection on each monomer model in the first oblique photography model.

[0102] Optionally, the first sampling module 102 includes: a third determining submodule, configured to determine a second similarity between the search image and each point cloud monomer model in the at least one point cloud monomer model; The fourth determining submodule is configured to determine the point cloud monomer model corresponding to the second similarity whose second similarity is less than a preset second threshold as the target point cloud monomer model matching the search image.

[0103] Optionally, the first sampling module 102 includes: a fifth determining submodule, configured to determine a first spatial position of each target point cloud monomer model in the first point cloud model; The sixth determining submodule is configured to determine the monomer models located at the respective first spatial positions in the first oblique photography model as target monomer models hit by the search text.

[0104] The tilted photography model search device 10 in the embodiments of the present application can be an electronic device or a component of an electronic device, such as an integrated circuit or chip. The electronic device can be a terminal or other device other than a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc., and the embodiments of the present application do not specifically limit this.

[0105] The tilt photography model search device 10 in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0106] The tilt photography model search device 10 provided in the embodiment of the present application can achieve Figures 1 to 8 To avoid repetition, the various processes implemented in the method embodiment are not described here.

[0107] In some optional embodiments, such as Figure 10 As shown, an embodiment of the present application also provides an electronic device 1300, including a processor 1301 and a memory 1302, wherein the memory 1302 stores a program or instruction that can be run on the processor 1301, and when the program or instruction is executed by the processor 1301, the various steps of the above-mentioned embodiment of the search method for the oblique photography model are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0108] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0109] Figure 11 A schematic diagram of the hardware structure of an electronic device implementing an embodiment of the present application.

[0110] The electronic device 170 includes, but is not limited to, components such as a radio frequency unit 1701, a network module 1702, an audio output unit 1703, an input unit 1704, a sensor 1705, a display unit 1706, a user input unit 1707, an interface unit 1708, a memory 1709, and a processor 17010. Those skilled in the art will appreciate that the electronic device 170 may also include a power supply (such as a battery) to power the various components. The power supply may be logically connected to the processor 17010 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. Figure 11 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be repeated here.

[0111] The processor 17010 is configured to: Acquire a search text, a first oblique photography model, and a first point cloud model of the first oblique photography model; the first oblique photography model and the first point cloud model are both singulated; Inputting the search text into a text-to-image conversion neural network for processing to obtain a search image; the text-to-image conversion neural network is obtained by pre-training a preset first neural network based on a sample image, and the sample image is generated by rendering at least one sample point cloud monomer model under at least one viewing angle; At least one target point cloud monomer model that matches the search image is matched from at least one point cloud monomer model in the first point cloud model, and the monomer model corresponding to each of the at least one target point cloud monomer model in the first oblique photography model is determined as the target monomer model hit by the search text.

[0112] It should be understood that in this embodiment of the present application, the input unit 1704 may include a graphics processing unit (GPU) 17041 and a microphone 17042. The GPU 17041 processes image data of still images or videos captured by an image capture device (e.g., a camera) in video capture mode or image capture mode. The display unit 1706 may include a display panel 17061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 1707 includes a touch panel 17071 or at least one of other input devices 17072. The touch panel 17071, also known as a touch screen, may include a touch detection device and a touch controller. Other input devices 17072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, on / off keys, etc.), a trackball, a mouse, and a joystick, which are not described in detail here.

[0113] Memory 1709 can be used to store software programs and various data. Memory 1709 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store an operating system, applications or instructions required for at least one function (such as sound playback or image playback), and the like. Furthermore, memory 1709 may include volatile memory or non-volatile memory, or both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM RAM (DRRAM). Memory 1709 in the embodiments of the present application includes, but is not limited to, these and any other suitable types of memory.

[0114] Processor 17010 may include one or more processing units. Optionally, processor 17010 integrates an application processor and a modem processor. The application processor primarily handles operations related to the operating system, user interface, and application programs, while the modem processor primarily processes wireless communication signals, such as a baseband processor. It is understood that the modem processor may not be integrated into processor 17010.

[0115] Any of the above-mentioned product embodiments can implement the various processes of the above-mentioned tilt photography model search method embodiment through its own processor, and can achieve the same technical effect. To avoid repetition, they will not be described one by one.

[0116] The embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned search method embodiment of the oblique photography model is implemented, and the same technical effect can be achieved. To avoid repetition, it is not described here. The processor is the processor in the electronic device or electronic system described in the above embodiment. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a disk or an optical disk, etc.

[0117] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned embodiment of the search method for the oblique photography model, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0118] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0119] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned search method embodiment for the oblique photography model, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0120] In the embodiments provided in the examples of the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0121] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of this embodiment.

[0122] In addition, each functional unit in each implementation of the embodiment of the present application may be integrated into a processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The above-mentioned integrated units may be implemented in the form of hardware or software functional units.

[0123] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program code, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0124] The above description is only an implementation method of the embodiment of the present application, and does not limit the patent scope of the embodiment of the present application. The above specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can make equivalent structural or equivalent process changes using the description and drawings of the embodiment of the present application, or directly or indirectly apply them in other related technical fields. Without departing from the scope of protection of the purpose of this application and the claims, many forms can be made, which are also included in the patent protection scope of the embodiment of the present application.

Claims

1. A method for searching an oblique photography model, characterized in that: The method comprises: Acquire a search text, a first oblique photography model, and a first point cloud model of the first oblique photography model; the first oblique photography model and the first point cloud model are both singulated; Inputting the search text into a text-to-image conversion neural network for processing to obtain a search image; the text-to-image conversion neural network is obtained by pre-training a preset first neural network based on a sample image, and the sample image is generated by rendering at least one sample point cloud monomer model under at least one viewing angle; At least one target point cloud monomer model that matches the search image is matched from at least one point cloud monomer model in the first point cloud model, and the monomer model corresponding to each of the at least one target point cloud monomer model in the first oblique photography model is determined as the target monomer model hit by the search text.

2. The method according to claim 1, characterized in that The matching of at least one target point cloud monomer model that matches the search image from at least one point cloud monomer model in the first point cloud model includes: Determining a first similarity between the search image and the search text and a second similarity between the search image and each point cloud monomer model in the at least one point cloud monomer model; determining a difference between the first similarity and each of the second similarities; The point cloud monomer model corresponding to the second similarity whose difference is less than the preset first threshold is determined as the target point cloud monomer model matching the search image.

3. The method according to claim 1, characterized in that The step of obtaining a first point cloud model of the first oblique photography model includes: Acquire a second oblique photography model; the second oblique photography model is an oblique photography model of the first oblique photography model that has not been individualized; performing back calculation on the second oblique photography model to obtain a second point cloud model; The second point cloud model is input into the individualized neural network for processing to obtain a first point cloud model; the individualized neural network is obtained by pre-training a preset second neural network based on the sample point cloud model and the individual model label corresponding to the sample point cloud model.

4. The method according to claim 3, characterized in that The back-calculating the second oblique photography model to obtain a second point cloud model includes: Determining the average gradient value of each patch in the second oblique photography model and the geometric complexity corresponding to each patch; wherein the geometric complexity corresponding to any patch is the arithmetic mean of the angles between the normals of adjacent patches and the normal of the patch itself; Determining a sampling weight for each facet in the second oblique photography model according to an average gradient value of each facet, a geometric complexity corresponding to each facet, and a preset weight determination rule; According to the sampling weight of each facet and the preset sampling rules, each facet in the second oblique photography model is sampled, and the vertices corresponding to each facet are determined based on the sampling points sampled from each facet, thereby obtaining a second point cloud model composed of the sampling points and vertices.

5. The method according to claim 3, characterized in that The obtaining of the first oblique photography model includes: Determining spatial position information of each point cloud monomer model in the first point cloud model; According to the spatial position information of each point cloud monomer model in the at least one point cloud monomer model in the first point cloud model, the monomer models located at the same position in the second oblique photography model are segmented to obtain the first oblique photography model.

6. The method according to claim 3, characterized in that The image-text conversion neural network is trained by the following steps: Obtain the individualized sample point cloud model; Extracting at least one sample point cloud monomer model from the monomerized sample point cloud models; Rendering the at least one sample point cloud monomer model under at least one viewing angle to obtain a sample image; The first neural network is trained based on the sample image until a preset number of iterations are performed or the first neural network meets preset requirements, thereby obtaining the image-text conversion neural network.

7. The method according to claim 6, characterized in that The obtaining of the individualized sample point cloud model includes: Obtaining a sample oblique photography model; Performing back-calculation processing on the sample oblique photography model to obtain a sample point cloud model; The sample point cloud model is input into the individualized neural network for processing to obtain a individualized sample point cloud model.

8. The method according to claim 7, characterized in that The individualized neural network is trained by the following steps: Obtaining a single model label corresponding to the sample point cloud model; The second neural network is trained based on the sample point cloud model and the monomer model label until a preset number of rounds is reached or the second neural network meets preset requirements, thereby obtaining the monomerized neural network.

9. The method according to claim 3, characterized in that The method further comprises: Attributes are linked to the individual models in the first oblique photography model to inject semantics into the individual models in the first oblique photography model.

10. The method according to claim 1, characterized in that The matching of at least one target point cloud monomer model that matches the search image from at least one point cloud monomer model in the first point cloud model includes: Determining a second similarity between the search image and each point cloud monomer model in the at least one point cloud monomer model; The point cloud monomer model corresponding to the second similarity that is less than a preset second threshold is determined as the target point cloud monomer model that matches the search image.

11. The method according to claim 1, wherein The determining the monomer model corresponding to each of the at least one target point cloud monomer models in the first oblique photography model as the target monomer model hit by the search text includes: Determining a first spatial position of each target point cloud monomer model in the first point cloud model; The monomer models located at each of the first spatial positions in the first oblique photography model are determined as target monomer models hit by the search text.

12. A search device for an oblique photography model, characterized in that: The device comprises: A first acquisition module is configured to acquire a search text, a first oblique photography model, and a first point cloud model of the first oblique photography model; the first oblique photography model and the first point cloud model are both singulated; a first processing module configured to input the search text into a text-to-image conversion neural network for processing to obtain a search image; the text-to-image conversion neural network is obtained by pre-training a preset first neural network based on a sample image, wherein the sample image is generated by rendering at least one sample point cloud monomer model under at least one viewing angle; The first sampling module is used to match at least one target point cloud monomer model that matches the search image from at least one point cloud monomer model in the first point cloud model, and determine the monomer model corresponding to each of the at least one target point cloud monomer model in the first oblique photography model as the target monomer model hit by the search text.

13. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the method for searching the oblique photography model according to any one of claims 1 to 11 are implemented.

14. A readable storage medium, characterized in that The readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the steps of the method for searching the oblique photography model according to any one of claims 1 to 11 are implemented.

15. A computer program product, characterized in that The computer program includes a program or an instruction, and when the program or the instruction is executed by the processor, the steps of the method for searching the oblique photography model according to any one of claims 1 to 11 are implemented.

Images