A CLIP-based house type drawing retrieval method and system, a computer device, and a storage medium
By using a CLIP-based method that combines text encoders and image encoders to learn the alignment relationship between floor plans and interior design cues, the problem of time-consuming and poor generalization ability in existing floor plan retrieval technologies is solved. This achieves efficient and accurate floor plan retrieval, and supports zero-shot learning and multi-shot adaptation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING NANSHE TECH CO LTD
- Filing Date
- 2024-12-13
- Publication Date
- 2026-06-16
Smart Images
Figure CN122220552A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image retrieval technology, specifically a CLIP-based floor plan retrieval method, system, computer equipment, and storage medium. Background Technology
[0002] When performing image retrieval, both manual visual inspection and existing image retrieval technologies have obvious defects, which include: (1) manual visual inspection is time-consuming and labor-intensive; (2) the K-Means machine learning clustering algorithm based on a limited amount of image data has very low universality; (3) the clustering results of this algorithm will no longer be accurate for new data after the image dataset is expanded; (4) every time a batch of data with large differences in data structure distribution is replaced, it is necessary to retrain once, resulting in poor generalization ability.
[0003] Currently, there are few methods for retrieving floor plans. Existing methods are mostly based on traditional machine learning, such as image retrieval based on the Turning function and the K-Means algorithm. This method has significant drawbacks, including:
[0004] The problems with the Turning function include: (1) First, the outer contour of the image needs to be extracted and segmented. For noisy images, the error is very large when the outer contour is segmented; (2) The angle of each corner is calculated based on the previous step, and the error is greatly affected by the previous step; (3) It depends on the reasonable setting of the threshold T_similar, otherwise it will greatly affect the search results; (4) The similarity calculation method, such as L1, L2, Euclidean distance, will also affect the search results.
[0005] Problems with K-means include: the K-means clustering algorithm is limited by the number of cluster centers and is highly dependent on the user-defined cluster center data, which can easily lead to inaccurate search results. Floor plans are mostly irregular in structure, diverse in category, and vary in structure depending on the geographical location, resulting in non-fixed cluster centers and making it difficult to guarantee the accuracy of search results; the method of centroid calculation also affects search results; and the number of iterations also affects the accuracy of search results.
[0006] Traditional retrieval methods have very low universality, which seriously affects the retrieval results. Every time the database is updated, the model needs to be retrained and all parameters need to be readjusted. Summary of the Invention
[0007] Purpose of the invention: To address the problem that existing image retrieval methods are not suitable for floor plan retrieval, this invention proposes a floor plan retrieval method, system, computer equipment, and storage medium based on CLIP.
[0008] Technical solution: A CLIP-based floor plan retrieval method, comprising the following steps:
[0009] Step 1: Obtain the JSON data of the outer outline of the floor plan and the internal floor plan design prompts;
[0010] Step 2: Input the internal floor plan design prompts into the CLIP model to obtain a set of floor plans similar to the input internal floor plan design prompts;
[0011] Step 3: Using machine learning algorithms, cluster and retrieve floor plans that are similar to the outer outline JSON data of the floor plans from the set of floor plans that are similar to the input internal floor plan design prompts. Finally, obtain a set of floor plans that are similar to the outer outline of the floor plans and whose internal layout is similar to the input internal floor plan design prompts, thus completing the retrieval.
[0012] Furthermore, the CLIP model is trained according to the following steps:
[0013] Step 1: Construct a training dataset, which includes floor plan images of completed interior layouts and corresponding interior layout design prompts;
[0014] Step 2: Train the CLIP model using the training dataset to learn the alignment relationship between the floor plan image of the completed interior layout and the interior design prompts, and obtain the final usable CLIP model.
[0015] Furthermore, the CLIP model is trained using a training dataset to learn the alignment relationship between the floor plan image of the completed interior layout and the interior layout design prompts, resulting in the final usable CLIP model. Specific operations include:
[0016] The text encoder using the CLIP model encodes the internal apartment layout design prompts, converting them into high-dimensional vectors of the prompts.
[0017] The image encoder using the CLIP model encodes the floor plan image with the completed internal layout, converting it into a high-dimensional image vector;
[0018] By maximizing the similarity between the high-dimensional vectors of the image and the high-dimensional vectors of the prompt words through contrastive learning, the final usable CLIP model is obtained.
[0019] Furthermore, the floor plan outline JSON data includes vectorized JSON data categorized as doors, walls, windows, areas, and columns.
[0020] Furthermore, the vectorized JSON data categorized as doors, walls, and windows all include: a unique identifier, three-dimensional information (length, width, and height), and the coordinates of the starting point of the line in the top view of the floor plan.
[0021] The vectorized JSON data categorized as a region includes: a unique identifier, three-dimensional information (length, width, and height), and the coordinates of each inflection point in the top view of the floor plan.
[0022] This invention discloses a CLIP-based floor plan retrieval method, comprising the following steps:
[0023] The outer contour image data of the floor plan to be retrieved is input into the pre-trained CLIP model to obtain the retrieval results, which include: outer contour image data of multiple floor plans and their corresponding vectorized JSON data of the floor plans;
[0024] The pre-trained CLIP model is obtained according to the following steps:
[0025] S1: Construct a training dataset that corresponds one-to-one with the floor plan image data and the vectorized JSON data of the floor plan; the floor plan image data is the outer contour image data of the floor plan.
[0026] S2: Train the CLIP model using the training dataset to obtain a pre-trained CLIP model.
[0027] Furthermore, the step of inputting the outer contour image data of the floor plan to be retrieved into the pre-trained CLIP model to obtain the retrieval results can be replaced by the following steps:
[0028] The vectorized JSON data of the floor plan to be retrieved is input into the pre-trained CLIP model to obtain the retrieval results, which include: multiple vectorized JSON data of floor plans and their corresponding outer contour image data of the floor plans.
[0029] Furthermore, the floor plan vectorized JSON data includes vectorized JSON data categorized as doors, walls, windows, areas, and columns.
[0030] Furthermore, the outer contour image data of the floor plan is obtained by extracting the outer contour of the floor plan using an edge detection algorithm.
[0031] Furthermore, the vectorized JSON data categorized as doors, walls, and windows all include: a unique identifier, three-dimensional information (length, width, and height), and the coordinates of the starting point of the line in the top view of the floor plan.
[0032] The vectorized JSON data categorized as a region includes: a unique identifier, three-dimensional information (length, width, and height), and the coordinates of each inflection point in the top view of the floor plan.
[0033] Furthermore, the process of inputting the outer contour image data of the floor plan to be retrieved into the pre-trained CLIP model to obtain the retrieval results includes:
[0034] The outer contour image data of the floor plan to be retrieved is input into the pre-trained CLIP model;
[0035] The image encoder of the pre-trained CLIP model encodes the outer contour image data of the floor plan to be retrieved and calculates its high-dimensional image vector input_image;
[0036] Calculate the similarity between the high-dimensional vector of the image input_image and the high-dimensional vectors of all floor plan images in the training dataset. Sort the calculated similarities from largest to smallest and select the high-dimensional vectors of the outer contour images of the top N floor plans.
[0037] Output the outer contour image of the N floor plans, corresponding to the high-dimensional vector of the outer contour image, and the corresponding vectorized JSON data of the floor plans.
[0038] Furthermore, the process of inputting the vectorized JSON data of the floor plan to be retrieved into the pre-trained CLIP model to obtain the retrieval results includes:
[0039] The vectorized JSON data of the floor plan to be retrieved is input into the pre-trained CLIP model;
[0040] The pre-trained CLIP model's text encoder performs text encoding on the vectorized JSON data of the floor plan to be retrieved, and calculates its high-dimensional text vector input_text;
[0041] Calculate the similarity between the high-dimensional vector of the text input_text and the high-dimensional vectors of all floor plan vectorized JSON in the training dataset; sort the calculated similarities from largest to smallest, and select the top N high-dimensional vectors of floor plan vectorized JSON.
[0042] Output the vectorized JSON data of the N floor plan diagrams and the corresponding outer contour image data of the floor plan diagrams.
[0043] This invention discloses a CLIP-based floor plan retrieval system, comprising:
[0044] The input module is used to input the JSON data of the outer outline of the floor plan and the internal floor plan design prompts;
[0045] The CLIP model module is used to input internal floor plan design prompts into the CLIP model and obtain a set of floor plans similar to the input internal floor plan design prompts.
[0046] The machine learning module is used to use machine learning algorithms to cluster and retrieve floor plans that are similar to the outer outline JSON data of the floor plans from a set of floor plans that are similar to the input internal floor plan design prompts. Finally, it obtains a set of floor plans that are similar to the outer outline of the floor plans and whose internal layout is similar to the input internal floor plan design prompts, thus completing the retrieval.
[0047] The CLIP model is trained according to the following steps:
[0048] Step 1: Construct a training dataset, which includes floor plan images of completed interior layouts and corresponding interior layout design prompts;
[0049] Step 2: Train the CLIP model using the training dataset to learn the alignment relationship between the floor plan image of the completed interior layout and the interior design prompts, and obtain the final usable CLIP model.
[0050] This invention discloses a CLIP-based floor plan retrieval system, comprising:
[0051] The input module is used to obtain the outer contour image data of the floor plan to be retrieved, or to obtain the vectorized JSON data of the floor plan to be retrieved.
[0052] The CLIP model module is used to input the outer contour image data of the floor plan to be retrieved into a pre-trained CLIP model to obtain retrieval results. The retrieval results include: outer contour image data of multiple floor plans and their corresponding vectorized JSON data; or input the vectorized JSON data of the floor plan to be retrieved into a pre-trained CLIP model to obtain retrieval results. The retrieval results include: vectorized JSON data of multiple floor plans and their corresponding outer contour image data.
[0053] The output module is used to display the search results to the user;
[0054] The pre-trained CLIP model is obtained according to the following steps:
[0055] S1: Construct a training dataset that corresponds one-to-one with the floor plan image data and the vectorized JSON data of the floor plan; the floor plan image data is the outer contour image data of the floor plan.
[0056] S2: Train the CLIP model using the training dataset to obtain a pre-trained CLIP model.
[0057] This invention discloses a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of a CLIP-based floor plan retrieval method.
[0058] The present invention discloses a storage medium storing a floor plan retrieval program, which, when executed by at least one processor, implements the steps of a CLIP-based floor plan retrieval method.
[0059] Beneficial effects: Compared with the prior art, the present invention has the following advantages:
[0060] (1) The method of the present invention is based on CLIP for floor plan retrieval and has a good generalization ability;
[0061] (2) In the prediction stage, the CLIP model of the present invention generates predictions by calculating the cosine similarity between the floor plan text and the floor plan image vector. It is particularly suitable for zero-shot learning tasks and can make predictions without seeing new training examples of floor plan images or floor plan text.
[0062] (3) The method of the present invention has the advantages of simple construction process and no need to retrain the model. It supports the retrieval of the outer contour image of the floor plan. The retrieval results include both the floor plan image and its corresponding floor plan JSON text. It also supports the retrieval of the outer contour text of the floor plan. The retrieval results include both the floor plan image and its corresponding floor plan JSON text.
[0063] (4) The method of the present invention can retrieve floor plans that are similar to the outer contour of the input unfinished house through machine learning, and can also retrieve floor plans that are similar to the interior design described by the prompt words through CLIP model combined with prompt words. The final result satisfies the above two conditions, but the existing search can only realize the outer contour search. Attached Figure Description
[0064] Figure 1 Flowchart for pre-training the dataset;
[0065] Figure 2 This is a flowchart of a CLIP-based floor plan retrieval method proposed in Example 3;
[0066] Figure 3 The unfinished house floor plan is shown in the unfinished house .json file;
[0067] Figure 4 The search results are those obtained in Example 3. Detailed Implementation
[0068] To facilitate understanding of the technical solution of this invention, the relevant technical terms are now explained.
[0069] Tuning function: A function that transforms a polygon according to its interior angles and side lengths;
[0070] K-Means: Divide a feature matrix X of N samples into K non-overlapping clusters;
[0071] CLIP: Constrastive Language-Image Pre-training.
[0072] To make the objectives, technical solutions, and advantages of this invention clearer, the following description, in conjunction with the accompanying drawings, will further illustrate the CLIP-based floor plan retrieval method, system, computer equipment, and storage medium proposed in this invention.
[0073] Example 1:
[0074] This embodiment discloses a CLIP-based floor plan retrieval method for retrieving the outer contour image of a floor plan, which mainly includes the following steps:
[0075] Step 1: Construct the training dataset, which includes a floor plan image dataset and a floor plan vectorized JSON dataset;
[0076] Vectorized JSON data of floor plans in the training dataset, including but not limited to:
[0077] (1) Types of doors, including but not limited to: entrance doors, non-entrance doors, and doorways;
[0078] (2) Wall categories, including but not limited to: load-bearing walls, non-load-bearing walls, and tiled walls;
[0079] (3) Types of windows, including but not limited to: ordinary windows and bay windows;
[0080] (4) Category areas, including but not limited to: living room and dining room, master bedroom, guest bedroom, kitchen, bathroom, storage room, dressing room, children's room, elderly room, flue, and well;
[0081] (5) Categories of columns, including but not limited to: round columns, square columns, and irregularly shaped columns.
[0082] Each category includes: a unique ID for identification and three-dimensional information of length, width, and height (unit: mm); for categories such as doors, walls, and windows, the coordinates of the starting point of the center line in the top view should also be included; for categories such as regions, the coordinates of each inflection point in the top view should also be included; and for circular regions, a straight line should be used for segmented fitting.
[0083] Edge detection algorithms (such as Canny) are used to extract the outer contours of floor plan images in the training dataset;
[0084] The floor plan image data with extracted outer contours is matched one-to-one with the corresponding vectorized JSON data of the floor plan to complete the construction of the training dataset.
[0085] Step 2: Pre-train the CLIP model using the training dataset to learn the alignment between the floor plan image and the vectorized JSON data of the floor plan. For example... Figure 1 As shown, the specific operations include:
[0086] The CLIP model text encoder is used to encode the floor plan vectorized JSON data, converting it into a high-dimensional vector of floor plan vectorized JSON.
[0087] The floor plan image data is encoded using an image encoder based on the CLIP model, converting it into a high-dimensional vector of the floor plan image.
[0088] By maximizing the similarity between the high-dimensional vectors of the floor plan vectorized JSON and the high-dimensional vectors of the floor plan image through contrastive learning, a joint embedding space is learned. In this embedding space, similar pairs of high-dimensional vectors of the floor plan vectorized JSON and the high-dimensional vectors of the floor plan image are mapped to nearby positions, thus obtaining a well-trained CLIP model.
[0089] Step 3: Input the outer contour image of the floor plan to be retrieved into the trained CLIP model;
[0090] The CLIP model's image encoder encodes the outer contour image of the floor plan to be retrieved and calculates its high-dimensional vector input_image.
[0091] Calculate the similarity between the high-dimensional vector of the image input_image and the high-dimensional vectors of all floor plan images in the training dataset. Sort the calculated similarities from largest to smallest and select the top N high-dimensional vectors of the floor plan images.
[0092] Output the floor plan image corresponding to the high-dimensional vector of the N floor plan images and the corresponding vectorized JSON data of the floor plan, thus completing the retrieval of the outer contour image of the floor plan.
[0093] Example 2:
[0094] This embodiment proposes a CLIP-based floor plan retrieval method for retrieving vectorized JSON data of floor plan text, specifically including the following steps:
[0095] Step 1: Same as in Example 1, construct the training dataset;
[0096] Step 2: Same as in Example 1, obtain the trained CLIP model;
[0097] Step 3: Input the vectorized JSON data of the floor plan to be retrieved into the trained CLIP model;
[0098] The CLIP model's text encoder performs text encoding on the vectorized JSON data of the floor plan to be retrieved, and calculates its high-dimensional text vector input_text.
[0099] Calculate the similarity between the high-dimensional vector of the text input_text and the high-dimensional vectors of all floor plan vectorized JSON in the training dataset; sort the calculated similarities from largest to smallest, and select the top N high-dimensional vectors of floor plan vectorized JSON.
[0100] Output the vectorized JSON data of the N floor plan layouts and the corresponding floor plan images, thus completing the retrieval of the floor plan text vectorized JSON data.
[0101] Example 3:
[0102] like Figure 2 As shown, this embodiment proposes a CLIP-based floor plan retrieval method to retrieve the outer outline and internal floor plan design prompts of a floor plan. The specific operations include:
[0103] Step 1: Obtain the JSON data of the outer outline of the floor plan and the internal floor plan design prompts;
[0104] Step 2: Input the internal floor plan design prompts into the CLIP model to obtain a set of floor plans similar to the input internal floor plan design prompts; in this step, the outer contour of the floor plans is not constrained, and only the degree of similarity of the internal floor plan design is considered.
[0105] Step 3: Using machine learning methods, cluster the set of floor plans that are similar to the input internal floor plan design prompts to obtain a set of floor plans that are similar to the JSON data of the floor plan outline. This will result in a set of floor plans with similar outlines and internal layouts to the input internal floor plan design prompts, thus completing the retrieval.
[0106] The CLIP model used in this embodiment is trained according to the following steps:
[0107] Step 1: Construct a training dataset, which includes floor plan images of completed interior layouts and corresponding interior layout design prompts;
[0108] The floor plan image with the completed internal layout includes, but is not limited to:
[0109] Door categories, including but not limited to: entrance doors, non-entrance doors, and doorways;
[0110] Wall types, including but not limited to: load-bearing walls, non-load-bearing walls, and tiled walls;
[0111] Types of windows, including but not limited to: regular windows and bay windows;
[0112] Category areas include, but are not limited to: living room and dining room, master bedroom, guest bedroom, kitchen, bathroom, storage room, dressing room, children's room, elderly room, chimney, and well;
[0113] Columns can be categorized, including but not limited to: round columns, square columns, and irregularly shaped columns.
[0114] Each category includes: a unique ID for identification and three-dimensional information of length, width, and height (unit: mm); for categories such as doors, walls, and windows, the coordinates of the starting point of the center line in the top view should also be included; for categories such as regions, the coordinates of each inflection point in the top view should also be included; and for circular regions, a straight line should be used for segmented fitting.
[0115] For interior unit design prompts, descriptions should include categories and the positional or connection relationships between categories.
[0116] Step 2: Train the CLIP model using the training dataset to learn the alignment relationship between the floor plan image of the completed interior layout and the interior design prompts, resulting in the final usable CLIP model; specific operations include:
[0117] The text encoder using the CLIP model encodes the internal apartment layout design prompts, converting them into high-dimensional vectors of the prompts.
[0118] The image encoder using the CLIP model encodes the floor plan image with the completed internal layout, converting it into a high-dimensional image vector;
[0119] By maximizing the similarity between high-dimensional image vectors and high-dimensional cue word vectors through contrastive learning, a joint embedding space is learned in which similar pairs of high-dimensional image vectors and high-dimensional cue word vectors are mapped to nearby positions.
[0120] The machine learning methods used in this embodiment include: Tuning function calculation and K-Means clustering. Specific operations include:
[0121] First, train the model according to the following steps:
[0122] (1) Perform edge detection on all floor plans in the image training set to extract the outer contour of the floor plan; the outer contour includes: load-bearing walls, non-load-bearing walls, windows, doors, etc.; doors include: entrance door, non-entrance door, doorway;
[0123] (2) Calculate the Tuning function: Divide the extracted floor plan outline into segments according to the segmentation rules. The segmentation rule is to divide the line into segments if there is a corner. Calculate the included angle between two adjacent line segments. Calculate the polar coordinate vector formed by the corner and the line segment based on the segmented line segments and the included angle.
[0124] (3) Perform K-Means clustering on polar coordinate vectors: Input the number of cluster centers k; For each data point in the image training set, calculate its distance to k cluster centers and assign it to the class of the nearest cluster center; Calculate the average value of the data points in the class of each cluster center and use it as the new cluster center; Repeat this process until the cluster centers no longer change or the preset stopping condition is met, and finally divide the image training set into k different classes.
[0125] Secondly, inference is performed using the trained model, including:
[0126] (1) Input the outer outline image of the floor plan to be retrieved;
[0127] (2) Calculate the turning function and the distance to the cluster centers of the outer contour image of the floor plan to be retrieved;
[0128] (3) Select the nearest cluster center and its corresponding cluster;
[0129] (4) Output the top n results with the highest similarity as the outer contour similarity search results.
[0130] For example, the internal floor plan design prompt is: 1 master bedroom, 1 guest bedroom, 2 bathrooms, 1 kitchen; inputting this internal floor plan design prompt into the CLIP model will generate one or more floor plan sets similar to this internal floor plan design prompt.
[0131] Machine learning methods are used to cluster and retrieve data from a set of floor plans, such as... Figure 3 The set of floor plans similar to the "unfinished house.json" file shown is ultimately a set of floor plans with similar outer outlines, internal layouts, and prompts. Figure 4 The search results are shown.
[0132] Figure 4 The RGB identification of each component is shown in Table 1:
[0133] Table 1
[0134]
[0135]
[0136] In all the above embodiments, the following data processing methods can be used to augment the data when creating the training dataset, including:
[0137] Rotate the image data: Set a rotation angle every 15° and 25° to further increase the diversity of the data.
[0138] Image data cropping: If the maximum size of the original image is greater than 1024*1024 pixels, scale it down to 512*512 pixels, and then crop it into multiple 256*256 pixel image blocks. If the maximum size of the original image is less than 512*512 pixels, directly scale it down to 256*256 pixels.
[0139] Flip the image data, with values ranging from [90°, 180°, 270°];
[0140] The image data is color-transformed and normalized by dividing the pixel values of the image by 255 (for an 8-bit image) to convert it to floating-point numbers.
[0141] The CLIP model trained in this way has better accuracy.
[0142] In the above embodiments, adversarial sample generation methods based on GAN models can also be used to create or expand the training dataset, mainly including:
[0143] Create a clean dataset consisting of pairs of images of unfinished houses and floor plan images of completed interior layouts;
[0144] AdvGAN++ was used as the pre-trained model, and the pre-trained model was trained using a clean dataset.
[0145] One or more clean samples are input into a trained generator, which maps the clean samples to adversarial perturbations (i.e., minor modifications or noise). These adversarial perturbations are added to the original samples to generate adversarial samples.
[0146] The CLIP model is trained using adversarial examples to create a training dataset.
[0147] Example 4:
[0148] This embodiment proposes a CLIP-based floor plan retrieval system, which is used to execute the method provided in any of the above embodiments and has the corresponding functions and beneficial effects of the method execution.
[0149] Example 5
[0150] This embodiment provides a computer device for retrieving floor plans based on CLIP, and this embodiment provides services for implementing the method of any of the above embodiments. The computer device may include: a memory storing a computer-executable program, a processor coupled to the memory, and the processor calling the computer-executable program stored in the memory to execute the steps of the method described in Embodiment 1.
[0151] The memory may include computer system readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory. The device may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the memory may be used to read and write non-removable, non-volatile magnetic media (commonly referred to as a "hard disk drive"). A program / utility having a set (at least one) of program modules may be stored in, for example, memory. Such program modules include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. The computer-executable program of the program modules typically performs the functions and / or methods described in the embodiments of the present invention.
[0152] The code for performing the operations of the present invention can be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, and C++, as well as conventional procedural programming languages such as the "C" language or similar programming languages.
[0153] The processor executes various functional applications and data processing by running programs stored in memory, thereby implementing the method provided in Embodiment 1.
[0154] Example 6:
[0155] This invention provides a storage medium containing a computer-executable program, which, when executed by a computer processor, is used to perform the method of any of the above embodiments.
[0156] The storage medium of embodiments of the present invention may be any combination of one or more computer-readable media. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.
[0157] The code for a computer-executable program that performs the operations of this invention can be written in one or more programming languages or a combination thereof. Programming languages include object-oriented programming languages such as Java, Smalltalk, and C++, as well as conventional procedural programming languages such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0158] Of course, the computer-executable program provided in this embodiment is not limited to the above-described method operation, but can also perform related operations in the method provided in any embodiment of the present invention.
Claims
1. A CLIP-based floor plan retrieval method, characterized in that: Includes the following steps: Step 1: Obtain the JSON data of the outer outline of the floor plan and the internal floor plan design prompts; Step 2: Input the internal floor plan design prompts into the CLIP model to obtain a set of floor plans similar to the input internal floor plan design prompts; Step 3: Using machine learning algorithms, cluster and retrieve floor plans that are similar to the outer outline JSON data of the floor plans from the set of floor plans that are similar to the input internal floor plan design prompts. Finally, obtain a set of floor plans that are similar to the outer outline of the floor plans and whose internal layout is similar to the input internal floor plan design prompts, thus completing the retrieval.
2. The CLIP-based floor plan retrieval method according to claim 1, characterized in that: The CLIP model is trained according to the following steps: Step 1: Construct a training dataset, which includes floor plan images of completed interior layouts and corresponding interior layout design prompts; Step 2: Train the CLIP model using the training dataset to learn the alignment relationship between the floor plan image of the completed interior layout and the interior design prompts, and obtain the final usable CLIP model.
3. The CLIP-based floor plan retrieval method according to claim 1, characterized in that: The method involves training the CLIP model using a training dataset to learn the alignment relationship between floor plan images of completed interior layouts and interior design prompts, resulting in a usable CLIP model. Specific operations include: The text encoder using the CLIP model encodes the internal apartment layout design prompts, converting them into high-dimensional vectors of the prompts. The image encoder using the CLIP model encodes the floor plan image with the completed internal layout, converting it into a high-dimensional image vector; By maximizing the similarity between the high-dimensional vectors of the image and the high-dimensional vectors of the prompt words through contrastive learning, the final usable CLIP model is obtained.
4. The CLIP-based floor plan retrieval method according to claim 2, characterized in that: The floor plan outline JSON data includes vectorized JSON data categorized as doors, walls, windows, areas, and columns.
5. The CLIP-based floor plan retrieval method according to claim 4, characterized in that: The vectorized JSON data categorized as doors, walls, and windows all include: a unique identifier, three-dimensional information (length, width, and height), and the coordinates of the starting point of the line in the top view of the floor plan. The vectorized JSON data categorized as a region includes: a unique identifier, three-dimensional information (length, width, and height), and the coordinates of each inflection point in the top view of the floor plan.
6. A CLIP-based floor plan retrieval method, characterized in that: Includes the following steps: The outer contour image data of the floor plan to be retrieved is input into the pre-trained CLIP model to obtain the retrieval results, which include: outer contour image data of multiple floor plans and their corresponding vectorized JSON data of the floor plans; The pre-trained CLIP model is obtained according to the following steps: S1: Construct a training dataset that corresponds one-to-one with the floor plan image data and the vectorized JSON data of the floor plan; the floor plan image data is the outer contour image data of the floor plan. S2: Train the CLIP model using the training dataset to obtain a pre-trained CLIP model.
7. The CLIP-based floor plan retrieval method according to claim 6, characterized in that: The step of inputting the outer contour image data of the floor plan to be retrieved into a pre-trained CLIP model to obtain the retrieval results can be replaced by the following steps: The vectorized JSON data of the floor plan to be retrieved is input into the pre-trained CLIP model to obtain the retrieval results, which include: multiple vectorized JSON data of floor plans and their corresponding outer contour image data of the floor plans.
8. The CLIP-based floor plan retrieval method according to claim 7, characterized in that: The floor plan vectorized JSON data includes vectorized JSON data categorized as doors, walls, windows, areas, and columns.
9. A CLIP-based floor plan retrieval method according to claim 7, characterized in that: The outer contour image data of the floor plan is obtained by extracting the outer contour of the floor plan using an edge detection algorithm.
10. A CLIP-based floor plan retrieval method according to claim 8, characterized in that: The vectorized JSON data categorized as doors, walls, and windows all include: a unique identifier, three-dimensional information (length, width, and height), and the coordinates of the starting point of the line in the top view of the floor plan. The vectorized JSON data categorized as a region includes: a unique identifier, three-dimensional information (length, width, and height), and the coordinates of each inflection point in the top view of the floor plan.
11. A CLIP-based floor plan retrieval method according to claim 6, characterized in that: The process involves inputting the outer contour image data of the floor plan to be retrieved into a pre-trained CLIP model to obtain the retrieval results; the specific operations include: The outer contour image data of the floor plan to be retrieved is input into the pre-trained CLIP model; The image encoder of the pre-trained CLIP model encodes the outer contour image data of the floor plan to be retrieved and calculates its high-dimensional image vector input_image; Calculate the similarity between the high-dimensional vector of the image input_image and the high-dimensional vectors of all floor plan images in the training dataset. Sort the calculated similarities from largest to smallest and select the high-dimensional vectors of the outer contour images of the top N floor plans. Output the outer contour image of the N floor plans, corresponding to the high-dimensional vector of the outer contour image, and the corresponding vectorized JSON data of the floor plans.
12. The CLIP-based floor plan retrieval method according to claim 7, characterized in that: The process involves inputting the vectorized JSON data of the floor plan to be retrieved into a pre-trained CLIP model to obtain the retrieval results; the specific operations include: The vectorized JSON data of the floor plan to be retrieved is input into the pre-trained CLIP model; The pre-trained CLIP model's text encoder performs text encoding on the vectorized JSON data of the floor plan to be retrieved, and calculates its high-dimensional text vector input_text; Calculate the similarity between the high-dimensional vector of the text input_text and the high-dimensional vectors of all floor plan vectorized JSON in the training dataset; sort the calculated similarities from largest to smallest, and select the top N high-dimensional vectors of floor plan vectorized JSON. Output the vectorized JSON data of the N floor plan diagrams and the corresponding outer contour image data of the floor plan diagrams.
13. A CLIP-based floor plan retrieval system, characterized in that: include: The input module is used to input the JSON data of the outer outline of the floor plan and the internal floor plan design prompts; The CLIP model module is used to input internal floor plan design prompts into the CLIP model and obtain a set of floor plans similar to the input internal floor plan design prompts. The machine learning module is used to use machine learning algorithms to cluster and retrieve floor plans that are similar to the outer outline JSON data of the floor plans from a set of floor plans that are similar to the input internal floor plan design prompts. Finally, it obtains a set of floor plans that are similar to the outer outline of the floor plans and whose internal layout is similar to the input internal floor plan design prompts, thus completing the retrieval. The CLIP model is trained according to the following steps: Step 1: Construct a training dataset, which includes floor plan images of completed interior layouts and corresponding interior layout design prompts; Step 2: Train the CLIP model using the training dataset to learn the alignment relationship between the floor plan image of the completed interior layout and the interior design prompts, and obtain the final usable CLIP model.
14. A CLIP-based floor plan retrieval system, characterized in that: include: The input module is used to obtain the outer contour image data of the floor plan to be retrieved, or to obtain the vectorized JSON data of the floor plan to be retrieved. The CLIP model module is used to input the outer contour image data of the floor plan to be retrieved into a pre-trained CLIP model to obtain retrieval results. The retrieval results include: outer contour image data of multiple floor plans and their corresponding vectorized JSON data; or input the vectorized JSON data of the floor plan to be retrieved into a pre-trained CLIP model to obtain retrieval results. The retrieval results include: vectorized JSON data of multiple floor plans and their corresponding outer contour image data. The output module is used to display the search results to the user; The pre-trained CLIP model is obtained according to the following steps: S1: Construct a training dataset that corresponds one-to-one with the floor plan image data and the vectorized JSON data of the floor plan; the floor plan image data is the outer contour image data of the floor plan. S2: Train the CLIP model using the training dataset to obtain a pre-trained CLIP model.
15. A computer device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the CLIP-based floor plan retrieval method as described in any one of claims 1 to 5 or any one of claims 6 to 12.
16. A storage medium, characterized in that, The storage medium stores a floor plan retrieval program, which, when executed by at least one processor, implements the steps of a CLIP-based floor plan retrieval method as described in any one of claims 1 to 5 or any one of claims 6 to 12.