Model training method and device, electronic equipment and readable storage medium

Abstract

The invention discloses a model training method and device, electronic equipment and a readable storage medium. The method comprises the steps of obtaining a building facade picture sample, and layout characterization sample data and size characterization sample data of a building facade in the building facade picture sample; an image data reconstruction model is trained with the layout representation sample data as input until the image data reconstruction model converges, and the image data reconstruction model comprises a data coding sub-model and a data decoding sub-model; constructing an image data generation model based on the image coding model and the trained data decoding sub-model; and training the image data generation model based on the size representation sample data by taking the building facade picture sample as an input until the image data generation model converges. According to the method, a graph data generation model of layout information and size information of a building facade in a picture can be generated based on the building facade picture.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and specifically to a model training method, device, electronic device, and readable storage medium. Background Art

[0002] In game development, building facade configuration is a necessary functional module. When configuring the appearance of building facades in the game, you can refer to the appearance of building facades in the real world to increase the realism of the building in the game scene.

[0003] At present, when configuring the appearance of a building facade in a game, developers usually manually select various basic asset components and manually combine the selected basic asset components to achieve a building facade configuration that is similar or identical to the appearance of a building facade in the referenced real world. Among them, the basic asset components in the building facade include at least windows, doors, walls, etc. That is, the appearance of the building facade in the existing configuration game mainly relies on manual configuration by developers. However, the process of manually configuring the appearance of the building facade by developers is very cumbersome, requires a lot of time and resources, and has high costs and low efficiency, resulting in high development costs. Summary of the invention

[0004] The present application provides a model training method, device, electronic device and readable storage medium, which solves the problem of high cost and low efficiency in configuring building facade appearance, resulting in high development cost.

[0005] In a first aspect, an embodiment of the present application provides a model training method, the method comprising:

[0006] Acquire a building facade image sample, and layout representation sample data and size representation sample data of the building facade in the building facade image sample;

[0007] Taking the layout representation sample data as input, training an image data reconstruction model until the image data reconstruction model converges, wherein the image data reconstruction model includes a data encoding sub-model and a data decoding sub-model;

[0008] Construct an image data generation model based on the image encoding model and the trained data decoding sub-model;

[0009] The building facade image sample is used as input, and the image data generation model is trained based on the size representation sample data until the image data generation model converges.

[0010] In some embodiments, the training of the image data reconstruction model using the layout representation sample data as input includes:

[0011] Encoding the layout representation sample data through a data encoding sub-model to obtain a first continuous latent representation of the layout of the building facade in a latent space;

[0012] quantizing the first continuous latent representation to obtain a first discrete representation of the layout of the building facade;

[0013] Decoding the first discrete representation through a data decoding sub-model to obtain layout representation reconstruction data of the building facade;

[0014] Calculating a first loss function based on the layout representation sample data and the layout representation reconstruction data;

[0015] According to the first loss function, the parameters of the image data reconstruction model are adjusted.

[0016] In some embodiments, the method further comprises:

[0017] Inputting the layout representation sample data into the trained data encoding sub-model to obtain a second continuous latent representation of the layout representation sample data in a latent space;

[0018] The step of taking the building facade image sample as input and training the image data generation model based on the size representation sample data until the image data generation model converges includes:

[0019] Taking the building facade image sample as input, the image data generation model is trained based on the second continuous latent representation and the size representation sample data until the image data generation model converges.

[0020] In some embodiments, taking the building facade image sample as input and training the image data generation model based on the second continuous potential representation and the size representation sample data includes:

[0021] Inputting the building facade picture sample into the image coding model, and the image coding model outputs first intermediate feature data;

[0022] Encoding the first intermediate feature data to obtain a predicted continuous latent representation of the layout of the building facade in the building facade image sample in a latent space;

[0023] Calculating a second loss function based on the predicted continuous latent representation and the second continuous latent representation;

[0024] Pooling the first intermediate feature data to obtain predicted size data of the building facade in the building facade image sample;

[0025] Calculating a third loss function based on the predicted size data and the size characterization sample data;

[0026] Adjust the parameters of the image coding model according to the second loss function and the third loss function.

[0027] In some embodiments, the step of pooling the first intermediate feature data to obtain predicted size data of the building facade in the building facade image sample includes:

[0028] Performing horizontal pooling on the first intermediate feature data to obtain horizontal predicted size data of the building facade in the building facade image sample;

[0029] The first intermediate feature data is pooled longitudinally to obtain longitudinal predicted size data of the building facade in the building facade image sample.

[0030] In some embodiments, the sample data representing the layout of the building facade includes sample data representing component types of the building facade components included in the building facade, and sample data representing component arrangement of the building facade components included in the building facade.

[0031] In some embodiments, the component type sample characterization data of the building facade components included in the building facade include component type sample characterization data of each building facade component included in the building facade, and the component type sample characterization data of each building facade component include the relative position relationship of the building facade components on the building facade.

[0032] In some embodiments, the component arrangement sample characterization data of the building facade components included in the building facade include component arrangement sample characterization data of the building facade components in the horizontal direction and component arrangement sample characterization data of the building facade components in the vertical direction.

[0033] In some embodiments, the component arrangement of the building facade components includes a single arrangement and a repeated arrangement.

[0034] In some embodiments, the method further comprises:

[0035] For repeatedly arranged building facade components, data deduplication is performed in the component arrangement sample representation data and the component type sample representation data respectively.

[0036] In some embodiments, the building facade components include independent building facade components and combined building facade components.

[0037] In some embodiments, the step of constructing an image data generation model based on the image encoding model and the trained data decoding sub-model includes:

[0038] Freezing parameters of the data decoding sub-model in the image data reconstruction after training;

[0039] An image generation model is constructed based on the image encoding model and the data decoding sub-model with frozen parameters.

[0040] In a second aspect, an embodiment of the present application further provides a data processing method, the method comprising:

[0041] Get the first building facade image;

[0042] Extracting second intermediate feature data for the first building facade in the first building facade image;

[0043] encoding the second intermediate feature data to obtain a third continuous latent representation of the layout of the first building facade in a latent space;

[0044] quantizing the third continuous latent representation to obtain a second discrete representation of the layout of the first building facade;

[0045] Decoding the second discrete representation to obtain layout representation data of the first building facade;

[0046] Pooling the second intermediate feature data to obtain dimension representation data of the first building facade;

[0047] A second building facade image is generated according to the layout representation data of the first building facade and the size representation data of the first building facade.

[0048] In a third aspect, an embodiment of the present application further provides a model training device, the device comprising:

[0049] A first acquisition module is used to acquire a building facade image sample, and layout representation sample data and size representation sample data of the building facade in the building facade image sample;

[0050] A first training module, used for training an image data reconstruction model with the layout representation sample data as input until the image data reconstruction model converges, wherein the image data reconstruction model includes a data encoding sub-model and a data decoding sub-model;

[0051] A construction module, for constructing an image data generation model based on the image encoding model and the trained data decoding sub-model;

[0052] The second training module is used to take the building facade image sample as input and train the image data generation model based on the size representation sample data until the image data generation model converges.

[0053] In a fourth aspect, an embodiment of the present application further provides a data processing device, the device comprising:

[0054] A second acquisition module is used to acquire a first building facade image;

[0055] An extraction module, configured to extract second intermediate feature data for the first building facade in the first building facade image;

[0056] an encoding module, configured to encode the second intermediate feature data to obtain a third continuous latent representation of the layout of the first building facade in a latent space;

[0057] a quantization module, configured to quantize the third continuous potential representation to obtain a second discrete representation of the layout of the first building facade;

[0058] A decoding module, used to decode the second discrete representation to obtain layout representation data of the first building facade;

[0059] A pooling module, used for pooling the second intermediate feature data to obtain dimension representation data of the first building facade;

[0060] A generating module is used to generate a second building facade image according to the layout representation data of the first building facade and the size representation data of the first building facade.

[0061] In a fifth aspect, an embodiment of the present application further provides an electronic device, including:

[0062] Processor; and

[0063] The memory is used to store a data processing program. After the electronic device is powered on and the program is run by the processor, the method described in the first aspect or the second aspect is executed.

[0064] In a sixth aspect, an embodiment of the present application further provides a computer-readable storage medium storing a data processing program, which is executed by a processor to perform the method described in the first aspect or the second aspect.

[0065] Compared with the prior art, this application has the following advantages:

[0066] The embodiment of the present application provides a model training method, the method obtains a building facade picture sample, a layout representation sample data and a size representation sample data of the building facade in the building facade picture sample; takes the layout representation sample data as input, trains an image data reconstruction model until the image data reconstruction model converges, wherein the image data reconstruction model includes a data encoding sub-model and a data decoding sub-model; constructs an image data generation model based on the image encoding model and the trained data decoding sub-model; takes the building facade picture sample as input, and trains the image data generation model based on the size representation sample data until the image data generation model converges. The method can generate an image data generation model of the layout information and size information of the building facade in the building facade picture based on the building facade picture. The image data generation model can predict the layout and size of the building facade in the building facade picture, thereby reducing the cost of configuring the building facade appearance and improving the efficiency of configuring the building facade appearance. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 It is a flow chart of a model training method provided in an application embodiment;

[0068] Figure 2 It is a schematic diagram of a building facade picture provided in an embodiment of the present application;

[0069] Figure 3 is a schematic diagram of a building facade component area provided in an embodiment of the present application;

[0070] Figure 4 It is a schematic diagram of the multi-channel discrete layout arrangement data;

[0071] Figure 5 is a training example diagram of an image data reconstruction model provided in an embodiment of the present application;

[0072] Figure 6 is a schematic diagram of an image data generation model provided in an embodiment of the present application;

[0073] Figure 7 is a structural block diagram of a model training device provided in an embodiment of the present application;

[0074] Figure 8 is a structural block diagram of a data processing device provided in an embodiment of the present application;

[0075] Fig. 9 It is a structural block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0076] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0077] It should be noted that the terms "first", "second", "third", etc. in the claims, description and drawings of the present application are used to distinguish similar objects and are not used to describe a specific order or sequence. The data used in this way are interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including", "having" and their variants are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0078] In game development, building facade configuration is a necessary functional module. When configuring the appearance of building facades in the game, you can refer to the appearance of building facades in the real world to increase the realism of the building in the game scene.

[0079] At present, when configuring the appearance of a building facade in a game, developers usually manually select various basic asset components and manually combine the selected basic asset components to achieve a building facade configuration that is similar or identical to the appearance of a building facade in the referenced real world. Among them, the basic asset components in the building facade include at least windows, doors, walls, etc. That is, the appearance of the building facade in the existing configuration game mainly relies on manual configuration by developers. However, the process of manually configuring the appearance of the building facade by developers is very cumbersome, requires a lot of time and resources, and has high costs and low efficiency, resulting in high development costs.

[0080] In order to solve the above problems in the related art, the embodiment of the present application provides a model training method. The execution subject of the solution provided by the present application can be an electronic device, which can be a desktop computer, a laptop computer, a mobile device, a smart watch, a smart TV, a tablet computer, a server, etc., or other devices with data processing and display functions.

[0081] The model trained by the model training method provided in the present application can be used to predict the layout representation data and size representation data of the building facade in the reference building facade image to obtain the predicted layout representation data and size representation data. A second building facade image is generated based on the predicted layout representation data and size representation data.

[0082] Before describing the implementation methods of the present application in detail, relevant concepts are first introduced.

[0083] 1. Building facade configuration is a functional module in the game development process, which allows game artists to configure the appearance of buildings in the game by referring to building facades in the real world.

[0084] 2. Building facade components are the basic asset components used in the game building facade configuration system to build the appearance of buildings in the game, including walls, windows, bars and columns. Game artists configure the appearance of buildings in the game by selecting and combining these components.

[0085] 3. Vector Quantized Variational Autoencoder (VQ-VAE) is a generative model that combines variational autoencoder (VAE) and vector quantization technology. The encoder maps the input data to the latent space, and then the vector quantization layer converts the continuous latent representation into discrete codebook features. Finally, the decoder reconstructs the discrete representation into reconstructed data corresponding to the input data. VQ-VAE is mainly used to efficiently learn the representation of discrete data and can be widely used in image generation, computer vision and other fields.

[0086] 4. One-hot encoding, a technique for representing categorical data, where each category is represented by a binary vector whose length is equal to the total number of categories and has 1s in the positions corresponding to the category and 0s in the rest of the positions.

[0087] 5. Gradient descent algorithm, an optimization algorithm used to minimize the objective function, is used to find the parameter value that minimizes the loss function (or cost function). It is widely used in parameter optimization problems in machine learning and artificial intelligence. It is a classic first-order optimization iterative algorithm.

[0088] like Figure 1 As shown, it is a flow chart of a model training method provided in an embodiment of the present application, and the model training method includes the following steps S101 to S104.

[0089] S101: Acquire a building facade image sample, and layout representation sample data and size representation sample data of the building facade in the building facade image sample;

[0090] In the embodiment of the present application, the building facade image may be an image of the appearance of the building facade in the game that has been manually configured, or may be an image of the appearance of the building facade in the real world. Figure 2 As shown, it is a schematic diagram of a building facade picture provided in an embodiment of the present application. After obtaining the building facade picture samples, the layout representation data and the size representation data of the building facade in each building facade picture sample are obtained to obtain the layout representation sample data and the size representation sample data of the building facade. Among them, the layout representation data of the building facade is used to represent all elements contained on the external vertical surface of the building, and the positional relationship of all elements arranged on the building facade. The size representation data is used to represent the horizontal and vertical dimensions of the building facade, as well as the horizontal and vertical dimensions of each element.

[0091] In an optional embodiment, the sample data representing the layout of the building facade includes sample data representing component types of the building facade components included in the building facade, and sample data representing component arrangement of the building facade components included in the building facade.

[0092] In an embodiment of the present application, a building facade is composed of a plurality of building facade components. Each building facade image includes a building facade, and each building facade image includes at least a portion of the building facade components of the building facade. The layout representation sample data of the building facade includes component type sample representation data of the building facade components included in the building facade, and component arrangement sample representation data of the building facade components included in the building facade. The component type sample representation data and arrangement sample representation data of the building facade components included in the building facade can be represented in the following manner, specifically:

[0093] First, each building facade image is divided to obtain the building facade component area corresponding to each building facade component. Figure 3 Shown is a schematic diagram of a building facade component area provided in an embodiment of the present application. Figure 3 (a) Figure 2 (a) is a schematic diagram of the corresponding building facade component area after the building facade is divided based on all the building facade components included. Figure 3 (b) Figure 2 (b) is a schematic diagram of the corresponding building facade component area after the building facade is divided based on all the building facade components included.

[0094] A data structure in the form of cells is constructed based on the building facade component area, and each cell corresponds to a building facade component included in the building facade image. That is, the number of cells is consistent with the number of building facade components included in the building facade image.

[0095] Fill in each cell with the type of building facade component corresponding to each cell, as shown in Table 1. Figure 2 The cell form of the sample characterization data of the component types and arrangement patterns of the building facades in the building facade image in (a) is shown in Table 2. Figure 2 (b) Sample characterization data of component types and arrangement of building facades in the building facade image in the cell format of the sample characterization data:

[0096] Table 1 Figure 2 (a) Sample representation data of component types and arrangement of building facades in the building facade image. Cell format of sample representation data

[0097] E1 E2 E3 E2 E3 E2 E3 E5 E6 Q1 W1 Q2 W1 Q2 W1 Q2 W2 Q3 E1 E2 E3 E2 E3 E2 E3 E5 E6 Q1 W1 Q2 W1 Q2 W1 Q2 W2 Q3 …… …… …… …… …… …… …… …… ……

[0098] Table 2 Figure 2 (b) Sample representation data of component types and arrangement of building facades in the building facade image. Cell format of sample representation data

[0099]

[0100]

[0101] It should be noted that the building facade components included in the building facade can be determined based on the common components of existing buildings. At present, most buildings in real life can be obtained by combining the four building facade components of wall Q, window W, horizontal bar E, and column P. In Table 1 and Table 2, Q represents the wall, Q1 represents the first type of wall, Q2 represents the second type of wall, ..., Qn represents the nth type of wall. W represents the window, W1 represents the first type of window, W2 represents the second type of window, ..., Wn represents the nth type of window. P represents the column, P1 represents the first type of column, P2 represents the second type of column, ..., Pn represents the nth type of column. E represents the horizontal bar, E1 represents the first type of horizontal bar, E2 represents the second type of horizontal bar, ..., En represents the nth type of horizontal bar. Wherein, n is a positive integer.

[0102] The arrangement of each cell in Table 1 and the type of building facade components in each cell represent Figure 3 (a) shows the arrangement of the building facade components contained in the building facade and the building facade component type corresponding to each building facade component area.

[0103] The arrangement of each cell in Table 2 and the type of building facade components in each cell represent Figure 3 (b) shows the arrangement of the building facade components contained in the building facade and the building facade component type corresponding to each building facade component area.

[0104] Fill each cell with the size data of the building facade component corresponding to each cell, and obtain the size representation sample data of the building facade in the form of a cell, wherein the size data includes horizontal size data and vertical size data.

[0105] The size data of the building facade component corresponding to each cell can match the lateral size of the building facade component with the lateral size of the corresponding cell, and match the longitudinal size of the building facade component with the longitudinal size of the corresponding cell. That is, the lateral size and longitudinal size of each building facade component are represented by the lateral size of the corresponding cell. The size of the cell can be set based on the lateral size and longitudinal size of the building facade component corresponding to the cell. As shown in Table 1 and Table 2, the sizes of the cells are different, indicating that the sizes of the component types corresponding to the cells are different.

[0106] It should be noted that the transverse dimensions and longitudinal dimensions of different types of building facade components may be the same. The same type of building facade components may include multiple transverse dimensions and longitudinal dimensions.

[0107] The horizontal and vertical dimensions of each cell may be the same as those of the building facade component corresponding to the cell, or the horizontal and vertical dimensions of the building facade component may be scaled based on a preset ratio and set as the horizontal and vertical dimensions of the cell.

[0108] The dimension data of the building facade component corresponding to each cell may also be such that the horizontal dimension and the vertical dimension of the building facade component are directly filled in the cell corresponding to each building facade component, or the horizontal dimension and the vertical dimension of the building facade component are added in the cell corresponding to the building facade component by other means.

[0109] It should be noted that, in the cell-format data structures shown in Tables 1 and 2, each cell contains information including: the type of building facade component, the arrangement of the facades of the building facade component, and the horizontal and vertical dimensions of each building facade component.

[0110] That is, in the embodiment of the present application, the layout representation sample data and the size representation sample data of the building facade in the building facade image sample can be represented by a data structure in the form of a cell.

[0111] In an optional embodiment, the building facade components include independent building facade components and combined building facade components.

[0112] In the embodiment of the present application, the building facade component may be an independent building facade component. An independent building facade component is a building facade component that is not combined, and is a single building facade component. For example, it may be an independent wall, an independent window, an independent horizontal bar, an independent column, etc., or other types of independent building facade components.

[0113] A building facade component may also be a combination of two or more building facade components, such as a combination of a wall and a window, a combination of a wall, a window and a column, or a combination of a window, a column and a window.

[0114] It should be noted that the types of building facade components included in the combined building facade components and the combination method between the building facade components can be determined or adjusted based on actual needs.

[0115] In an optional embodiment, the component arrangement of the building facade components includes a single arrangement and a repeated arrangement.

[0116] In an embodiment of the present application, each building facade component may appear and be used only once in the horizontal direction, or may be continuously repeated or repetitively at intervals. Each building facade component may appear and be used only once in the vertical direction, or may be continuously repeated or repetitively at intervals. Herein, repeated arrangement means that the component types of multiple building facade components and the component arrangement methods of the building facade components are the same. The same component types of building facade components include the same types of building facade components and the same sizes of building facade components. Exemplarily, as shown in Table 1, in the first row of Table 1, E2 and E3 are repeatedly arranged at intervals, and E1, E5 and E6 are arranged separately.

[0117] In an optional embodiment, the method further includes:

[0118] For repeatedly arranged building facade components, data deduplication is performed in the component arrangement sample representation data and the component type sample representation data respectively.

[0119] In the embodiment of the present application, whether the building facade components are repeatedly arranged includes whether the building facade components are repeatedly arranged in the horizontal direction and whether the building facade components are repeatedly arranged in the vertical direction. When there are repeated arrangements of building facade components in the horizontal direction or the vertical direction, deduplication processing is performed on the repeated building facade components.

[0120] Deduplication processing is performed on repeatedly arranged building facade components, and the minimum repeated arrangement unit of the repeated arrangement of the building facade components can be determined. For example, in the sample characterization data of the component type of the building facade in the form of cells and the sample characterization data of the arrangement method shown in Table 1, the building facade components are repeatedly arranged from the second cell to the seventh cell in the first row, and the minimum repeated arrangement unit of the repeated arrangement of the building facade components is a combination of E2 and E3. The building facade components are repeatedly arranged from the first cell to the fourth cell in the first column, and the minimum repeated arrangement unit of the repeated arrangement of the building facade components is a combination of E2 and Q1. The method for determining whether there is a repeated arrangement and the minimum repeated arrangement unit in the remaining cells is the same as above.

[0121] Deduplication of data is performed in the component arrangement sample characterization data and the component type sample characterization data, and only one minimum repeated arrangement unit is retained, and the remaining repeated arrangement units are deleted. Exemplarily, the component type sample characterization data and the arrangement sample characterization data of the building facade in the form of cells shown in Table 1 are deduplicated, including: in the horizontal direction, the building facade components from the second cell to the seventh cell of the first row are repeatedly arranged, and the minimum repeated arrangement unit is the combination of E2 and E3. The building facade components from the second cell to the seventh cell of the second row are repeatedly arranged, and the minimum repeated arrangement unit is the combination of W1 and Q2. In the vertical direction, in the second to seventh columns, the building facade components are repeatedly arranged in the first cell to the fourth cell, and the minimum repeated arrangement unit in the first column is the combination of E2 and W1, and the minimum repeated arrangement unit in the second column is the combination of E3 and Q2. Deduplication is performed on the component type sample characterization data and the arrangement sample characterization data of the building facade in the form of cells shown in Table 1, and the deduplication results are shown in Table 3:

[0122] Table 3 Component types and arrangement of sample characterization data of building facades in cell form in Table 1 after deduplication of sample characterization data

[0123] E1 E2 E3 E5 E6 Q1 W1 Q2 W2 Q3 …… …… …… …… ……

[0124] It should be noted that Table 3 assumes that in Table 1, in the horizontal direction, the repetitive arrangement of the building facade components from the second to the seventh columns is the same as that of the first four rows.

[0125] In an optional embodiment, the component type sample characterization data of the building facade components included in the building facade include component type sample characterization data of each type of building facade components included in the building facade, and the component type sample characterization data of each type of building facade component include the relative positional relationship of the building facade components on the building facade.

[0126] In the embodiment of the present application, the number of types contained in each building facade component can be used as the number of channels in a one-hot coding map, that is, each type of each building facade component corresponds to a channel in the one-hot coding map. For example, if the existing windows contain 3 types, the one-hot coding map of the windows in Table 3 is shown in Table 4:

[0127] Table 4-1 shows the channels corresponding to the first type of windows in Table 3

[0128] 0 0 0 0 0 0 1 0 0 0 …… …… …… …… ……

[0129] Table 4-2 shows the channels corresponding to the second type of windows in Table 3

[0130] 0 0 0 0 0 0 0 0 1 0 …… …… …… …… ……

[0131] Table 4-3 shows the channels corresponding to the third type of windows in Table 3

[0132] 0 0 0 0 0 0 0 0 0 0 …… …… …… …… ……

[0133] In Table 4-1, if the value is 1, it means that the building facade component corresponding to the cell with the value 1 is the first type of window. If the value is 0, it means that the building facade component corresponding to the cell with the value 0 is not the first type of window. In Table 4-2, if the value is 1, it means that the building facade component corresponding to the cell with the value 1 is the second type of window. If the value is 0, it means that the building facade component corresponding to the cell with the value 0 is not the second type of window. In Table 4-3, if all the values ​​are 0, it means that the sample characterization data of the component type of the building facade in Table 3 does not include the third type of window.

[0134] It should be noted that the number and arrangement of cells in Table 4 are the same as the number and arrangement of cells in Table 3.

[0135] In an optional embodiment, the component arrangement sample characterization data of the building facade components included in the building facade include component arrangement sample characterization data of the building facade components in the horizontal direction and component arrangement sample characterization data of the building facade components in the vertical direction.

[0136] In the embodiment of the present application, the horizontal arrangement mode, the vertical arrangement mode and the vertical arrangement mode of all building facade components in Table 3 can be represented by single-channel binary images respectively. As shown in Table 5, a single-channel binary image of the arrangement mode in Table 3 provided in the embodiment of the present application is shown. Among them, Table 5-1 is a single-channel binary image corresponding to the horizontal arrangement mode in Table 3, and Table 5-2 is a single-channel binary image corresponding to the vertical arrangement mode in Table 3.

[0137] 5-1 is the single-channel binary image corresponding to the horizontal arrangement in Table 3

[0138] 0 1 1 0 0 0 1 1 0 0 …… …… …… …… ……

[0139] 5-2 is the single-channel binary image corresponding to the vertical arrangement in Table 3

[0140] 1 1 1 1 1 1 1 1 1 1 …… …… …… …… ……

[0141] In Table 5-1 and Table 5-2, if the value in a cell is 1, it means that the building facade component corresponding to the cell with the value 1 is arranged repeatedly. If the value in a cell is 0, it means that the building facade component corresponding to the cell with the value 0 is not arranged repeatedly.

[0142] It should be noted that when generating a horizontal single-channel binary image, the vertical arrangement is not considered. When generating a vertical single-channel binary image, the horizontal arrangement is not considered.

[0143] After obtaining the longitudinal single-channel binary image, the transverse single-channel binary image, and the one-hot encoding images of all building facade component types, the single-channel binary image, the transverse single-channel binary image, and the one-hot encoding images of all building facade components can be spliced ​​together along the channel direction to obtain the component type sample representation data of the building facade and the multi-channel discrete representation corresponding to the arrangement sample representation data. For example, after splicing and combining the data in Table 4 and Table 5 along the channel direction, the multi-channel discrete layout arrangement data obtained is as follows: Figure 4 shown.

[0144] It should be noted that the building facade component corresponding to the cell with a value of 1 in Table 5 is a repeated arrangement. The number of repeated arrangements can be added to the table so that the number of repeated arrangements can be learned in the later model learning process to construct a building facade appearance that is exactly the same as the building facade appearance included in the building facade image. It is also possible not to mark the number of repeated arrangements, but to allow the model to learn that the building facade component corresponding to the cell is a repeated arrangement, and not to limit the number of repeated arrangements, so that the number of repeated arrangements is not learned in the later model learning process to construct a building facade appearance similar to the building facade appearance included in the building facade image.

[0145] S102: taking the layout representation sample data as input, training an image data reconstruction model until the image data reconstruction model converges, wherein the image data reconstruction model includes a data encoding sub-model and a data decoding sub-model;

[0146] In an embodiment of the present application, layout representation sample data can be input into an image data reconstruction model, and the image data reconstruction model outputs reconstructed layout representation sample data. The difference between the layout representation sample data and the reconstructed layout representation sample data is used as the first loss function to adjust the parameters of the image data reconstruction model until the model converges to obtain a trained image data reconstruction model.

[0147] In an optional embodiment, the image data reconstruction model includes a data encoding sub-model and a data decoding sub-model. The data encoding sub-model can generate a continuous potential representation by passing the above-mentioned multi-channel discrete layout arrangement data through an encoder network; the data decoding sub-model can convert the continuous potential representation into reconstructed multi-channel discrete layout arrangement data. Based on the multi-channel discrete layout arrangement data input to the data encoding sub-model and the reconstructed multi-channel discrete layout arrangement data output by the data decoding sub-model, the loss function of the data reconstruction model is calculated, and the data reconstruction model is trained until the data reconstruction model converges to obtain a trained data reconstruction model.

[0148] In an optional embodiment, step S102 includes:

[0149] Encoding the layout representation sample data through a data encoding sub-model to obtain a first continuous latent representation of the layout of the building facade in a latent space;

[0150] quantizing the first continuous latent representation to obtain a first discrete representation of the layout of the building facade;

[0151] Decoding the first discrete representation through a data decoding sub-model to obtain layout representation reconstruction data of the building facade;

[0152] Calculating a first loss function based on the layout representation sample data and the layout representation reconstruction data;

[0153] According to the first loss function, the parameters of the image data reconstruction model are adjusted.

[0154] In the embodiments of the present application, Figure 5As shown, it is a training example diagram of an image data reconstruction model provided by an embodiment of the present application. Multi-channel discrete layout arrangement data is input into the data encoding sub-model, and the input multi-channel discrete layout arrangement data is passed through the data encoding sub-model to generate a continuous latent representation Zh. The continuous latent representation Zh is mapped to an index S in a learnable discrete code book (codebook) through nearest neighbor search, and the continuous latent representation is quantized to the nearest feature in the code book (codebook) through the index S, thereby obtaining the quantized discrete representation Zc corresponding to the continuous latent representation. The quantized discrete representation Zc is input into the data decoding sub-model, and is converted by the data decoding sub-model into reconstructed multi-channel discrete layout arrangement data.

[0155] The reconstruction error is calculated based on the multi-channel discrete layout arrangement data of the input encoding sub-model and the reconstructed multi-channel discrete layout arrangement data output by the data decoding sub-model. The quantization error is calculated based on the continuous potential representation output by the data encoding sub-model and the quantized discrete representation. The sum of the reconstruction error and the quantization error is used as the loss function of the data reconstruction model, and the data reconstruction model is trained until the model converges to obtain a trained image data reconstruction model.

[0156] Among them, the relationship between the continuous potential representation Zh and the quantized discrete representation Zc in the nearest neighbor search is:

[0157]

[0158] in, is the discrete representation of the quantized position (i, j), is the continuous potential representation of position (i, j), C k is any code vector in the codebook predefined code vectors, C is a set of code vectors predefined in the codebook, and arg min||·|| is the minimum Euclidean distance.

[0159] S103: constructing an image data generation model based on the image encoding model and the trained data decoding sub-model;

[0160] In an embodiment of the present application, after the image data reconstruction model is trained, the trained data decoding sub-model is used as a part of the image data generation model to form a generation model together with the image encoding model.

[0161] In some embodiments, the step of constructing an image data generation model based on the image encoding model and the trained data decoding sub-model includes:

[0162] Freezing parameters of the data decoding sub-model in the image data reconstruction after training;

[0163] An image generation model is constructed based on the image encoding model and the data decoding sub-model with frozen parameters.

[0164] In the embodiment of the present application, the parameters of the trained data decoding sub-model are frozen. The data decoding sub-model with frozen parameters is used as a part of the image data generation model, and forms a generation model with the image coding model. During the training process of the image data generation model, only the parameters of the image coding model are adjusted, and the parameters of the data decoding sub-model are not adjusted.

[0165] like Figure 6 As shown, it is a schematic diagram of an image data generation model provided in an embodiment of the present application, and the generation model includes a picture encoding sub-network and a data decoding sub-model with frozen parameters.

[0166] S104: Taking the building facade image sample as input, training the image data generation model based on the size representation sample data until the image data generation model converges.

[0167] In the embodiment of the present application, the building facade image sample is used as the input data of the generation model, and the image data generation model outputs the predicted size representation data. The difference between the predicted size representation data and the size representation sample data is used as the loss function to adjust the parameters of the image coding model.

[0168] In an optional embodiment, the method further comprises: inputting the layout representation sample data into the trained data encoding sub-model to obtain a second continuous latent representation of the layout representation sample data in a latent space;

[0169] S104 includes: taking the building facade image sample as input, and training the image data generation model based on the second continuous latent representation and the size representation sample data until the image data generation model converges.

[0170] In an embodiment of the present application, the layout representation sample data corresponding to the building facade image sample can be first input into the data coding sub-model trained in step S102 to obtain a second continuous potential representation of the layout representation sample data in the latent space. Then the building facade image sample is input into the image coding model, and the image coding model outputs a predicted continuous potential representation and predicted size representation data. The difference between the second continuous potential representation and the predicted continuous potential representation is used as the second loss function of the image coding model, the difference between the predicted size representation data and the size representation sample data is used as the third loss function of the image coding model, and the sum of the second loss function and the third loss function is used as the loss function of the image coding model. The image coding model is trained until the image data generation model converges, and a trained image coding model is obtained.

[0171] In an optional embodiment, taking the building facade image sample as input and training the image data generation model based on the second continuous potential representation and the size representation sample data includes:

[0172] Inputting the building facade picture sample into the image coding model, and the image coding model outputs first intermediate feature data;

[0173] Encoding the first intermediate feature data to obtain a predicted continuous latent representation of the layout of the building facade in the building facade image sample in a latent space;

[0174] Calculating a first loss function based on the predicted continuous latent representation and the second continuous latent representation;

[0175] Pooling the intermediate feature data to obtain predicted size data of the building facade in the building facade image sample;

[0176] Calculating a second loss function based on the predicted size data and the size characterization sample data;

[0177] Adjust parameters of the image coding model according to the first loss function and the second loss function.

[0178] In the embodiment of the present application, Figure 6 The training process of the generative model shown is as follows: input the building facade picture sample into the image coding model, and the image coding model outputs the first intermediate feature data; encode the first intermediate feature data to obtain the predicted continuous potential representation of the layout of the building facade in the building facade picture sample in the latent space. The difference between the predicted continuous potential representation and the second continuous potential representation is used as the first loss function. Pooling is performed on the first intermediate feature data to obtain the predicted size data of the building facade in the building facade picture sample; the difference between the predicted size data and the size representation sample data is used as the second loss function. The sum of the first loss function and the second loss function is used as the loss function of the image coding model, and the parameters of the image coding model are adjusted until the model converges. A trained image coding model is obtained.

[0179] Since the data decoding sub-model with frozen model parameters is trained to convergence in step S102, after the image coding model is trained to convergence, a trained image data generation model is obtained.

[0180] In an optional embodiment, the step of pooling the first intermediate feature data to obtain predicted size data of the building facade in the building facade image sample includes:

[0181] Performing horizontal pooling on the first intermediate feature data to obtain horizontal predicted size data of the building facade in the building facade image sample;

[0182] The first intermediate feature data is pooled longitudinally to obtain longitudinal predicted size data of the building facade in the building facade image sample.

[0183] In the embodiment of the present application, the pooling operation of the first intermediate feature data is divided into horizontal pooling and vertical pooling, so that the generation model can pool the horizontal size and vertical size of the building facade in the building facade picture, and pool the horizontal size and vertical size of the building facade components of the building facade, so that the generation model outputs horizontal predicted size data and vertical predicted size data. The generation model after training can output the horizontal size data and vertical size data of the building facade in the building facade picture.

[0184] The embodiment of the present application provides a model training method, the method obtains a building facade picture sample, a layout representation sample data and a size representation sample data of the building facade in the building facade picture sample; takes the layout representation sample data as input, trains an image data reconstruction model until the image data reconstruction model converges, wherein the image data reconstruction model includes a data encoding sub-model and a data decoding sub-model; constructs an image data generation model based on the image encoding model and the trained data decoding sub-model; takes the building facade picture sample as input, and trains the image data generation model based on the size representation sample data until the image data generation model converges. The method can generate an image data generation model of the layout information and size information of the building facade in the building facade picture based on the building facade picture. The image data generation model can predict the layout and size of the building facade in the building facade picture, thereby reducing the cost of configuring the building facade appearance and improving the efficiency of configuring the building facade appearance.

[0185] In some embodiments, the present application also provides a data processing method, the method comprising:

[0186] Get the first building facade image;

[0187] Extracting second intermediate feature data for the first building facade in the first building facade image;

[0188] encoding the second intermediate feature data to obtain a third continuous latent representation of the layout of the first building facade in a latent space;

[0189] quantizing the third continuous latent representation to obtain a second discrete representation of the layout of the first building facade;

[0190] Decoding the second discrete representation to obtain layout representation data of the first building facade;

[0191] Pooling the second intermediate feature data to obtain dimension representation data of the first building facade;

[0192] A second building facade image is generated according to the layout representation data of the first building facade and the size representation data of the first building facade.

[0193] In an embodiment of the present application, when a building facade needs to be constructed, a first building facade image for reference can be first obtained. The first building facade image is input into the generation model, and the generation model extracts the second intermediate feature data in the first building facade image. The second intermediate feature data is encoded to generate a third continuous latent representation of the layout of the first building facade in the latent space, and the third continuous latent representation is quantized to obtain a second discrete representation of the layout of the first building facade. The second discrete representation is decoded to obtain the layout representation data of the first building facade. The second intermediate feature data is pooled to obtain the size representation data of the first building facade. Based on the layout representation data of the first building facade and the size representation data of the first building facade, a second building facade image is generated.

[0194] When generating the second building facade image, based on whether the number of repeated arrangement of the repeatedly arranged building facade components is marked, a second building facade image having the same or similar building facade appearance as the first building facade image can be generated. When the number of repeated arrangement of the repeatedly arranged building facade components is marked, a second building facade image having the same building facade appearance as the first building facade image can be generated. When the number of repeated arrangement of the repeatedly arranged building facade components is not marked, a second building facade image having a similar building facade appearance as the first building facade image can be generated.

[0195] Corresponding to the model training method provided in the embodiment of the present application, the embodiment of the present application also provides a model training device, such as Figure 7 As shown, the device comprises:

[0196] A first acquisition module 701 is used to acquire a building facade image sample, and layout representation sample data and size representation sample data of the building facade in the building facade image sample;

[0197] A first training module 702 is used to train an image data reconstruction model using the layout representation sample data as input until the image data reconstruction model converges, wherein the image data reconstruction model includes a data encoding sub-model and a data decoding sub-model;

[0198] A construction module 703 is used to construct an image data generation model based on the image encoding model and the trained data decoding sub-model;

[0199] The second training module 704 is used to take the building facade image sample as input and train the image data generation model based on the size representation sample data until the image data generation model converges.

[0200] In some embodiments, the first training module is further configured to:

[0201] Encoding the layout representation sample data through a data encoding sub-model to obtain a first continuous latent representation of the layout of the building facade in a latent space;

[0202] quantizing the first continuous latent representation to obtain a first discrete representation of the layout of the building facade;

[0203] Decoding the first discrete representation through a data decoding sub-model to obtain layout representation reconstruction data of the building facade;

[0204] Calculating a first loss function based on the layout representation sample data and the layout representation reconstruction data;

[0205] According to the first loss function, the parameters of the image data reconstruction model are adjusted.

[0206] In some embodiments, the layout representation sample data is input into the trained data encoding sub-model to obtain a second continuous latent representation of the layout representation sample data in a latent space;

[0207] The second training module is also used to:

[0208] Taking the building facade image sample as input, the image data generation model is trained based on the second continuous latent representation and the size representation sample data until the image data generation model converges.

[0209] In some embodiments, the second training module is further configured to:

[0210] Inputting the building facade picture sample into the image coding model, and the image coding model outputs first intermediate feature data;

[0211] Encoding the first intermediate feature data to obtain a predicted continuous latent representation of the layout of the building facade in the building facade image sample in a latent space;

[0212] Calculating a second loss function based on the predicted continuous latent representation and the second continuous latent representation;

[0213] Pooling the intermediate feature data to obtain predicted size data of the building facade in the building facade image sample;

[0214] Calculating a third loss function based on the predicted size data and the size characterization sample data;

[0215] Adjust the parameters of the image coding model according to the second loss function and the third loss function.

[0216] In some embodiments, the second training module is further configured to:

[0217] Performing horizontal pooling on the intermediate feature data to obtain horizontal predicted size data of the building facade in the building facade image sample;

[0218] The intermediate feature data is pooled longitudinally to obtain longitudinal predicted size data of the building facade in the building facade image sample.

[0219] In some embodiments, the sample data representing the layout of the building facade includes sample data representing component types of the building facade components included in the building facade, and sample data representing component arrangement of the building facade components included in the building facade.

[0220] In some embodiments, the component type sample characterization data of the building facade components included in the building facade include component type sample characterization data of each building facade component included in the building facade, and the component type sample characterization data of each building facade component include the relative position relationship of the building facade components on the building facade.

[0221] In some embodiments, the component arrangement sample characterization data of the building facade components included in the building facade include component arrangement sample characterization data of the building facade components in the horizontal direction and component arrangement sample characterization data of the building facade components in the vertical direction.

[0222] In some embodiments, the component arrangement of the building facade components includes a single arrangement and a repeated arrangement.

[0223] In some embodiments, for repeatedly arranged building facade components, data deduplication is performed in the component arrangement sample representation data and the component type sample representation data respectively.

[0224] In some embodiments, the building facade components include independent building facade components and combined building facade components.

[0225] In some embodiments, the building blocks are further configured to:

[0226] Freezing parameters of the data decoding sub-model in the image data reconstruction after training;

[0227] An image generation model is constructed based on the image encoding model and the data decoding sub-model with frozen parameters.

[0228] Corresponding to the data processing method provided in the embodiment of the present application, the embodiment of the present application also provides a data processing device, such as Figure 8 As shown, the device comprises:

[0229] The second acquisition module 801 is used to acquire a first building facade image;

[0230] An extraction module 802 is used to extract second intermediate feature data for the first building facade in the first building facade image;

[0231] An encoding module 803 is used to encode the second intermediate feature data to obtain a third continuous latent representation of the layout of the first building facade in a latent space;

[0232] A quantization module 804 is used to quantize the third continuous potential representation to obtain a second discrete representation of the layout of the first building facade;

[0233] A decoding module 805, configured to decode the second discrete representation to obtain layout representation data of the first building facade;

[0234] A pooling module 806 is used to perform pooling on the second intermediate feature data to obtain dimension representation data of the first building facade;

[0235] The generating module 807 is used to generate a second building facade image according to the layout representation data of the first building facade and the size representation data of the first building facade.

[0236] Corresponding to the model training method provided in the embodiment of the present application, the embodiment of the present application also provides an electronic device. Fig. 9 As shown, the electronic device 1500 includes: a processor 1501; and a memory 1502, which is used to store a program of a model training method. After the electronic device is powered on and runs the program storing the model training method through the processor, the following steps are performed:

[0237] Acquire a building facade image sample, and layout representation sample data and size representation sample data of the building facade in the building facade image sample;

[0238] Taking the layout representation sample data as input, training an image data reconstruction model until the image data reconstruction model converges, wherein the image data reconstruction model includes a data encoding sub-model and a data decoding sub-model;

[0239] Construct an image data generation model based on the image encoding model and the trained data decoding sub-model;

[0240] The building facade image sample is used as input, and the image data generation model is trained based on the size representation sample data until the image data generation model converges.

[0241] Corresponding to the model training method provided in the embodiment of the present application, the embodiment of the present application further provides a computer-readable storage medium storing a program of the model training method, which is executed by a processor to perform the following steps:

[0242] Acquire a building facade image sample, and layout representation sample data and size representation sample data of the building facade in the building facade image sample;

[0243] Taking the layout representation sample data as input, training an image data reconstruction model until the image data reconstruction model converges, wherein the image data reconstruction model includes a data encoding sub-model and a data decoding sub-model;

[0244] Construct an image data generation model based on the image encoding model and the trained data decoding sub-model;

[0245] The building facade image sample is used as input, and the image data generation model is trained based on the size representation sample data until the image data generation model converges.

[0246] Corresponding to the model training method provided in the embodiment of the present application, the embodiment of the present application also provides a computer program product, the program product includes: a computer program, the computer program is stored in a readable storage medium, at least one processor of the electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device performs the following steps:

[0247] Acquire a building facade image sample, and layout representation sample data and size representation sample data of the building facade in the building facade image sample;

[0248] Taking the layout representation sample data as input, training an image data reconstruction model until the image data reconstruction model converges, wherein the image data reconstruction model includes a data encoding sub-model and a data decoding sub-model;

[0249] Construct an image data generation model based on the image encoding model and the trained data decoding sub-model;

[0250] The building facade image sample is used as input, and the image data generation model is trained based on the size representation sample data until the image data generation model converges.

[0251] It should be noted that for the detailed description of the apparatus, electronic device, computer-readable storage medium and computer program product provided in the embodiments of the present application, reference can be made to the relevant description of the method in the embodiments of the present application, which will not be repeated here.

[0252] Although the present application is disclosed as above in the form of a preferred embodiment, it is not intended to limit the present application. Any technical personnel in this field may make possible changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be based on the scope defined by the claims of the present application.

[0253] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0254] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for enabling an electronic device (which can be a personal computer, a server, or a network device, etc.) or a processor (English: processor) to perform some steps of the method described in each embodiment of the present application.

[0255] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be implemented by a combination of hardware and software modules in the processor.

[0256] The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk, etc.

[0257] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0258] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0259] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0260] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A model training method, characterized in that: The method comprises: Acquire a building facade image sample, and layout representation sample data and size representation sample data of the building facade in the building facade image sample; Taking the layout representation sample data as input, training an image data reconstruction model until the image data reconstruction model converges, wherein the image data reconstruction model includes a data encoding sub-model and a data decoding sub-model; Construct an image data generation model based on the image encoding model and the trained data decoding sub-model; The building facade image sample is used as input, and the image data generation model is trained based on the size representation sample data until the image data generation model converges.

2. The model training method according to claim 1, characterized in that: The step of taking the layout representation sample data as input and training the image data reconstruction model comprises: Encoding the layout representation sample data through a data encoding sub-model to obtain a first continuous latent representation of the layout of the building facade in a latent space; quantizing the first continuous latent representation to obtain a first discrete representation of the layout of the building facade; Decoding the first discrete representation through a data decoding sub-model to obtain layout representation reconstruction data of the building facade; Calculating a first loss function based on the layout representation sample data and the layout representation reconstruction data; According to the first loss function, the parameters of the image data reconstruction model are adjusted.

3. The model training method according to claim 1, characterized in that: The method further comprises: Inputting the layout representation sample data into the trained data encoding sub-model to obtain a second continuous latent representation of the layout representation sample data in a latent space; The step of taking the building facade image sample as input and training the image data generation model based on the size representation sample data until the image data generation model converges includes: Taking the building facade image sample as input, the image data generation model is trained based on the second continuous latent representation and the size representation sample data until the image data generation model converges.

4. The model training method according to claim 3, characterized in that: The step of taking the building facade image sample as input and training the image data generation model based on the second continuous potential representation and the size representation sample data comprises: Inputting the building facade picture sample into the image coding model, and the image coding model outputs first intermediate feature data; Encoding the first intermediate feature data to obtain a predicted continuous latent representation of the layout of the building facade in the building facade image sample in a latent space; Calculating a second loss function based on the predicted continuous latent representation and the second continuous latent representation; Pooling the first intermediate feature data to obtain predicted size data of the building facade in the building facade image sample; Calculating a third loss function based on the predicted size data and the size characterization sample data; Adjust the parameters of the image coding model according to the second loss function and the third loss function.

5. The model training method according to claim 4, characterized in that: The step of pooling the first intermediate feature data to obtain predicted size data of the building facade in the building facade image sample includes: Performing horizontal pooling on the first intermediate feature data to obtain horizontal predicted size data of the building facade in the building facade image sample; The first intermediate feature data is pooled longitudinally to obtain longitudinal predicted size data of the building facade in the building facade image sample.

6. The model training method according to claim 1, characterized in that: The sample data representing the layout of the building facade includes sample data representing the component types of the building facade components included in the building facade, and sample data representing the component arrangement mode of the building facade components included in the building facade.

7. The model training method according to claim 6, characterized in that: The component type sample characterization data of the building facade components included in the building facade include component type sample characterization data of each building facade component included in the building facade, and the component type sample characterization data of each building facade component include the relative position relationship of the building facade components on the building facade.

8. The model training method according to claim 6, characterized in that: The component arrangement sample characterization data of the building facade components included in the building facade include the component arrangement sample characterization data of the building facade components in the horizontal direction and the component arrangement sample characterization data of the building facade components in the vertical direction.

9. The model training method according to claim 6, characterized in that: The component arrangement of the building facade components includes single arrangement and repeated arrangement.

10. The model training method according to claim 9, characterized in that: The method further comprises: For repeatedly arranged building facade components, data deduplication is performed in the component arrangement sample representation data and the component type sample representation data respectively.

11. The model training method according to claim 9, characterized in that: The building facade components include independent building facade components and combined building facade components.

12. The model training method according to claim 1, characterized in that: The image data generation model is constructed based on the image encoding model and the trained data decoding sub-model, including: Freezing parameters of the data decoding sub-model in the image data reconstruction after training; An image generation model is constructed based on the image encoding model and the data decoding sub-model with frozen parameters.

13. A data processing method, characterized in that: The method comprises: Get the first building facade image; Extracting second intermediate feature data for the first building facade in the first building facade image; encoding the second intermediate feature data to obtain a third continuous latent representation of the layout of the first building facade in a latent space; quantizing the third continuous latent representation to obtain a second discrete representation of the layout of the first building facade; Decoding the second discrete representation to obtain layout representation data of the first building facade; Pooling the second intermediate feature data to obtain dimension representation data of the first building facade; A second building facade image is generated according to the layout representation data of the first building facade and the size representation data of the first building facade.

14. A model training device, characterized in that: The device comprises: A first acquisition module is used to acquire a building facade image sample, and layout representation sample data and size representation sample data of the building facade in the building facade image sample; A first training module, used for training an image data reconstruction model with the layout representation sample data as input until the image data reconstruction model converges, wherein the image data reconstruction model includes a data encoding sub-model and a data decoding sub-model; A construction module, for constructing an image data generation model based on the image encoding model and the trained data decoding sub-model; The second training module is used to take the building facade image sample as input and train the image data generation model based on the size representation sample data until the image data generation model converges.

15. A data processing device, characterized in that: The device comprises: A second acquisition module is used to acquire a first building facade image; An extraction module, configured to extract second intermediate feature data for the first building facade in the first building facade image; an encoding module, configured to encode the second intermediate feature data to obtain a third continuous latent representation of the layout of the first building facade in a latent space; a quantization module, configured to quantize the third continuous potential representation to obtain a second discrete representation of the layout of the first building facade; A decoding module, configured to decode the second discrete representation to obtain layout representation data of the first building facade; A pooling module, used for pooling the second intermediate feature data to obtain dimension representation data of the first building facade; A generating module is used to generate a second building facade image according to the layout representation data of the first building facade and the size representation data of the first building facade.

16. An electronic device, characterized in that: include: processor; as well as The memory is used to store a data processing program. After the electronic device is powered on and the program is run by the processor, the method according to any one of claims 1 to 13 is executed.

17. A computer-readable storage medium, characterized in that: A data processing program is stored, and the program is run by a processor to execute the method according to any one of claims 1 to 13.

Images