Indoor structure determination method, model training method, data processing method, equipment and medium

Abstract

The embodiment of the invention provides an indoor structure determination method, a model training method, a data processing method, equipment and a medium. In the embodiment of the invention, the indoor structure is classified into a plurality of preset indoor structure models in advance, and in the process of predicting the three-dimensional space information of the indoor structure of the indoor image, the target indoor structure model of the indoor structure corresponding to the indoor image is predicted; and adding the predicted target indoor structure model of the indoor structure corresponding to the indoor image into the network as priori knowledge to obtain three-dimensional space information of the indoor structure of the indoor image. Wherein the indoor structure model is added when the three-dimensional space information of the indoor structure is predicted, reference information for predicting the three-dimensional space information of the indoor structure is increased, and the prediction accuracy of the three-dimensional space information of the indoor structure corresponding to the indoor image can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, device and medium for indoor structure determination, model training and data processing. Background Art

[0002] In actual house decoration scenarios, if the three-dimensional spatial information of the indoor structure of the house can be known, it can assist in decoration design, virtual decoration and other applications. Therefore, indoor structure estimation has attracted the attention of a wide range of researchers. Indoor structure estimation refers to predicting indoor structures based on indoor scene pictures or multi-frame videos of indoor scenes input by users. Indoor structure, also known as indoor space structure, mainly refers to the spatial three-dimensional structure formed by indoor building basic components such as walls, floors and roofs.

[0003] In actual interior decoration scenarios, users can often only provide two-dimensional scene images of the house. However, due to the lack of spatial depth information in two-dimensional images, it is extremely challenging to determine the three-dimensional spatial information of the indoor structure using two-dimensional images of indoor scenes. Summary of the invention

[0004] Multiple aspects of the present application provide a method, device and medium for indoor structure determination, model training and data processing, so as to improve the accuracy of the determined indoor structure.

[0005] The present application provides a method for determining an indoor structure, including:

[0006] Acquire an indoor image to be processed;

[0007] Using the first network model in the target indoor structure prediction model, extracting features from the indoor image to obtain image features of the indoor image;

[0008] Determining a target indoor structure model of the indoor structure corresponding to the indoor image from a plurality of preset indoor structure models according to image features of the indoor image using the second network model in the target indoor structure prediction model;

[0009] The third network model in the target indoor structure prediction model is used to determine the three-dimensional spatial information of the indoor structure corresponding to the indoor image according to the image features of the indoor image and the target indoor structure model.

[0010] The present application also provides a model training method, including:

[0011] Acquire a plurality of indoor image samples pre-labeled with prior information; the prior information includes: prior three-dimensional spatial information of indoor structures corresponding to the plurality of indoor image samples and prior probabilities that the indoor structures of the plurality of indoor image samples are preset multiple indoor structure models;

[0012] Using a first network model in the indoor structure prediction model to be trained, extracting features from the plurality of indoor image samples to obtain image features of the plurality of indoor image samples;

[0013] Using the second network model in the indoor structure prediction model to be trained, and according to the image features of the multiple indoor image samples, determining the prediction probability that the multiple indoor image samples are preset multiple indoor structure models;

[0014] Determining predicted indoor structure models of indoor structures corresponding to the multiple indoor image samples according to the predicted probabilities that the multiple indoor image samples are the multiple indoor structure models;

[0015] Determine predicted three-dimensional spatial information of indoor structures corresponding to the plurality of indoor image samples respectively according to image features of the plurality of indoor image samples and predicted indoor structure models of indoor structures corresponding to the plurality of indoor image samples by using a third network model in the indoor structure prediction model to be trained;

[0016] According to a first difference between the predicted three-dimensional spatial information and the prior three-dimensional spatial information, and a second difference between the predicted probability and the prior probability, the model parameters of the indoor structure model to be trained are adjusted to obtain a target indoor structure prediction model.

[0017] Optionally, the prior information further includes: prior pose information of image acquisition devices corresponding to the multiple indoor image samples; the method further includes:

[0018] Determine predicted position information of image acquisition devices corresponding to the plurality of indoor image samples respectively according to the image features of the plurality of indoor samples and the predicted indoor structure models of the indoor structures corresponding to the plurality of indoor image samples by using the third network model in the indoor structure prediction model to be trained;

[0019] The adjusting the model parameters of the indoor structure model to be trained according to the first difference between the predicted three-dimensional spatial information and the prior three-dimensional spatial information, and the second difference between the predicted probability and the prior probability, comprises:

[0020] According to the first difference, the second difference and the third difference between the predicted pose information and the prior pose information, the model parameters of the indoor structure model to be trained are adjusted to obtain the target indoor structure prediction model.

[0021] The present application also provides a data processing method applicable to a cloud server, the method comprising:

[0022] In response to a request to call a target service, determining a processing resource corresponding to the target service; the target service is a service that provides indoor structure prediction and / or model training services;

[0023] The processing resources corresponding to the target service are used to execute the steps in the indoor structure determination method and / or model training method.

[0024] The embodiment of the present application further provides a computing device, comprising: a memory and a processor; wherein the memory is used to store a computer program;

[0025] The processor is coupled to the memory and is used to execute the computer program to perform the steps in the indoor structure determination method and / or model training method.

[0026] An embodiment of the present application also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed by one or more processors, the one or more processors are caused to execute the steps in the above-mentioned indoor structure determination method and / or model training method.

[0027] In the embodiment of the present application, the indoor structure is pre-classified into a plurality of preset indoor structure models, and in the process of predicting the three-dimensional spatial information of the indoor structure of the indoor image, the target indoor structure model of the indoor structure corresponding to the indoor image is first predicted, and the predicted target indoor structure model of the indoor structure corresponding to the indoor image is added to the network as prior knowledge to predict the three-dimensional spatial information of the indoor structure of the indoor image. Among them, the indoor structure model is added when predicting the three-dimensional spatial information of the indoor structure, which increases the reference information for predicting the three-dimensional spatial information of the indoor structure, and helps to improve the prediction accuracy of the three-dimensional spatial information of the indoor structure corresponding to the indoor image. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0029] Figure 1 A flowchart of a model training method provided in an embodiment of the present application;

[0030] Figure 2 A schematic diagram of the structure of the indoor structure model provided in the embodiment of the present application;

[0031] Figure 3 A schematic diagram of the principle of the indoor structure prediction model provided in the embodiment of the present application;

[0032] Figure 4 A schematic diagram of the structure of a residual block provided in an embodiment of the present application;

[0033] Figure 5 A schematic diagram of the specific structure of the indoor structure prediction model provided in the embodiment of the present application;

[0034] Figure 6 A schematic diagram of a flow chart of a method for determining an indoor structure provided in an embodiment of the present application;

[0035] Figure 7 A schematic diagram of another indoor structure prediction model provided in an embodiment of the present application;

[0036] Figure 8 A schematic diagram of the correspondence between the three-dimensional spatial information of the indoor structure corresponding to the indoor image and the indoor image provided in the embodiment of the present application;

[0037] Fig. 9 A schematic diagram of the result of indoor structure prediction performed by the target indoor structure prediction model provided in the embodiment of the present application;

[0038] Fig.10 A schematic diagram of the structure of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0040] In order to determine the three-dimensional spatial information of the indoor structure using the two-dimensional image of the indoor scene, in some embodiments of the present application, the indoor structure is pre-classified into a plurality of preset indoor structure models, and in the process of predicting the three-dimensional spatial information of the indoor structure of the indoor image, the target indoor structure model of the indoor structure corresponding to the indoor image is first predicted, and the predicted target indoor structure model of the indoor structure corresponding to the indoor image is added to the network as prior knowledge to predict the three-dimensional spatial information of the indoor structure of the indoor image. Among them, the indoor structure model is added when predicting the three-dimensional spatial information of the indoor structure, which increases the reference information for predicting the three-dimensional spatial information of the indoor structure, and helps to improve the prediction accuracy of the three-dimensional spatial information of the indoor structure corresponding to the indoor image.

[0041] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0042] It should be noted that the same reference numerals denote the same objects in the following drawings and embodiments, and therefore, once an object is defined in one drawing or embodiment, it does not need to be further discussed in the subsequent drawings and embodiments.

[0043] In the building interior decoration scene, indoor structure estimation can predict the indoor structure based on the indoor scene picture or indoor scene video input by the user. Then, based on the determined indoor structure, the building interior can be virtually decorated and virtual furniture can be placed to obtain a 3D indoor decoration simulation map. Users can view the decoration effect through the 3D indoor decoration simulation map.

[0044] In practical applications, the indoor images taken by users correspond to complex spatial scenes with many occlusions and complex layouts. Indoor structure estimation requires predicting the indoor three-dimensional spatial structure based on a single indoor image. However, it is difficult to extract depth information from a two-dimensional image, especially from a single two-dimensional image. This undoubtedly poses a great challenge to using a single two-dimensional image of an indoor scene to determine the three-dimensional spatial information of the indoor structure.

[0045] In order to determine the three-dimensional spatial information of indoor structures using a single indoor image, the embodiment of the present application introduces a neural network model, and estimates the indoor structure of a single indoor image through the neural network model to obtain the three-dimensional spatial information of the indoor structure corresponding to the indoor image. In the embodiment of the present application, the neural network model is mainly used to predict indoor structures, and therefore, the adopted neural network model is defined as an indoor structure prediction model. Before the indoor structure prediction model is used online, the initial indoor structure prediction model needs to be trained. The following is an exemplary description of the model training method of the indoor structure prediction model provided in the embodiment of the present application.

[0046] Figure 1The following is a flow chart of the model training method provided in the embodiment of the present application. Figure 1 As shown, the model training method mainly includes:

[0047] 101. Acquire a plurality of indoor image samples pre-labeled with prior information; the prior information includes: prior three-dimensional spatial information of indoor structures corresponding to the plurality of indoor image samples and prior probabilities that the plurality of indoor image samples are preset multiple indoor structure models.

[0048] 102. Using the first network model in the indoor structure prediction model to be trained, perform feature extraction on multiple indoor image samples to obtain image features of the multiple indoor image samples.

[0049] 103. Using the second network model in the indoor structure prediction model to be trained, according to the image features of the multiple indoor image samples, determine the prediction probabilities that the multiple indoor image samples are preset multiple indoor structure models.

[0050] 104. Determine predicted indoor structure models of indoor structures corresponding to the multiple indoor image samples according to prediction probabilities that the multiple indoor image samples are multiple indoor structure models.

[0051] 105. Determine predicted three-dimensional spatial information of indoor structures corresponding to the multiple indoor image samples respectively according to image features of the multiple indoor image samples and predicted indoor structure models of the indoor structures corresponding to the multiple indoor image samples using the third network model in the indoor structure prediction model to be trained.

[0052] 106. According to a first difference between the predicted three-dimensional spatial information and the prior three-dimensional spatial information, and a second difference between the predicted probability and the prior probability, adjust model parameters of the indoor structure model to be trained to obtain a target indoor structure prediction model.

[0053] In this embodiment, in order to train the indoor structure prediction model, multiple indoor image samples may be obtained in step 101. Multiple refers to 2 or more. In this embodiment, the specific number of indoor image samples is not limited. For the accuracy of the indoor structure prediction model, the number of indoor image samples is generally hundreds, thousands, tens of thousands, or even more.

[0054] In an embodiment of the present application, indoor image samples may include: indoor image samples acquired by using an image acquisition device to acquire a real indoor scene. The image acquisition device refers to an electronic device with an image acquisition function, such as a camera, a webcam, or other electronic devices with a camera (such as a mobile phone, a tablet computer, etc.). In some embodiments, in order to reduce the workload of sample acquisition, a computing device may also be used to render a virtual indoor scene image as an indoor image sample. Accordingly, indoor image samples may include: indoor image samples acquired by using an image acquisition device to acquire a real indoor scene, and virtual indoor scene images (i.e., virtual indoor image samples) rendered by a computing device.

[0055] Since the accuracy of model training using indoor images of real indoor scenes is relatively high, in order to reduce the workload of acquiring indoor image samples while ensuring the accuracy of the indoor structure prediction model, the number of indoor image samples collected by the image acquisition device for the real indoor scene can be controlled to be greater than or equal to the number of rendered virtual indoor image samples, that is, the ratio between the number of indoor image samples collected by the image acquisition device for the real indoor scene and the number of rendered virtual indoor image samples is greater than or equal to 1:1.

[0056] The indoor image samples are pre-labeled with prior information. The prior information includes: prior three-dimensional spatial information of the indoor structure corresponding to the indoor image sample, and prior probability that the indoor structure corresponding to the indoor image sample is a preset multiple indoor structure model. Multiple refers to 2 or more.

[0057] Among them, the prior three-dimensional spatial information of the indoor structure corresponding to the indoor image sample refers to the three-dimensional spatial information of the indoor structure corresponding to the pre-labeled indoor image sample, which can be represented by the three-dimensional spatial information of the three-dimensional bounding box of the indoor structure corresponding to the indoor image sample. The three-dimensional spatial information of the three-dimensional bounding box of the indoor structure corresponding to the indoor image sample (defined as the prior three-dimensional bounding box) may include: coordinate information of eight vertices of the prior three-dimensional bounding box of the indoor structure corresponding to the indoor image sample, such as coordinate information of eight vertices of the prior three-dimensional bounding box of the indoor structure corresponding to the indoor image sample in a set coordinate system (such as a world coordinate system, etc.).

[0058] In some embodiments, in order to speed up the convergence speed of the indoor structure prediction model, the three-dimensional spatial information of the prior three-dimensional bounding box of the indoor structure corresponding to the indoor image sample may be improved, mainly by reducing the parameter dimension of the three-dimensional spatial information of the prior three-dimensional bounding box of the indoor structure corresponding to the indoor image sample. For example, the three-dimensional spatial information of the prior three-dimensional bounding box of the indoor structure corresponding to the indoor image sample may include: the coordinates of the center point of the prior three-dimensional bounding box of the indoor structure corresponding to the indoor image sample, the length of the length, width and height of the prior three-dimensional bounding box, and the unit vectors of the length, width and height of the prior three-dimensional bounding box.

[0059] Since the coordinate information of the 8 vertices of the above three-dimensional bounding box corresponds to 3 coordinate values ​​(x, y, z), 8 vertices have 24 coordinate values, that is, the three-dimensional spatial information of the three-dimensional bounding box corresponds to 14 coordinate values. The coordinates of the center point of the three-dimensional bounding box of the indoor structure corresponding to the indoor image sample, the length (l, w, h) of the length, width and height of the three-dimensional bounding box, and the unit vectors of the length, width and height of the three-dimensional bounding box are In the example, the coordinates of the center point are (x, y, z) 3 coordinate values, the length, width and height are 3 values, and the unit vectors of the length, width and height of the 3D bounding box are 3 values. Then the 3D spatial information of the 3D bounding box corresponds to 9 values. Compared with the aforementioned embodiment in which the 3D spatial information of the 3D bounding box is represented by the coordinate information of 8 vertices, the parameter dimension and data volume of the 3D spatial information of the 3D bounding box are reduced, that is, the output of the indoor structure prediction model: the parameter dimension and data volume of the 3D spatial information of the indoor structure are reduced. When training the indoor structure prediction model, the amount of calculation of the loss function can be reduced, and then the amount of calculation of adjusting the model parameters of the indoor structure prediction model based on the loss function can be reduced, which helps to accelerate the convergence of the model.

[0060] The representation method of the three-dimensional spatial information of the indoor structure shown in the above embodiment is only an example and does not constitute a limitation.

[0061] The inventor of the present application has conducted research and analysis on indoor structures and found that indoor structures can be divided into multiple preset indoor structure models. Multiple refers to 2 or more types. The indoor structure model refers to the three-dimensional space structure formed by the components inside the building, mainly the three-dimensional space structure formed by the ground, walls and top of the building. The top of the building can be a ceiling or a roof, etc. The inventor of the present application has conducted research and analysis and found that the preset multiple indoor structure models can mainly include Figure 2The following 11 indoor structure models are shown: (1) Type 0: an indoor structure model consisting of three walls, a building top (such as a ceiling, etc.) and a ground; (2) Type 1: an indoor structure model consisting of three walls and a ground; (3) Type 2: an indoor structure model consisting of three walls and a building top (such as a ceiling); (4) Type 3: an indoor structure model consisting of two walls and a ceiling; (5) Type 4: an indoor structure model consisting of two walls and a ground; (6) Type 5: an indoor structure model consisting of two walls, a building top (such as a ceiling) and a ground; (7) Type 6: an indoor structure model consisting of one wall, a building top (such as a ceiling) and a ground; (8) Type 7: an indoor structure model consisting of three walls; (9) Type 8: an indoor structure model consisting of one wall and a building top (such as a ceiling); (10) Type 9: an indoor structure model consisting of one wall and a ground; and, Type 10: an indoor structure model consisting of two walls.

[0062] In this embodiment, the indoor structure model corresponding to the indoor structure corresponding to the indoor image sample may also be pre-labeled. Among them, the indoor structure model corresponding to the indoor structure corresponding to the pre-labeled indoor image sample is defined as the prior indoor structure model. Accordingly, the prior probability that the indoor structure corresponding to the above-mentioned indoor image sample is a plurality of preset indoor structure models may include: the prior probability that the indoor structure corresponding to the indoor image sample is a pre-labeled prior indoor structure model is 1; the prior probability that the indoor structure corresponding to the indoor image sample is other indoor structure models is 0. For example, if the prior indoor structure model of the indoor structure corresponding to a certain indoor image sample is type 0, then the prior probability that the indoor structure corresponding to the indoor image sample is type 0 is 1, and the prior probability that it is other types 1-10 is 0, etc.

[0063] In this embodiment, if Figure 3 As shown, the indoor structure prediction model to be trained can be divided into a first network model, a second network model and a third network model according to the function. Among them, the first network model is mainly used for image feature extraction; the second network model is mainly used for predicting the indoor structure model of the indoor image or the indoor image sample corresponding to the indoor structure; the third network model is used to predict the three-dimensional spatial information of the indoor structure corresponding to the indoor image or the indoor image sample.

[0064] In the embodiments of the present application, the specific implementation form of the neural network model used in the indoor structure prediction model is not limited. Optionally, the indoor structure prediction model can be a convolutional neural network (CNN), a deep learning neural network (DNN), a recurrent neural network (RNN), a convolutional neural tensor network (CNTN) or a residual neural network (ResNet), etc. The indoor structure prediction model can adopt a network structure of a convolutional layer and a fully connected layer. For example, a residual neural network can be selected as a network for image feature extraction. Among them, compared with a simple convolutional neural network, the residual neural network can cross the fully connected layer, which can prevent the gradient from disappearing during the model training process.

[0065] Among them, the basic structure of the residual block in the residual neural network is as follows Figure 4 As shown. Figure 4 In the residual neural network shown in FIG. 1 , in the residual neural network, the input x of the residual block acts on the output F(x) of the convolutional layer of the residual block as the output y of the residual block. That is, the output y of the residual block is y = F(x) + x. The residual block uses a skip connection to alleviate the gradient vanishing problem caused by increasing the depth in the deep neural network.

[0066] Accordingly, the indoor structure prediction model can be implemented as Figure 5 The network structure shown in Figure 5 In the convolution blocks 1-3, each circle can represent one or more residual blocks. Multiple refers to 2 or more. Figure 5 In the embodiment, the first network model of the indoor structure prediction model may include: a convolution layer and a convolution block 1. The convolution block 1 may include multiple residual blocks. The convolution layer and the convolution block 1 are mainly used for image feature extraction. The second network model of the indoor structure prediction model may include: a fully connected layer, which is mainly used to predict the indoor structure model of the indoor structure corresponding to the indoor image or the indoor image sample. The third network model of the indoor structure prediction model may include: one or more convolution blocks. Multiple refers to 2 or more, Figure 5 The third network model includes only two convolution blocks (convolution blocks 2 and 3) as an example for illustration, but it does not constitute a limitation. Multiple convolution blocks can be connected by convolution layers and pooling layers. Of course, the third network model may also include: a pooling layer and a fully connected layer connected to the last convolution block in the multiple convolution blocks. Among them, the third network model is used to predict the three-dimensional spatial information of the indoor structure corresponding to the indoor image or the indoor image sample.

[0067] Based on the indoor structure prediction model, in step 102, the first network model in the indoor structure prediction model to be trained may be used to perform feature extraction on a plurality of indoor image samples respectively, so as to obtain image features of each of the plurality of indoor image samples.

[0068] Further, in step 103, the second network model in the indoor structure prediction model to be trained can be used to determine the prediction probability that the multiple indoor image samples are preset multiple indoor structure models according to the image features of the multiple indoor image samples, that is, the probability of each indoor image sample being each indoor structure model can be predicted. For example, the second network model in the indoor structure prediction model to be trained can be used to predict the probability that each indoor image sample is each type of the above-mentioned 11 indoor structure models according to the image features of the multiple indoor image samples. The predicted probability is defined as the predicted probability.

[0069] Further, in step 104, the predicted indoor structure model of the indoor structure corresponding to the multiple indoor image samples can be determined according to the prediction probability that the multiple indoor image samples are preset multiple indoor structure models, that is, it is predicted which one of the multiple preset indoor structure models the indoor structure corresponding to each indoor image sample is. The predicted indoor structure model is defined as the predicted indoor structure model.

[0070] In some embodiments, for any indoor image sample A among multiple indoor image samples, the indoor structure model whose prediction probability meets the set probability conditions can be determined based on the prediction probability of the indoor image sample A being a preset multiple indoor structure models, and is the predicted structure model of the indoor structure corresponding to the indoor image sample A.

[0071] For example, the probability condition may be set as the maximum prediction probability. Accordingly, based on the prediction probability of the indoor image sample A being a preset multiple indoor structure model, the indoor structure model with the maximum corresponding prediction probability may be determined as the prediction structure model of the indoor structure corresponding to the indoor image sample A. For another example, the probability condition may be set as the prediction probability being greater than or equal to a set probability threshold, such as 0.85, 0.9 or 0.95. Accordingly, based on the prediction probability of the indoor image sample A being a preset multiple indoor structure model, the indoor structure model with the corresponding prediction probability being greater than or equal to the set probability threshold may be determined as the prediction structure model of the indoor structure corresponding to the indoor image sample A.

[0072] After determining the indoor structure model of the indoor structure corresponding to each indoor image sample, in step 105, the third network model in the indoor structure prediction model to be trained can be used to determine the predicted three-dimensional spatial information of the indoor structures corresponding to the multiple indoor image samples according to the image features of each of the multiple indoor image samples and the predicted indoor structure model of the indoor structure corresponding to the multiple indoor image samples, that is, predict the three-dimensional spatial information of the indoor structures corresponding to the multiple indoor image samples. The predicted three-dimensional spatial information is defined as the predicted three-dimensional spatial information.

[0073] Specifically, for any indoor image sample A among multiple indoor image samples, the predicted three-dimensional spatial information of the indoor structure corresponding to the indoor image sample A can be determined based on the image features of the indoor image sample A and the predicted indoor structure model of the indoor structure corresponding to the indoor image sample A.

[0074] After determining the predicted three-dimensional spatial information of the indoor structure corresponding to the indoor image sample and the predicted indoor structure model of the indoor structure corresponding to the indoor image sample, in step 106, the model parameters of the indoor structure model to be trained can be adjusted according to the first difference between the predicted three-dimensional spatial information and the prior three-dimensional spatial information, and the second difference between the predicted probability and the prior probability, so as to obtain the target indoor structure prediction model.

[0075] The first difference between the predicted three-dimensional spatial information and the prior three-dimensional spatial information includes: the difference between the predicted three-dimensional spatial information corresponding to each indoor image sample and the prior three-dimensional spatial information corresponding to the same indoor image sample. The second difference between the predicted probability and the prior probability includes: the difference between the predicted probability corresponding to each indoor image sample and the prior probability corresponding to the same indoor image sample.

[0076] In an embodiment of the present application, the sum of the first difference and the second difference can be used to form a loss function. Among them, the first difference between the predicted three-dimensional spatial information and the prior three-dimensional spatial information can be expressed as: the mean square error or variance between each parameter in the predicted three-dimensional spatial information corresponding to each indoor image sample and each parameter in the prior three-dimensional spatial information. For an embodiment in which the first difference between the predicted three-dimensional spatial information and the prior three-dimensional spatial information is expressed as the mean square error between each parameter in the predicted three-dimensional spatial information and each parameter in the prior three-dimensional spatial information, the three-dimensional spatial information takes the center point coordinates of the three-dimensional bounding box of the indoor structure corresponding to the indoor image sample, the length, width and height of the three-dimensional bounding box, and the unit vectors of the length, width and height as an example, and the first difference can be expressed as:

[0077]

[0078] Among them, in formula (1),

[0079]

[0080]

[0081] In the above formulas (1) to (4), N represents the total number of indoor image samples; n represents the nth indoor image sample, n=1, 2, ..., N. Represents the center point coordinates of the three-dimensional bounding box in the predicted three-dimensional space information of the indoor structure corresponding to the n-th indoor image sample; Represents the length, width and height of the three-dimensional bounding box in the predicted three-dimensional spatial information of the indoor structure corresponding to the n-th indoor image sample; and They represent the unit vectors of the length, width and height of the three-dimensional bounding box in the predicted three-dimensional spatial information of the indoor structure corresponding to the nth indoor image sample. (x n ,y n , z n ) represents the center point coordinates of the three-dimensional bounding box in the prior three-dimensional spatial information of the indoor structure corresponding to the n-th indoor image sample; (l n , w n ,h n ) represents the length, width and height of the three-dimensional bounding box in the prior three-dimensional spatial information of the indoor structure corresponding to the n-th indoor image sample; and They respectively represent the unit vectors of the length, width and height of the three-dimensional bounding box in the prior measured three-dimensional space information of the indoor structure corresponding to the n-th indoor image sample.

[0082] The second difference between the predicted probability and the prior probability can be expressed as: mean square error, variance or cross entropy between the predicted probability and the prior probability corresponding to each indoor image sample. For the embodiment using the cross entropy between the predicted probability and the prior probability corresponding to each indoor image sample, the second difference between the predicted probability and the prior probability can be expressed as:

[0083]

[0084] In the above formula (5), N represents the total number of indoor image samples; n represents the nth indoor image sample, n = 1, 2, ..., N. M represents the number of preset multiple indoor structure models; m represents the mth indoor structure model. mn Indicates the prior probability that the indoor structure corresponding to the nth indoor image sample is the mth indoor structure model; represents the predicted probability that the indoor structure corresponding to the nth indoor image sample is the mth indoor structure model. mn is 0 or 1; Any value between 0 and 1.

[0085] Based on the above first difference and second difference, the loss function of the indoor structure prediction model can be expressed as:

[0086] L 00 =L1+L2 (6).

[0087] Based on the above loss function, when training the indoor structure prediction model, the model parameters of the indoor structure model to be trained can be adjusted with the goal of minimizing the loss function shown in the above formula (6), which is the sum of the first difference and the second difference, to obtain the target indoor structure prediction model.

[0088] In this embodiment, the gradient descent method can be used to adjust the model parameters of the indoor structure prediction model to be trained. Specifically, it is assumed that θ is the model parameter set of the text matching model to be trained, and an initial value is randomly set. Further, the partial differential of each model parameter is calculated to obtain a partial differential set, that is, a gradient set. Further, the new model parameters can be obtained by updating the gradient corresponding to each model parameter in the gradient set, and this is cyclically executed until the model parameters minimize the value of the loss function. Among them, the indoor structure prediction model corresponding to the new model parameters is the indoor structure prediction model to be trained in the next cycle.

[0089] It should be noted that the model training process is generally a step-by-step learning process, and the architecture of the model obtained in each stage is the same, but the model parameters are different. The indoor structure prediction model to be trained in step 102-step 105 can be an indoor structure prediction model of any stage, and in the same cycle process, the indoor structure prediction model to be trained in step 102-step 105 is an indoor structure prediction model of the same stage.

[0090] In this embodiment, the indoor structure is pre-classified into a plurality of preset indoor structure models. In the process of training the indoor structure prediction model, the indoor structure model of the indoor structure corresponding to the indoor image sample is first predicted. When predicting the three-dimensional spatial information of the indoor structure of the indoor image sample, the predicted indoor structure model of the indoor structure corresponding to the indoor image sample is added as prior knowledge to the network for model training. In particular, the indoor structure model is added to the model for training the three-dimensional spatial information of the indoor structure, which increases the reference information for model training, helps to improve the accuracy of the trained indoor structure prediction model, and further helps to improve the accuracy of the three-dimensional spatial information of the indoor structure corresponding to the indoor image when the indoor structure prediction model is used online later.

[0091] The following is an exemplary description of an implementation method of using the above-mentioned indoor structure prediction model to predict the indoor structure corresponding to the indoor image.

[0092] Figure 6Schematic diagram of the process of determining the indoor structure provided by the embodiment of the present application. Figure 6 As shown, the method mainly includes:

[0093] 601. Obtain an indoor image to be processed.

[0094] 602. Use the first network model in the target indoor prediction model to extract features of the indoor image to obtain image features of the indoor image.

[0095] 603. Determine a target indoor structure model of the indoor structure corresponding to the indoor image from among a plurality of preset indoor structure models, using the second network model in the target indoor prediction model according to image features of the indoor image.

[0096] 604. Determine the three-dimensional spatial information of the indoor structure corresponding to the indoor image according to the image features of the indoor image and the target indoor structure model by using the third network model in the target indoor structure prediction model.

[0097] In this embodiment, the indoor image to be processed is mainly a two-dimensional image of an indoor scene, which can be an image collected for an indoor structure. Accordingly, the indoor structure corresponding to the indoor image is the indoor structure in the indoor scene collected by the indoor image. Based on the above-mentioned trained target indoor structure prediction model, for the indoor image to be processed in step 601, in step 602, the first network model in the target indoor prediction model can be used to extract features of the indoor image to obtain image features of the indoor image.

[0098] Further, in step 603, the target indoor structure model of the indoor structure corresponding to the indoor image is determined from a plurality of preset indoor structure models according to the image features of the indoor image by using the second network model in the target indoor prediction model.

[0099] Specifically, the second network model in the target indoor structure prediction model can be used to predict the probability that the indoor structure corresponding to the indoor image is a preset multiple indoor structure model according to the image features of the indoor image. For the description of the preset multiple indoor structure models, please refer to the relevant content of the above-mentioned model training method embodiment, which will not be repeated here.

[0100] After predicting the probability that the indoor structure model corresponding to the indoor image is various indoor structure models, the target indoor structure model can be determined from the preset multiple indoor structure models according to the probability that the indoor structure corresponding to the indoor image is the preset multiple indoor structure models. Optionally, according to the probability that the indoor structure corresponding to the indoor image is the preset multiple indoor structure models, the indoor structure model whose corresponding probability satisfies the set probability condition can be determined from the preset multiple indoor structure models as the target indoor structure model.

[0101] In some embodiments, the probability condition may be set as the maximum probability, then the indoor structure model with the maximum corresponding probability may be determined from the preset multiple indoor structure models according to the probability that the indoor structure corresponding to the indoor image is a preset multiple indoor structure model, and the indoor structure model with the maximum corresponding probability may be determined as the target indoor structure model. In some embodiments, the probability condition may be set as the probability greater than or equal to the set probability threshold, then the indoor structure model with the corresponding probability greater than or equal to the set probability threshold may be determined from the preset multiple indoor structure models according to the probability that the indoor structure corresponding to the indoor image is a preset multiple indoor structure model, and the indoor structure model with the corresponding probability greater than or equal to the set probability threshold may be determined as the target indoor structure model.

[0102] After determining the target indoor structure model, combined with Figure 3 and Figure 5 , the target indoor structure model and the image features of the indoor image can be input into the third network model in the target indoor structure prediction model; and the third network model in the target indoor structure prediction model can be used to determine the three-dimensional spatial information of the indoor structure corresponding to the indoor image according to the image features of the indoor image and the target indoor structure model.

[0103] In this embodiment, the indoor structure is pre-classified into a plurality of preset indoor structure models, and in the process of predicting the three-dimensional spatial information of the indoor structure of the indoor image, the target indoor structure model of the indoor structure corresponding to the indoor image is first predicted, and the predicted target indoor structure model of the indoor structure corresponding to the indoor image is added to the network as prior knowledge to predict the three-dimensional spatial information of the indoor structure of the indoor image. In particular, the indoor structure model is added when predicting the three-dimensional spatial information of the indoor structure, which increases the reference information for predicting the three-dimensional spatial information of the indoor structure, and helps to improve the prediction accuracy of the three-dimensional spatial information of the indoor structure corresponding to the indoor image.

[0104] In addition, the target indoor structure model provided in the embodiment of the present application is an end-to-end neural network model, which can obtain the three-dimensional spatial information of the indoor structure, that is, the three-dimensional spatial structure corresponding to the indoor image, from a single input indoor image in an end-to-end manner.

[0105] For the representation of the three-dimensional spatial information of the indoor structure, please refer to the relevant content of the above-mentioned model training method embodiment. For the three-dimensional spatial information of the indoor structure in the above-mentioned model training stage, the embodiment using the center point coordinates of the three-dimensional bounding box of the indoor structure, the length of the length, width and height of the three-dimensional bounding box, and the unit vectors of the length, width and height of the three-dimensional bounding box to represent it, the three-dimensional spatial information of the indoor structure can also be converted to the vertex coordinates of the three-dimensional bounding box to be compatible with the traditional solution.

[0106] Specifically, the vertex position information of the three-dimensional bounding box can be calculated according to the center point coordinates of the three-dimensional bounding box corresponding to the indoor structure of the indoor image, the lengths of the length, width and height of the three-dimensional bounding box, and the unit vectors of the length, width and height of the three-dimensional bounding box. Specifically, the vertex position information of the three-dimensional bounding box, that is, the vertex coordinates of the vertex position information of the three-dimensional bounding box, can be obtained by offsetting the length, width and height of the three-dimensional bounding box in the direction of the unit vectors of the length, width and height of the three-dimensional bounding box with the center point coordinates of the three-dimensional bounding box as a reference.

[0107] In some embodiments of the present application, in order to ensure that the three-dimensional spatial information of the indoor structure corresponding to the indoor image output by the target indoor structure prediction model is consistent with the indoor structure corresponding to the indoor image, the three-dimensional spatial information of the indoor structure corresponding to the indoor image can also be mapped to the indoor image. To map the three-dimensional spatial information of the indoor structure corresponding to the indoor image to the indoor image, it is also necessary to know the position information of the image acquisition device corresponding to the indoor image. Among them, the image acquisition device corresponding to the indoor image is the position information of the image acquisition device in the process of acquiring the indoor image. The position information of the image acquisition device includes: the position information and attitude information of the image acquisition device. Among them, the position information of the image acquisition device can be characterized by the position information of the image acquisition device in the camera coordinate system, compared with the position deviation of the image acquisition device in other set coordinate systems (such as the world coordinate system). The attitude information of the image acquisition device can be characterized by the rotation angle of the coordinate axis of the camera coordinate system corresponding to the image acquisition device compared to the set coordinate system (such as the world coordinate system), and the coordinate representation of the focus of the image acquisition device in the set coordinate system (such as the world coordinate system).

[0108] Wherein, in an embodiment of model training for the above-mentioned indoor structure prediction model, the indoor image samples may include: indoor image samples acquired by an image acquisition device for a real indoor scene, and virtual indoor scene images (i.e., virtual indoor image samples) rendered by a computing device. Wherein, the pose information of the image acquisition device corresponding to the indoor image samples acquired by the image acquisition device for a real indoor scene is the pose information of the image acquisition device in the process of acquiring the indoor image samples. The pose information of the image acquisition device corresponding to the virtual indoor image samples is the pose information of the virtual camera used in the process of rendering the virtual indoor image samples. Accordingly, in the above-mentioned step 101, the prior information pre-annotated by the indoor image samples may also include: the pose information of the image acquisition device corresponding to the indoor image samples, which is defined as the prior pose information.

[0109] In order to obtain the pose information of the image acquisition device corresponding to the indoor image, the above-mentioned target indoor structure prediction model can also be used to predict the pose information of the image acquisition device corresponding to the indoor image. Accordingly, when the indoor structure prediction model is trained, the indoor structure prediction model can also be trained to obtain the pose information of the image acquisition device corresponding to the indoor image.

[0110] Specifically, combined Figure 5 and Figure 7 In the process of training the target indoor structure prediction model, the third network model in the indoor structure model to be trained can also be used to determine the predicted pose information of the image acquisition devices corresponding to the multiple indoor image samples according to the image features of the multiple indoor samples and the predicted indoor structure model of the indoor structure corresponding to the multiple indoor image samples, that is, to predict the pose information of the image acquisition device corresponding to each indoor image sample. The predicted pose information is called predicted pose information.

[0111] Accordingly, in the above step 106, when adjusting the model parameters of the indoor structure model to be trained, the first difference between the predicted three-dimensional spatial information and the prior three-dimensional spatial information, the second difference between the predicted probability and the prior probability, and the third difference between the predicted posture information and the prior posture information can be used. The third difference between the predicted posture information and the prior posture information includes: the difference between the predicted posture information of the image acquisition device corresponding to each indoor image sample and the prior posture information of the image acquisition device corresponding to the same indoor image sample.

[0112] In this embodiment, the sum of the first difference, the second difference and the third difference can constitute a loss function. Among them, the third difference between the predicted pose information and the prior pose information can be expressed as: the mean square error or variance between the predicted pose information of the image acquisition device corresponding to each indoor image sample and the prior pose information of the image acquisition device corresponding to the same indoor image sample. For the third difference between the predicted pose information and the prior pose information, the embodiment characterized by the mean square error between the predicted pose information of the image acquisition device corresponding to each indoor image sample and the prior pose information of the image acquisition device corresponding to the same indoor image sample, the third difference can be expressed as:

[0113]

[0114] In the above formula (7), N represents the total number of indoor image samples; n represents the nth indoor image sample, n=1, 2, ..., N. Represents the predicted pose information of the image acquisition device corresponding to the nth indoor image sample. n Represents the prior pose information of the image acquisition device corresponding to the nth indoor image sample.

[0115] Based on the above first difference, second difference and third difference, if Figure 5 As shown in Figure 2, the loss function of the indoor structure prediction model can also be expressed as:

[0116] L 01 =L1+L2+L3 (8).

[0117] Based on the above loss function formula (8), when training the indoor structure prediction model, the model parameters of the indoor structure model to be trained can be adjusted with the goal of minimizing the loss function shown in the above formula (8), which is the sum of the first difference, the second difference and the third difference, to obtain the target indoor structure prediction model.

[0118] For the specific implementation of adjusting the model parameters of the indoor structure model to be trained, reference may be made to the relevant contents of the above embodiments, which will not be described in detail here.

[0119] Based on the target indoor structure prediction model trained according to the first difference, the second difference and the third difference, the target indoor structure prediction model can determine the position information of the image acquisition device corresponding to the indoor image in addition to the three-dimensional spatial information of the indoor structure corresponding to the indoor image. Specifically, during the online use of the target indoor structure prediction model, the third network model in the target indoor structure prediction model can also be used to determine the position information of the image acquisition device corresponding to the indoor image according to the image features of the indoor image and the target indoor structure model.

[0120] In this embodiment, if Figure 8 As shown, the three-dimensional space information of the indoor structure corresponding to the indoor image can also be projected into a two-dimensional space image according to the position information of the image acquisition device corresponding to the indoor image; and the two-dimensional space image is added to the indoor image to obtain a target indoor image with the two-dimensional space image added. Optionally, a transparent layer of the two-dimensional space image can be added to the indoor image to obtain a target indoor image with the two-dimensional space image added.

[0121] Further, the target indoor image with the two-dimensional space image added can be output. Specifically, the target indoor image with the two-dimensional space image added can be displayed for the user to check whether the indoor structure corresponding to the two-dimensional space image is consistent with the indoor image. Alternatively, in some embodiments, the above-mentioned indoor structure determination method can also be executed on the server device, and the target indoor image with the two-dimensional space image added can be sent to the user's terminal device. The terminal device can display the target indoor image with the two-dimensional space image added, so that the user can check whether the indoor structure corresponding to the two-dimensional space image is consistent with the indoor image.

[0122] In order to make the indoor structure corresponding to the two-dimensional space image consistent with the indoor image, that is, the three-dimensional space information output by the target indoor structure prediction model is consistent with the indoor structure corresponding to the indoor image, an adjustment function of the three-dimensional space information of the indoor structure corresponding to the indoor image can also be provided, so that the user can independently adjust the three-dimensional space information of the indoor structure corresponding to the indoor image based on the two-dimensional space image.

[0123] Specifically, if the user determines that the indoor structure corresponding to the two-dimensional space image does not match the indoor image through the displayed target indoor image with the two-dimensional space image added, the user can autonomously adjust the two-dimensional space image on the target indoor image based on the target indoor image with the two-dimensional space image added, so that the indoor structure corresponding to the two-dimensional space image matches the indoor image. Accordingly, for a computing device that executes the above-mentioned indoor structure determination method, adjustment information for the two-dimensional space image can be obtained. The adjustment information for the two-dimensional space image mainly refers to the adjustment information of the structure outline shown in the two-dimensional space image.

[0124] Furthermore, the three-dimensional spatial information of the indoor structure corresponding to the indoor image can be adjusted according to the adjustment information of the two-dimensional spatial image and the posture information of the image acquisition device corresponding to the indoor image, so that the three-dimensional spatial information of the indoor structure corresponding to the indoor image is consistent with the indoor structure corresponding to the indoor image, which can further improve the accuracy of the determined three-dimensional spatial information of the indoor structure corresponding to the indoor image.

[0125] Based on the target indoor structure prediction model provided in the embodiment of the present application, the inventor of the present application uses Fig. 9 The indoor image to be processed is shown to test the target indoor structure prediction model, and the Fig. 9 The indoor structure model of the indoor structure corresponding to the indoor image to be processed and the three-dimensional bounding box of the indoor structure corresponding to the indoor image to be processed are shown. Fig. 9 It can be seen that by using the target indoor structure prediction model provided in the embodiment of the present application, the three-dimensional spatial information of the indoor structure corresponding to the indoor image to be processed can be accurately determined.

[0126] It is worth noting that the data processing method provided in the embodiment of the present application can be deployed on any computing device. In the present embodiment, the data processing method may include: a model training method and / or an indoor structure determination method. Optionally, the model training method and / or indoor structure determination method provided in the embodiment of the present application can also be deployed on a cloud server as a Software as a Service (SaaS) application. For a cloud server deployed with the SaaS application, the steps in the above-mentioned model training method and / or indoor structure determination method can be executed in response to a request to call a target service. Among them, the target service refers to a service that provides the above-mentioned model training method and / or indoor structure determination method.

[0127] Specifically, for a cloud server, in response to a request to call a target service, the processing resources corresponding to the target service can be determined. The processing resources corresponding to the target service refer to the processing resources required to execute the above rendering method, including but not limited to: processor resources, memory resources, and input / output (IO) resources.

[0128] In this embodiment, the above-mentioned model training method and / or indoor structure determination method can be deployed on a cloud server to provide users with model training and / or indoor structure prediction services, i.e., target services. Optionally, the cloud service end may provide an application programming interface (API) to the user. The service requester (i.e., the user) may call the API to call the target service. Accordingly, the request to call the target service is implemented as a call event generated by calling the API. The service requester (i.e., the user) may also call the target service through remote procedure call (RPC) or remote direct memory access (RDMA) technology.

[0129] For the cloud server, in response to a request to call the target service, the processing resources corresponding to the target service can be determined; and the processing resources corresponding to the target service can be used to execute the above-mentioned model training method and / or indoor structure determination method.

[0130] It should be noted that the execution subject of each step of the method provided in the above embodiment can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 101 and 102 can be device A; for another example, the execution subject of step 101 can be device A, and the execution subject of step 102 can be device B; and so on.

[0131] In addition, in some of the processes described in the above embodiments and the accompanying drawings, multiple operations appearing in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this document or may be executed in parallel, and the serial numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel.

[0132] Accordingly, an embodiment of the present application also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed by one or more processors, the one or more processors are caused to execute the steps in the above-mentioned model training method and / or indoor structure determination method.

[0133] Fig.10 This is a schematic diagram of the structure of a computing device provided in an embodiment of the present application. Fig.10 As shown, the computing device includes: a memory 100a and a processor 100b. The memory 100a is used to store computer programs.

[0134] The processor 100b is coupled to the memory 100a, and is used to execute a computer program for: obtaining an indoor image to be processed; using a first network model in a target indoor structure prediction model to perform feature extraction on the indoor image to obtain image features of the indoor image; using a second network model in the target indoor structure prediction model to determine, according to the image features of the indoor image, a target indoor structure model of the indoor structure corresponding to the indoor image from a plurality of preset indoor structure models; using a third network model in the target indoor structure prediction model to determine, according to the image features of the indoor image and the target indoor structure model, three-dimensional spatial information of the indoor structure corresponding to the indoor image.

[0135] In some embodiments, when the processor 100b uses the second network model in the target indoor structure prediction model to predict the target indoor structure model of the indoor structure corresponding to the indoor image from a plurality of preset indoor structure models based on the image features of the indoor image, it is specifically used to: use the second network model in the target indoor structure prediction model to predict the probability that the indoor structure corresponding to the indoor image is one of a plurality of preset indoor structure models based on the image features of the indoor image; determine the target indoor structure model from the plurality of preset indoor structure models based on the probability that the indoor structure corresponding to the indoor image is one of a plurality of preset indoor structure models.

[0136] In other embodiments, when the processor 100b uses the third network model in the target indoor structure prediction model to determine the three-dimensional spatial information of the indoor structure corresponding to the indoor image according to the image features of the indoor image and the target indoor structure model, it is specifically used to: use the third network model in the target indoor structure prediction model to determine the three-dimensional spatial information of the three-dimensional bounding box of the indoor structure corresponding to the indoor image according to the image features of the indoor image and the target indoor structure model, as the three-dimensional spatial information of the indoor structure corresponding to the indoor image; wherein the three-dimensional spatial information of the three-dimensional bounding box of the indoor structure corresponding to the indoor image includes: the coordinates of the center point of the three-dimensional bounding box, the length, width and height of the three-dimensional bounding box, and the unit vectors of the length, width and height of the three-dimensional bounding box.

[0137] Correspondingly, the processor 100b is further configured to calculate vertex position information of the 3D bounding box according to the center point coordinates of the 3D bounding box, the length, width and height of the 3D bounding box and the unit vectors of the length, width and height of the 3D bounding box.

[0138] In some embodiments of the present application, the processor 100b is also used to: use the third network model in the target indoor structure prediction model to determine the pose information of the image acquisition device corresponding to the indoor image according to the image features of the indoor image and the target indoor structure model; project the three-dimensional spatial information of the indoor structure corresponding to the indoor image into a two-dimensional spatial image according to the pose information of the image acquisition device corresponding to the indoor image; and add the two-dimensional spatial image to the indoor image to obtain the target indoor image with the two-dimensional spatial image added, and output the target indoor image.

[0139] In some embodiments, when outputting the target indoor image, the processor 100b is specifically used to: send the target indoor image to a terminal device requesting indoor structure prediction through the communication component 100c; or display the target indoor image through the display component 100d.

[0140] Furthermore, the processor 100b is also used to: obtain adjustment information for the two-dimensional space image; and adjust the three-dimensional space information of the indoor structure corresponding to the indoor image according to the adjustment information of the two-dimensional space image and the posture information of the image acquisition device corresponding to the indoor image.

[0141] In some other embodiments of the present application, the processor 100b is further used to: obtain a plurality of indoor image samples pre-labeled with prior information; the prior information includes: prior three-dimensional spatial information of indoor structures corresponding to the plurality of indoor image samples and prior probabilities that the plurality of indoor image samples are preset multiple indoor structure models; use the first network model in the indoor structure prediction model to be trained to perform feature extraction on the plurality of indoor image samples to obtain image features of the plurality of indoor image samples; use the second network model in the indoor structure prediction model to be trained to determine, according to the image features of the plurality of indoor image samples, the prediction probabilities that the plurality of indoor image samples are preset multiple indoor structure models; According to the prediction probabilities that the multiple indoor image samples are multiple indoor structure models, the predicted indoor structure models of the indoor structures corresponding to the multiple indoor image samples are determined; using the third network model in the indoor structure prediction model to be trained, according to the image features of the multiple indoor image samples and the predicted indoor structure models of the indoor structures corresponding to the multiple indoor image samples, the predicted three-dimensional spatial information of the indoor structures respectively corresponding to the multiple indoor image samples is determined; and according to the first difference between the predicted three-dimensional spatial information and the prior three-dimensional spatial information, and the second difference between the predicted probability and the prior probability, the model parameters of the indoor structure model to be trained are adjusted to obtain the target indoor structure prediction model.

[0142] Optionally, the prior information also includes: prior pose information of the image acquisition device corresponding to the multiple indoor image samples. Accordingly, the processor 100b is also used to: use the third network model in the indoor structure prediction model to be trained, according to the image features of the multiple indoor samples and the predicted indoor structure model of the indoor structure corresponding to the multiple indoor image samples, to determine the predicted pose information of the image acquisition device corresponding to the multiple indoor image samples.

[0143] Correspondingly, when the processor 100b adjusts the model parameters of the indoor structure model to be trained according to the first difference between the predicted three-dimensional spatial information and the prior three-dimensional spatial information, and the second difference between the predicted probability and the prior probability, it is specifically used to: adjust the model parameters of the indoor structure model to be trained according to the first difference, the second difference and the third difference between the predicted posture information and the prior posture information to obtain the target indoor structure prediction model.

[0144] Specifically, when the processor 100b adjusts the model parameters of the indoor structure model to be trained according to the first difference, the second difference and the third difference between the predicted posture information and the prior posture information, it is specifically used to: adjust the model parameters of the indoor structure model to be trained with the goal of minimizing the sum of the first difference, the second difference and the third difference to obtain the target indoor structure prediction model.

[0145] Among them, the above-mentioned multiple indoor image samples include: indoor image samples collected by the image acquisition device, and rendered virtual indoor image samples; the posture information of the image acquisition device corresponding to the virtual indoor image samples is the posture information of the virtual camera used in the process of rendering the virtual indoor image samples; among them, the number of virtual indoor image samples is less than or equal to the indoor image samples collected by the image acquisition device.

[0146] In some embodiments, when the processor 100b uses the third network model in the indoor structure prediction model to be trained to determine the predicted three-dimensional spatial information of the indoor structures corresponding to the multiple indoor image samples respectively according to the image features of the multiple indoor samples and the predicted indoor structure models of the indoor structures corresponding to the multiple indoor image samples, it is specifically used to: for any indoor image sample among the multiple indoor image samples, using the third network model, according to the image features of any indoor image sample and the predicted indoor structure models of the indoor structures corresponding to the multiple indoor image samples, determine the three-dimensional spatial information of the three-dimensional bounding box of the indoor structure corresponding to any indoor image sample respectively as the predicted three-dimensional spatial information of the indoor structure corresponding to any indoor image sample; wherein the three-dimensional spatial information of the three-dimensional bounding box includes: the coordinates of the center point of the three-dimensional bounding box, the length, width and height of the three-dimensional bounding box, and the unit vectors of the length, width and height of the three-dimensional bounding box. Accordingly, the prior three-dimensional spatial information of the indoor structure of any indoor image sample includes: the coordinates of the center point of the prior three-dimensional bounding box of the indoor structure of any indoor image sample, the prior length, width and height of the prior three-dimensional bounding box, and the unit vectors of the prior length, width and height of the prior three-dimensional bounding box.

[0147] In some embodiments of the present application, the computing device may be implemented as a cloud server. Accordingly, the processor 100b is used to: in response to a request to call a target service, determine the processing resources corresponding to the target service; the target service refers to a service that provides indoor structure prediction and / or model training services; and use the processing resources corresponding to the target service to perform the steps in the indoor structure determination method and / or model training method provided in the above embodiments.

[0148] In some optional embodiments, such as Fig.10 As shown, the computing device may also include optional components such as a power supply component 100e and an audio component 100f. Fig.10 The components are shown schematically only and do not necessarily include Fig.10 The components shown do not necessarily mean that the computing device can only include Fig.10 Components shown.

[0149] The computing device provided in this embodiment pre-classifies indoor structures into a plurality of preset indoor structure models, and in the process of predicting the three-dimensional spatial information of the indoor structure of the indoor image, first predicts the target indoor structure model of the indoor structure corresponding to the indoor image, and adds the predicted target indoor structure model of the indoor structure corresponding to the indoor image as prior knowledge to the network to predict the three-dimensional spatial information of the indoor structure of the indoor image. In particular, the indoor structure model is added when predicting the three-dimensional spatial information of the indoor structure, which increases the reference information for predicting the three-dimensional spatial information of the indoor structure, and helps to improve the prediction accuracy of the three-dimensional spatial information of the indoor structure corresponding to the indoor image.

[0150] In an embodiment of the present application, the memory is used to store a computer program and can be configured to store various other data to support operations on the device where it is located. Among them, the processor can execute the computer program stored in the memory to implement the corresponding control logic. The memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random-Access Memory, SRAM), electrically erasable programmable read only memory (Electrically Erasable Programmable Read Only Memory, EEPROM), erasable programmable read only memory (Electrical Programmable Read Only Memory, EPROM), programmable read only memory (Programmable Read Only Memory, PROM), read only memory (Read Only Memory, ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0151] In the embodiment of the present application, the processor can be any hardware processing device that can execute the logic of the above method. Optionally, the processor can be a central processing unit (CPU), a graphics processing unit (GPU) or a microcontroller unit (MCU); it can also be a field programmable gate array (FPGA), a programmable array logic device (PAL), a general array logic device (GAL), a complex programmable logic device (CPLD) and other programmable devices; or an advanced reduced instruction set (RISC) processor (Advanced RISC Machines, ARM) or a system on chip (SoC), etc., but not limited to this.

[0152] In an embodiment of the present application, the communication component is configured to facilitate wired or wireless communication between the device in which it is located and other devices. The device in which the communication component is located can access a wireless network based on a communication standard, such as Wireless Fidelity (WiFi), 2G or 3G, 4G, 5G or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component can also be based on Near Field Communication (NFC) technology, Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (BT) technology or other technologies.

[0153] In an embodiment of the present application, the display component may include a liquid crystal display (LCD) and a touch panel (TP). If the display component includes a touch panel, the display component may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0154] In an embodiment of the present application, a power supply component is configured to provide power to various components of the device in which it is located. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which the power supply component is located.

[0155] In an embodiment of the present application, the audio component may be configured to output and / or input audio signals. For example, the audio component includes a microphone (Microphone, MIC), and when the device where the audio component is located is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal may be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker for outputting an audio signal. For example, for a device with a language interaction function, voice interaction with a user can be achieved through an audio component.

[0156] It should be noted that the descriptions such as “first” and “second” in this article are used to distinguish different messages, devices, modules, etc., and do not represent the order of precedence, nor do they limit “first” and “second” to different types.

[0157] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, compact disc read-only memory (Compact Disc Read-Only Memory, CD-ROM), optical storage, etc.) that contain computer-usable program code.

[0158] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0159] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0160] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0161] In a typical configuration, a computing device includes one or more processors (CPU, etc.), input / output interfaces, network interfaces, and memory.

[0162] Memory may include non-permanent storage in a computer-readable medium, random-access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0163] The storage medium of a computer is a readable storage medium, which may also be referred to as a readable medium. The readable storage medium includes permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. The information can be a computer-readable instruction, a data structure, a module of a program, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0164] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of further restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the above elements.

[0165] The above contents are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A method for determining indoor structure, characterized in that: include: Acquire an indoor image to be processed; Using the first network model in the target indoor structure prediction model, extracting features from the indoor image to obtain image features of the indoor image; Determining a target indoor structure model of the indoor structure corresponding to the indoor image from a plurality of preset indoor structure models according to image features of the indoor image using the second network model in the target indoor structure prediction model; The third network model in the target indoor structure prediction model is used to determine the three-dimensional spatial information of the indoor structure corresponding to the indoor image according to the image features of the indoor image and the target indoor structure model.

2. The method according to claim 1, characterized in that The method of using the second network model in the target indoor structure prediction model to predict the target indoor structure model of the indoor structure corresponding to the indoor image from a plurality of preset indoor structure models according to the image features of the indoor image includes: Using the second network model in the target indoor structure prediction model, according to the image features of the indoor image, predict the probability that the indoor structure corresponding to the indoor image is a plurality of preset indoor structure models; According to the probability that the indoor structure corresponding to the indoor image is a plurality of preset indoor structure models, the target indoor structure model is determined from the plurality of preset indoor structure models.

3. The method according to claim 1 or 2, characterized in that: The method of using the third network model in the target indoor structure prediction model to determine the three-dimensional spatial information of the indoor structure corresponding to the indoor image according to the image features of the indoor image and the target indoor structure model includes: Using the third network model in the target indoor structure prediction model, according to the image features of the indoor image and the target indoor structure model, determine the three-dimensional spatial information of the three-dimensional bounding box of the indoor structure corresponding to the indoor image as the three-dimensional spatial information of the indoor structure corresponding to the indoor image; wherein the three-dimensional spatial information of the three-dimensional bounding box of the indoor structure corresponding to the indoor image includes: the coordinates of the center point of the three-dimensional bounding box, the length, width and height of the three-dimensional bounding box and the unit vectors of the length, width and height of the three-dimensional bounding box; The method further comprises: The vertex position information of the three-dimensional bounding box is calculated according to the coordinates of the center point of the three-dimensional bounding box, the length, width and height of the three-dimensional bounding box and the unit vectors of the length, width and height of the three-dimensional bounding box.

4. The method according to claim 1 or 2, characterized in that: Also includes: Determine the position information of the image acquisition device corresponding to the indoor image according to the image features of the indoor image and the target indoor structure model by using the third network model in the target indoor structure prediction model; According to the posture information of the image acquisition device corresponding to the indoor image, projecting the three-dimensional space information of the indoor structure corresponding to the indoor image into a two-dimensional space image; The two-dimensional space image is added to the indoor image to obtain a target indoor image added with the two-dimensional space image, and the target indoor image is output.

5. The method according to claim 4, characterized in that Also includes: Acquiring adjustment information for the two-dimensional space image; According to the adjustment information of the two-dimensional space image and the posture information of the image acquisition device corresponding to the indoor image, the three-dimensional space information of the indoor structure corresponding to the indoor image is adjusted.

6. The method according to claim 4, characterized in that Also includes: Acquire a plurality of indoor image samples pre-labeled with prior information; the prior information includes: prior three-dimensional spatial information of indoor structures corresponding to the plurality of indoor image samples and prior probabilities that the plurality of indoor image samples are preset multiple indoor structure models; Using a first network model in the indoor structure prediction model to be trained, extracting features from the plurality of indoor image samples to obtain image features of the plurality of indoor image samples; Using the second network model in the indoor structure prediction model to be trained, and according to the image features of the multiple indoor image samples, determining the prediction probability that the multiple indoor image samples are preset multiple indoor structure models; Determining predicted indoor structure models of indoor structures corresponding to the multiple indoor image samples according to the predicted probabilities that the multiple indoor image samples are the multiple indoor structure models; Determine predicted three-dimensional spatial information of indoor structures corresponding to the plurality of indoor image samples respectively according to image features of the plurality of indoor image samples and predicted indoor structure models of indoor structures corresponding to the plurality of indoor image samples by using a third network model in the indoor structure prediction model to be trained; According to a first difference between the predicted three-dimensional spatial information and the prior three-dimensional spatial information, and a second difference between the predicted probability and the prior probability, the model parameters of the indoor structure model to be trained are adjusted to obtain a target indoor structure prediction model.

7. The method according to claim 6, characterized in that The prior information also includes: prior pose information of the image acquisition device corresponding to the multiple indoor image samples; the method also includes: Determine predicted position information of image acquisition devices corresponding to the plurality of indoor image samples respectively according to the image features of the plurality of indoor samples and the predicted indoor structure models of the indoor structures corresponding to the plurality of indoor image samples by using the third network model in the indoor structure prediction model to be trained; The adjusting the model parameters of the indoor structure model to be trained according to the first difference between the predicted three-dimensional spatial information and the prior three-dimensional spatial information, and the second difference between the predicted probability and the prior probability, comprises: According to the first difference, the second difference and the third difference between the predicted pose information and the prior pose information, the model parameters of the indoor structure model to be trained are adjusted to obtain the target indoor structure prediction model.

8. The method according to claim 6, characterized in that The method of using the third network model in the indoor structure prediction model to be trained to determine predicted three-dimensional spatial information of the indoor structures corresponding to the multiple indoor image samples respectively according to the image features of the multiple indoor samples and the predicted indoor structure models of the indoor structures corresponding to the multiple indoor image samples comprises: For any indoor image sample among the multiple indoor image samples, using the third network model, according to the image features of the any indoor image sample and the predicted indoor structure model of the indoor structure corresponding to the multiple indoor image samples, determine the three-dimensional spatial information of the three-dimensional bounding box of the indoor structure corresponding to the any indoor image sample as the predicted three-dimensional spatial information of the indoor structure corresponding to the any indoor image sample; The three-dimensional spatial information of the three-dimensional bounding box includes: the coordinates of the center point of the three-dimensional bounding box, the length, width and height of the three-dimensional bounding box, and the unit vectors of the length, width and height of the three-dimensional bounding box; The prior three-dimensional spatial information of the indoor structure of any indoor image sample includes: the center point coordinates of the prior three-dimensional bounding box of the indoor structure of any indoor image sample, the prior length, width and height of the prior three-dimensional bounding box, and the unit vectors of the prior length, width and height of the prior three-dimensional bounding box.

9. The method according to claim 7, characterized in that: The adjusting the model parameters of the indoor structure model to be trained according to the first difference, the second difference and the third difference between the predicted posture information and the priori posture information includes: With the goal of minimizing the sum of the first difference, the second difference and the third difference, the model parameters of the indoor structure model to be trained are adjusted to obtain the target indoor structure prediction model.

10. A model training method, characterized in that: include: Acquire multiple indoor image samples pre-labeled with prior information; The prior information includes: prior three-dimensional spatial information of indoor structures corresponding to the multiple indoor image samples and prior probabilities that the indoor structures of the multiple indoor image samples are preset multiple indoor structure models; Using a first network model in the indoor structure prediction model to be trained, extracting features from the plurality of indoor image samples to obtain image features of the plurality of indoor image samples; Using the second network model in the indoor structure prediction model to be trained, and according to the image features of the multiple indoor image samples, determining the prediction probability that the multiple indoor image samples are preset multiple indoor structure models; Determining predicted indoor structure models of indoor structures corresponding to the multiple indoor image samples according to the predicted probabilities that the multiple indoor image samples are the multiple indoor structure models; Determine predicted three-dimensional spatial information of indoor structures corresponding to the plurality of indoor image samples respectively according to image features of the plurality of indoor image samples and predicted indoor structure models of indoor structures corresponding to the plurality of indoor image samples by using a third network model in the indoor structure prediction model to be trained; According to a first difference between the predicted three-dimensional spatial information and the prior three-dimensional spatial information, and a second difference between the predicted probability and the prior probability, the model parameters of the indoor structure model to be trained are adjusted to obtain a target indoor structure prediction model.

11. The method according to claim 10, characterized in that The prior information also includes: prior pose information of the image acquisition device corresponding to the multiple indoor image samples; the method also includes: Determine predicted position information of image acquisition devices corresponding to the plurality of indoor image samples respectively according to the image features of the plurality of indoor samples and the predicted indoor structure models of the indoor structures corresponding to the plurality of indoor image samples by using the third network model in the indoor structure prediction model to be trained; The adjusting the model parameters of the indoor structure model to be trained according to the first difference between the predicted three-dimensional spatial information and the prior three-dimensional spatial information, and the second difference between the predicted probability and the prior probability, comprises: According to the first difference, the second difference and the third difference between the predicted pose information and the prior pose information, the model parameters of the indoor structure model to be trained are adjusted to obtain the target indoor structure prediction model.

12. A data processing method, applicable to a cloud server, characterized in that: The method comprises: In response to a request to call a target service, determining a processing resource corresponding to the target service; the target service is a service that provides indoor structure prediction and / or model training services; Utilize the processing resources corresponding to the target service to execute the steps in the method described in any one of claims 1-11.

13. A computing device, characterized in that: include: A memory and a processor; wherein the memory is used to store a computer program; The processor is coupled to the memory and configured to execute the computer program to perform the steps of the method according to any one of claims 1 to 11.

14. A computer-readable storage medium storing computer instructions, characterized in that: When the computer instructions are executed by one or more processors, the one or more processors are caused to execute the steps in the method according to any one of claims 1 to 11.

Images