Method, device and electronic equipment for predicting indoor space layout

Abstract

The embodiments of the present application relate to the field of image processing technology, and in particular to a method, device, and electronic device for predicting indoor space layout. In the method for predicting indoor space layout, the environmental image of the indoor space to be predicted is input into a spatial layout prediction model to obtain a target segmentation image of the indoor space to be predicted, and the target segmentation image includes at least two segmentation regions for characterizing the spatial layout of the indoor space to be predicted. The loss function of the spatial layout prediction model includes at least two of a segmentation loss function, a boundary loss function, and a corner loss function, so that the spatial layout prediction model takes into account features such as segmentation regions, boundaries, and boundary corners to predict the spatial layout of the indoor environment, thereby improving the accuracy of predicting the indoor space layout.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to a method, device, and electronic device for predicting indoor space layout. Background Art

[0002] The fundamental goal of predicting indoor space layout is to recover the layout structure of an indoor space from a given image of the indoor space. Currently, spatial layout prediction models used to predict indoor space layout exhibit jagged edges and sudden changes at the boundaries of each segmented region, resulting in inaccurate predictions of the indoor space layout at these boundaries. Summary of the Invention

[0003] The embodiments of the present application provide a method, device, and electronic device for predicting indoor space layout, which can improve the accuracy of predicting indoor space layout.

[0004] To solve the above technical problems, a technical solution adopted in an embodiment of the present application is to provide a method for predicting indoor space layout, comprising:

[0005] Obtain an environmental image of the indoor space to be predicted;

[0006] The environment image is input into a spatial layout prediction model to obtain a target segmentation image of the indoor space to be predicted, wherein the target segmentation image includes at least two segmentation areas, and the at least two segmentation areas are used to represent the spatial layout of the indoor space to be predicted.

[0007] To solve the above technical problems, another technical solution adopted in the embodiments of the present application is to provide a device for predicting indoor space layout, comprising:

[0008] The first acquisition module is used to acquire an environmental image of the indoor space to be predicted;

[0009] The first input module is used to input the environmental image into a spatial layout prediction model to obtain a target segmentation image of the indoor space to be predicted, wherein the target segmentation image includes at least two segmentation areas, and the at least two segmentation areas are used to represent the spatial layout of the indoor space to be predicted.

[0010] To solve the above technical problems, another technical solution adopted in the embodiment of the present application is to provide an electronic device, including:

[0011] at least one processor; and

[0012] A memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can be used to execute the above-mentioned method for predicting the indoor space layout.

[0013] To solve the above technical problems, another technical solution adopted in the embodiment of the present application is: providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used by a processor to execute the above method for predicting indoor space layout.

[0014] Embodiments of the present application provide a method, apparatus, and electronic device for predicting indoor space layout. By inputting an acquired environmental image of the indoor space to be predicted into a spatial layout prediction model, a target segmented image of the indoor space to be predicted is obtained. The target segmented image includes at least two segmented regions used to characterize the spatial layout of the indoor space to be predicted. The loss function of the spatial layout prediction model includes at least two of a segmentation loss function, a boundary loss function, and a corner loss function. This allows the spatial layout prediction model to consider features such as segmented regions, boundaries, and boundary corners to predict the spatial layout of the indoor environment. The model is capable of accurately predicting the boundaries of each segmented region and accurately determining the corner points of each segmented region, thereby improving the accuracy of predicting the indoor space layout. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] One or more embodiments are exemplarily illustrated by pictures in the corresponding drawings. These exemplifications do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings are represented as similar elements. Unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0016] Figure 1 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0017] Figure 2 This is a flow chart of a method for predicting indoor space layout provided by an embodiment of the present application;

[0018] Figure 3 It is the correspondence diagram between the standard segmentation image and the standard boundary image;

[0019] Figure 4 Schematic diagram of a device for predicting indoor space layout provided in an embodiment of the present application;

[0020] Figure 5 is a structural schematic diagram of a device for predicting indoor space layout provided by another embodiment of the present application;

[0021] Figure 6This is a structural diagram of a device for predicting indoor space layout provided in another embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to make the technical solutions of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0023] It should be noted that when an element is described as being "connected" to another element, it may be directly connected to the other element, or one or more intermediate elements may be present therebetween. In addition, the technical features involved in the various embodiments of the present application described below may be combined with each other as long as they do not conflict with each other.

[0024] The present application provides a method and apparatus for predicting indoor space layout. The method and apparatus are applied to an electronic device, thereby enabling the electronic device to obtain the spatial layout of the indoor space to be predicted by inputting an acquired environmental image of the indoor space to be predicted into a spatial layout prediction model. Since the loss function of the spatial layout prediction model includes at least two of a segmentation loss function, a boundary loss function, and a corner loss function, the spatial layout prediction model can accurately predict the boundaries of each segmented area and accurately determine the corner points of each segmented area, thereby enabling the electronic device to improve the accuracy of predicting the indoor space layout.

[0025] It is understandable that in the present application, the electronic device may be a mobile phone, tablet computer, PC or other device with image acquisition function.

[0026] Below, the present application will be described through specific embodiments.

[0027] See also Figure 1 , is a hardware structure diagram of an electronic device provided in an embodiment of the present application, the electronic device includes: a memory 10, a processor 20, a shooting module 30 and a display module 40, the processor 20 is connected to the memory 10, the shooting module 30 and the display module 40 respectively. It should be understood by those skilled in the art that, Figure 1 The structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine or separate some components, or adopt a different component arrangement.

[0028] The memory 10 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, instructions, and modules.

[0029] The memory 10 may include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function; the data storage area may store data created based on the use of the electronic device, and may also store preset data.

[0030] In addition, the memory 10 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device.

[0031] In some embodiments, the memory 10 may optionally include a memory remote from the processor 20, which may be connected to the electronic device via a network, including but not limited to the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0032] The processor 20 is the control center of the electronic device, which can use various interfaces and lines to connect various parts of the entire electronic device. By running or executing non-volatile software programs, instructions and modules stored in the memory 10, and calling data stored in the memory 10, it performs various functions of the electronic device and processes data, thereby controlling the electronic device as a whole, such as realizing the method for predicting the indoor space layout described in any embodiment of the present application.

[0033] The processor 20 may be one or more, Figure 1 A processor 20 is taken as an example.

[0034] The processor 20, the memory 10, the camera module 30 and the display module 40 can be connected via a bus or other means. Figure 1 The bus connection is taken as an example.

[0035] The processor 20 may include a central processing unit (CPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a controller, a field programmable gate array (FPGA), etc. The processor 20 may also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0036] The camera module 30 is used to implement an image acquisition function. In the embodiment of the present application, the camera module 30 can capture an environmental image of the indoor space to be predicted and can send the captured environmental image of the indoor space to be predicted to the processor 20, so that the processor 20 can predict the spatial layout of the indoor space based on the environmental image of the indoor space to be predicted.

[0037] The display module 40 is used to display information input by the user, or to display information provided to the user, or to display various menus of the electronic device. In the embodiment of the present application, the display module 40 can display the target segmentation image to the user.

[0038] In some embodiments, the display module 40 can also accept user input. In this case, the display module 40 can be a touch screen.

[0039] Further, see Figure 2 , is a flow chart of a method for predicting indoor space layout provided in an embodiment of the present application. The method for predicting indoor space layout can be applied to the above-mentioned electronic device, and as one of the specific implementation methods, the method for predicting indoor space layout is executed by the above-mentioned processor 20, so that the electronic device can improve the accuracy of predicting indoor space layout.

[0040] Specifically, the method for predicting indoor space layout includes:

[0041] S100: Acquire an environmental image of the indoor space to be predicted.

[0042] In the embodiment of the present application, the environmental image of the indoor space to be predicted is an RGB image of the indoor space to be predicted, and each pixel of the RGB image is represented by a color value (RGB value).

[0043] In the embodiment of the present application, the environmental image of the indoor space to be predicted can be obtained by the camera module of the electronic device. It is understandable that in some alternative embodiments, the environmental image of the indoor space to be predicted can also be obtained by a camera device external to the electronic device.

[0044] S200: Inputting the environment image into the spatial layout prediction model to obtain a target segmentation image of the indoor space to be predicted.

[0045] In an embodiment of the present application, the spatial layout prediction model is used to output a target segmented image corresponding to the environmental image based on the environmental image. The target segmented image includes at least two segmented regions. Since predicting the layout of an indoor space essentially involves predicting the various regions of the indoor space, the at least two segmented regions included in the target segmented image can be used to represent the spatial layout of the indoor space to be predicted. It is understood that the regions of the indoor space include the ceiling, left wall, right wall, front wall, and floor.

[0046] The size of the target segmented image is consistent with the size of the environment image, and each pixel of the target segmented image is represented by a grayscale value. The grayscale values ​​of the pixels in the same segmented area are the same, while the grayscale values ​​of the pixels in different segmented areas are different. For example, the target segmented image includes area A, area B, area C, area D, and area E. The grayscale values ​​of the pixels in area A are the same, the grayscale values ​​of the pixels in area B are the same, the grayscale values ​​of the pixels in area C are the same, the grayscale values ​​of the pixels in area D are the same, and the grayscale values ​​of the pixels in area E are the same. However, the grayscale values ​​of the pixels in areas A, B, C, D, and E are different from each other, so that different segmented areas can be distinguished by grayscale values.

[0047] In the embodiment of the present application, the spatial layout prediction model is obtained through training. Specifically, before the environment image is input into the spatial layout prediction model, the spatial layout prediction model is obtained through training.

[0048] The spatial layout prediction model obtained through training specifically includes:

[0049] S1. Obtain training samples.

[0050] In an embodiment of the present application, the training samples are images stored in a sample database for training a spatial layout prediction model, and the training samples include an environmental sample image and a standard segmentation image corresponding to the environmental sample image.

[0051] The environment sample image is an RGB image, and each pixel of the environment sample image is represented by a color value (RGB value).

[0052] The standard segmentation image is used to represent the ideal indoor space layout of the environmental sample image. The size of the standard segmentation image is consistent with that of the environmental sample image, and each pixel of the standard segmentation image is represented by a grayscale value.

[0053] The standard segmented image includes at least two segmented areas. The grayscale values ​​of pixels in the same segmented area are the same, while the grayscale values ​​of pixels in different segmented areas are different.

[0054] The sample database includes a number of training samples. When obtaining a training sample, a training sample is randomly extracted from the sample database.

[0055] S2: Input the environment sample image and the standard segmentation image into the preset neural network model for training, and use the loss function to calculate the loss cost during training.

[0056] In the embodiment of the present application, the preset neural network model is a VGG model, and a loss function is set in the preset neural network model. The loss function can be used to calculate the loss cost of each training of the preset neural network model, and then the preset neural network model can be reversely adjusted according to the loss cost to obtain a spatial layout prediction model.

[0057] Since the spatial layout prediction model is trained based on a preset neural network model, the loss function of the preset neural network model is also the loss function of the spatial layout prediction model, which is obtained by the weighted sum of at least two of the segmentation loss function, the boundary loss function and the corner loss function.

[0058] Among them, since the more complete the loss function is, the higher the accuracy of the trained spatial layout prediction model in predicting the indoor space layout will be. Therefore, in order to further improve the accuracy of the spatial layout prediction model in predicting the indoor space layout, in an embodiment of the present application, the loss function is obtained by weighted summation of the segmentation loss function, the boundary loss function and the corner loss function, so as to train the preset neural network model according to the three dimensions of accuracy of the segmented area, accuracy of the boundary and accuracy of the corner point, and then obtain a spatial layout prediction model that can accurately predict the boundaries of each segmented area and accurately determine the corner points of each segmented area, thereby improving the accuracy of the spatial layout pre-stored model in predicting the indoor space layout.

[0059] Furthermore, after the environmental sample image is input into the preset neural network model, a predicted segmentation image of the environmental sample image is obtained, and the segmentation loss function, boundary loss function and corner loss function can be determined based on the predicted segmentation image and the standard segmentation image.

[0060] The predicted segmentation image is used to represent the actual indoor space layout of the environmental sample image. The size of the predicted segmentation image is consistent with that of the standard segmentation image, and each pixel of the predicted segmentation image is represented by a grayscale value.

[0061] The predicted segmented image includes at least two segmented areas. The grayscale values ​​of pixels in the same segmented area are the same, while the grayscale values ​​of pixels in different segmented areas are different.

[0062] In the predicted segmented image, pixels within the segmented region corresponding to the standard segmented image are represented using the same grayscale values ​​as in the standard segmented image. For example, if region F1 of the predicted segmented image corresponds to region F2 of the standard segmented image, the grayscale values ​​of the pixels in region F1 are the same as the grayscale values ​​of the pixels in region F2.

[0063] Based on this, based on the predicted segmentation image and the standard segmentation image, a segmentation loss function is determined, specifically including: comparing the grayscale values ​​of corresponding pixel points in the predicted segmentation image and the standard segmentation image, and determining the pixel points whose grayscale values ​​in the predicted segmentation image are the same as the grayscale values ​​of the corresponding pixel points in the standard segmentation image as the correctly predicted pixel points in the predicted segmentation image; counting the first number of correctly predicted pixel points and the total number of pixel points in the predicted segmentation image; and determining the segmentation loss function based on the first number and the total number of pixel points in the predicted segmentation image.

[0064] Pixels located at the same position after the predicted segmented image and the standard segmented image overlap are determined to be corresponding pixels in the predicted segmented image and the standard segmented image. For example, if pixel A1 in the predicted segmented image is located in the first row and first column of the predicted segmented image, and pixel B1 in the standard segmented image is located in the first row and first column of the standard segmented image, then pixel A1 and pixel B1 are determined to be corresponding pixels. At this time, the grayscale values ​​of pixel A1 and pixel B1 are compared. If the grayscale value of pixel A1 is the same as the grayscale value of pixel B1, pixel A1 is determined to be the correctly predicted pixel in the predicted segmented image.

[0065] The segmentation loss function specifically includes:

[0066]

[0067] Among them, S1 is the segmentation loss value, O T is the first number of correctly predicted pixels in the predicted segmented image, N L is the total number of pixels in the predicted segmented image.

[0068] Based on the predicted segmentation image and the standard segmentation image, a boundary loss function is determined, specifically including: determining a predicted boundary image based on the predicted segmentation image; determining a standard boundary image based on the standard segmentation image; and determining a boundary loss function based on the predicted boundary image and the standard boundary image.

[0069] The predicted boundary image is used to represent the boundaries of each segmented region in the predicted segmented image, that is, the predicted boundary image includes the predicted boundary. In the predicted boundary image, the grayscale values ​​of the pixels at the predicted boundary are different from the grayscale values ​​of the pixels at the non-prediction boundary.

[0070] In the embodiment of the present application, the predicted boundary image is obtained by performing edge extraction on the predicted segmented image.

[0071] The standard boundary image is used to represent the boundaries of each segmented area in the standard segmented image, that is, the standard boundary image contains a standard boundary.

[0072] In the embodiment of the present application, after the edge of the standard segmented image is obtained, a morphological dilation operation is performed on each edge, and a Gaussian filter is used to filter it to obtain a standard boundary image, such as Figure 3 shown.

[0073] When determining a boundary loss function based on the predicted boundary image and the standard boundary image, the grayscale values ​​of the boundary pixels of the predicted boundary image and the pixels corresponding to the boundary pixels in the standard boundary image are compared to determine the correctly predicted boundary pixels in the predicted boundary image; a second number of correctly predicted boundary pixels and the total number of pixel points of the boundary pixels of the predicted boundary image are counted; and a boundary loss function is determined based on the second number and the total number of pixel points of the boundary pixels of the predicted boundary image.

[0074] The boundary pixels of the predicted boundary image are the pixels of the predicted boundary. When the predicted boundary image and the standard boundary image are overlapped, the pixels in the standard boundary image that are at the same position as the pixels of the predicted boundary are the pixels in the standard boundary image that correspond to the boundary pixels.

[0075] If the grayscale value of a boundary pixel in the predicted boundary image is the same as the grayscale value of a corresponding pixel in the standard boundary image, the pixel in the predicted boundary image is determined as a correctly predicted boundary pixel.

[0076] The boundary loss function specifically includes:

[0077]

[0078] Among them, S2 is the boundary loss value, P T is the second number of correctly predicted boundary pixels in the predicted boundary image, N M is the total number of boundary pixels of the predicted boundary image.

[0079] Based on the predicted segmentation image and the standard segmentation image, a corner loss function is determined, specifically including: determining each first corner point in the predicted segmentation image and determining each second corner point in the standard segmentation image, with each first corner point corresponding to each second corner point one-to-one; calculating the offset parameter of each first corner point according to each corresponding first corner point and second corner point; counting the sum of the offset parameters of all first corner points and the number of first corner points; and determining the corner loss function according to the sum of the offset parameters and the number of first corner points.

[0080] In an embodiment of the present application, the intersection points of each boundary in the predicted segmented image and the intersection points of each boundary with the edge of the predicted segmented image are determined as first corner points; the intersection points of each boundary in the standard segmented image and the intersection points of each boundary with the edge of the standard segmented image are determined as second corner points.

[0081] The number of first corner points is the same as the number of second corner points, and each first corner point corresponds to each second corner point one-to-one. For example, if the first corner points include corner point A1, corner point B1, and corner point C1, and the second corner points include corner point A2, corner point B2, and corner point C2, then corner point A1 corresponds to corner point A2, corner point B1 corresponds to corner point B2, and corner point C1 corresponds to corner point C2.

[0082] The first corner point and the second corner point with the same meaning are determined as corresponding corner points. For example, if corner point A1 is the intersection of the first boundary and the second boundary in the predicted segmented image, and corner point A2 is the intersection of the first boundary and the second boundary in the standard segmented image, then corner point A1 and corner point A2 are determined to have the same meaning and are determined as corresponding corner points.

[0083] Specifically, according to each corresponding first corner point and second corner point, calculating the offset parameter of each first corner point includes: obtaining the first coordinates of each corresponding first corner point and the second coordinates of the second corner point; calculating the offset distance of each first corner point according to each corresponding first coordinate and second coordinate; determining the diagonal length of the predicted segmented image; and calculating the offset parameter of each first corner point according to the diagonal length and the offset distance of each first corner point.

[0084] For example, obtain the coordinates A1 of the corresponding corner point A1 and the coordinates A2 of the corner point A2, calculate the offset distance of the corner point A1 based on the corresponding coordinates A1 and coordinates A2, and calculate the offset parameter of the corner point A1 based on the diagonal length of the predicted segmented image and the offset distance of the corner point A1; obtain the coordinates B1 of the corresponding corner point B1 and the coordinates B2 of the corner point B2, calculate the offset distance of the corner point B1 based on the corresponding coordinates B1 and coordinates B2, and calculate the offset parameter of the corner point B1 based on the diagonal length of the predicted segmented image and the offset distance of the corner point B1; obtain the coordinates C1 of the corresponding corner point C1 and the coordinates C2 of the corner point C2, calculate the offset distance of the corner point C1 based on the corresponding coordinates C1 and coordinates C2, and calculate the offset parameter of the corner point C1 based on the diagonal length of the predicted segmented image and the offset distance of the corner point C1.

[0085] That is, after calculating the offset parameters of corner point A1 according to the corresponding corner points A1 and A2, calculate the offset parameters of corner point B1 according to the corresponding corner points B1 and B2, and then calculate the offset parameters of corner point C1 according to the corresponding corner points C1 and C2; then, count the sum of the offset parameters of corner points A1, corner point A2 and corner point A3, and count the number of corner points A1, corner point A2 and corner point A3; finally, determine the corner point loss function based on the sum of the offset parameters and the number of corner points A1, corner point A2 and corner point A3.

[0086] It is understandable that the calculation formula for the offset parameter of each first corner point is:

[0087]

[0088] Among them, Q i is the offset parameter of the first corner point i, d i is the offset distance of the i-th first corner point, and D is the diagonal length of the predicted segmented image.

[0089] At this time, the corner loss function specifically includes:

[0090]

[0091] Among them, S3 is the corner loss value, is the sum of the offset parameters of all first corner points, and n is the number of first corner points.

[0092] Based on this, the loss function specifically includes:

[0093] f=k1×S1+k2×S2+k3×S3

[0094] Among them, f is the loss cost, k1 is the segmentation weight, S1 is the segmentation loss value, k2 is the boundary weight, S2 is the boundary loss value, k3 is the corner weight, and S3 is the corner loss value.

[0095] It is understandable that in actual application, the numerical values ​​of the segmentation weight, boundary weight and corner weight can be set manually to highlight the corresponding constraints. For example, when it is necessary to highlight the corner loss constraint, the numerical value of the corner weight is increased; when it is necessary to highlight the segmentation loss constraint, the numerical value of the segmentation weight is increased; the segmentation weight, boundary weight and corner weight can also be set as variables, and the numerical values ​​of the segmentation weight, boundary weight and corner weight are determined through neural network learning.

[0096] Furthermore, the loss cost can be calculated based on the predicted segmentation image, the standard segmentation image and the loss function.

[0097] Since the predicted segmentation image represents the actual indoor space layout of the environmental sample image, and the standard segmentation image represents the ideal indoor space layout of the environmental sample image, the difference between the actual indoor space layout and the ideal indoor space layout can be determined based on the predicted segmentation image, the standard segmentation image and the loss function, and then the loss cost of the preset neural network model can be determined.

[0098] Among them, since the loss function includes the segmentation loss function, the boundary loss function and the corner loss function, the loss cost is calculated based on the predicted segmentation image, the standard segmentation image and the loss function, specifically including: calculating the segmentation loss value based on the predicted segmentation image, the standard segmentation image and the segmentation loss function; calculating the boundary loss value based on the predicted segmentation image, the standard segmentation image and the boundary loss function; calculating the corner loss value based on the predicted segmentation image, the standard segmentation image and the corner loss function; and taking the weighted sum of the segmentation loss value, the boundary loss value and the corner loss value to obtain the loss cost.

[0099] Specifically, based on the predicted segmentation image, the standard segmentation image and the segmentation loss function, calculating the segmentation loss value includes: comparing the grayscale values ​​of corresponding pixels in the predicted segmentation image and the standard segmentation image to determine the correctly predicted pixels in the predicted segmentation image; counting the first number of correctly predicted pixels and the total number of pixels in the predicted segmentation image; and calculating the segmentation loss value based on the first number, the total number of pixels in the predicted segmentation image and the segmentation loss function.

[0100] Pixels located at the same position after the predicted segmented image and the standard segmented image overlap are determined as corresponding pixels in the predicted segmented image and the standard segmented image. For example, if pixel A1 in the predicted segmented image is located in the first row and first column of the predicted segmented image, and pixel B1 in the standard segmented image is located in the first row and first column of the standard segmented image, then pixel A1 and pixel B1 are determined to be corresponding pixels.

[0101] If the grayscale value of a pixel in the predicted segmentation image is the same as the grayscale value of the corresponding pixel in the standard segmentation image, then the pixel in the predicted segmentation image is determined to be the correctly predicted pixel. For example, if pixel A1 in the predicted segmentation image corresponds to pixel B1 in the standard segmentation image, and the grayscale value of pixel A1 is the same as the grayscale value of pixel B1, then pixel A1 is determined to be the correctly predicted pixel.

[0102] At this time, the calculation formula of the segmentation loss value is:

[0103]

[0104] Among them, S1 is the segmentation loss value, O T is the first number of correctly predicted pixels in the predicted segmented image, N L is the total number of pixels in the predicted segmented image, is the segmentation loss function.

[0105] Calculating the boundary loss value based on the predicted segmentation image, the standard segmentation image and the boundary loss function includes: determining a predicted boundary image based on the predicted segmentation image; determining a standard boundary image based on the standard segmentation image; and calculating the boundary loss value based on the predicted boundary image, the standard boundary image and the boundary loss function.

[0106] The predicted boundary image is used to represent the boundary of each segmented area in the predicted segmented image, that is, the predicted boundary image includes the predicted boundary.

[0107] In the embodiment of the present application, the predicted boundary image is obtained by performing edge extraction on the predicted segmented image.

[0108] The standard boundary image is used to represent the boundaries of each segmented area in the standard segmented image, that is, the standard boundary image contains a standard boundary.

[0109] In the embodiment of the present application, after edges are obtained for the standard segmented image, a morphological dilation operation is performed on each edge, and a Gaussian filter is used for filtering to obtain a standard boundary image.

[0110] Based on the predicted boundary image, the standard boundary image and the boundary loss function, the boundary loss value is calculated, specifically including: comparing the grayscale values ​​of the boundary pixels of the predicted boundary image and the pixel points corresponding to the boundary pixels in the standard boundary image, and determining the correctly predicted boundary pixels in the predicted boundary image; counting the second number of correctly predicted boundary pixels and the total number of pixel points of the boundary pixels of the predicted boundary image; and calculating the boundary loss value based on the second number, the total number of boundary pixels of the predicted boundary image and the boundary loss function.

[0111] The boundary pixels of the predicted boundary image are the pixels of the predicted boundary. When the predicted boundary image and the standard boundary image are overlapped, the pixels in the standard boundary image that are at the same position as the pixels of the predicted boundary are the pixels in the standard boundary image that correspond to the boundary pixels.

[0112] If the grayscale value of a boundary pixel in the predicted boundary image is the same as the grayscale value of a corresponding pixel in the standard boundary image, the pixel in the predicted boundary image is determined as a correctly predicted boundary pixel.

[0113] At this time, the calculation formula of the boundary loss value is:

[0114]

[0115] Among them, S2 is the boundary loss value, P T is the second number of correctly predicted boundary pixels in the predicted boundary image, N M is the total number of boundary pixels of the predicted boundary image, is the boundary loss function.

[0116] Based on the predicted segmentation image, the standard segmentation image and the corner loss function, the corner loss value is calculated, specifically including: determining each first corner point in the predicted segmentation image and each second corner point in the standard segmentation image, with each first corner point corresponding to each second corner point one by one; calculating the offset parameter of each first corner point according to each corresponding first corner point and second corner point; counting the sum of the offset parameters of all first corner points and the number of first corner points; and calculating the corner loss value according to the sum of the offset parameters, the number of first corner points and the corner loss function.

[0117] In an embodiment of the present application, the intersection points of each boundary in the predicted segmented image and the intersection points of each boundary with the edge of the predicted segmented image are determined as first corner points; the intersection points of each boundary in the standard segmented image and the intersection points of each boundary with the edge of the standard segmented image are determined as second corner points.

[0118] The number of first corner points is the same as the number of second corner points, and each first corner point corresponds to each second corner point one-to-one. For example, if the first corner points include corner point A1, corner point B1, and corner point C1, and the second corner points include corner point A2, corner point B2, and corner point C2, then corner point A1 corresponds to corner point A2, corner point B1 corresponds to corner point B2, and corner point C1 corresponds to corner point C2.

[0119] The first corner point and the second corner point with the same meaning are determined as corresponding corner points. For example, if corner point A1 is the intersection of the first boundary and the second boundary in the predicted segmented image, and corner point A2 is the intersection of the first boundary and the second boundary in the standard segmented image, then corner point A1 and corner point A2 are determined to have the same meaning and are determined as corresponding corner points.

[0120] Specifically, according to each corresponding first corner point and second corner point, calculating the offset parameter of each first corner point includes: obtaining the first coordinates of each corresponding first corner point and the second coordinates of the second corner point; calculating the offset distance of each first corner point according to each corresponding first coordinate and second coordinate; determining the diagonal length of the predicted segmented image; and calculating the offset parameter of each first corner point according to the diagonal length and the offset distance of each first corner point.

[0121] For example, obtain the coordinates A1 of the corresponding corner point A1 and the coordinates A2 of the corner point A2, calculate the offset distance of the corner point A1 based on the corresponding coordinates A1 and coordinates A2, and calculate the offset parameter of the corner point A1 based on the diagonal length of the predicted segmented image and the offset distance of the corner point A1; then, obtain the coordinates B1 of the corresponding corner point B1 and the coordinates B2 of the corner point B2, calculate the offset distance of the corner point B1 based on the corresponding coordinates B1 and coordinates B2, and calculate the offset parameter of the corner point B1 based on the diagonal length of the predicted segmented image and the offset distance of the corner point B1; then, obtain the coordinates C1 of the corresponding corner point C1 and the coordinates C2 of the corner point C2, calculate the offset distance of the corner point C1 based on the corresponding coordinates C1 and coordinates C2, and calculate the offset parameter of the corner point C1 based on the diagonal length of the predicted segmented image and the offset distance of the corner point C1.

[0122] That is, after calculating the offset parameters of corner point A1 according to the corresponding corner points A1 and A2, calculate the offset parameters of corner point B1 according to the corresponding corner points B1 and B2, and then calculate the offset parameters of corner point C1 according to the corresponding corner points C1 and C2; then, count the sum of the offset parameters of corner points A1, A2 and A3, and count the number of corner points A1, A2 and A3; finally, calculate the corner point loss value based on the sum of the offset parameters, the number of corner points A1, A2 and A3, and the corner point loss function.

[0123] It is understandable that the calculation formula for the offset parameter of each first corner point is:

[0124]

[0125] Among them, Q i is the offset parameter of the first corner point i, d i is the offset distance of the i-th first corner point, and D is the diagonal length of the predicted segmented image.

[0126] At this time, the calculation formula of the corner loss value is:

[0127]

[0128] Among them, S3 is the corner loss value, is the sum of the offset parameters of all first corner points, n is the number of first corner points, is the corner loss function.

[0129] Based on this, the calculation formula for loss cost is:

[0130] f=k1×S1+k2×S2+k3×S3

[0131] Among them, f is the loss cost, k1 is the segmentation weight, S1 is the segmentation loss value, k2 is the boundary weight, S2 is the boundary loss value, k3 is the corner weight, and S3 is the corner loss value.

[0132] It is understandable that in actual application, the numerical values ​​of the segmentation weight, boundary weight and corner weight can be set manually to highlight the corresponding constraints. For example, when it is necessary to highlight the corner loss constraint, the numerical value of the corner weight is increased; when it is necessary to highlight the segmentation loss constraint, the numerical value of the segmentation weight is increased; the segmentation weight, boundary weight and corner weight can also be set as variables, and the numerical values ​​of the segmentation weight, boundary weight and corner weight are determined through neural network learning.

[0133] S3: Determine the preset neural network model when the loss cost meets the preset cost condition as the spatial layout prediction model.

[0134] Specifically, if the loss cost meets the preset cost condition, the current preset neural network model is determined to be the spatial layout prediction model; if the loss cost does not meet the preset cost condition, the loss cost is used to reversely adjust the parameters of the preset neural network model, and steps S1 to S3 are repeated based on the preset neural network model after adjusting the parameters until the loss cost meets the preset cost condition, and the preset neural network model when the loss cost meets the preset cost condition is determined to be the spatial layout prediction model.

[0135] In the process of repeatedly executing steps S1 to S3, the training samples obtained each time may be the same or different, and any training sample is randomly extracted from the sample database.

[0136] Among them, if the loss cost is less than the preset cost threshold or the loss cost remains unchanged, it is determined that the loss cost meets the preset cost condition.

[0137] Furthermore, in some embodiments, after obtaining the target segmentation image of the indoor space to be predicted, the position information of the target object contained in the indoor space to be predicted is determined based on the environmental image of the indoor space to be predicted, and the segmentation area where the target object is located is determined based on the position information of the target object and the target segmentation image.

[0138] The target objects are doors, windows, sofas and other objects contained in the indoor space to be predicted. Based on this, the environmental image of the indoor space to be predicted can be input into the target object detection model to determine the target objects and their location information.

[0139] The target object's location information is specifically the target object's center coordinates. By determining which segmented region of the target segmented image the target object's center coordinates fall within, the segmented region where the target object is located can be determined. For example, if the center coordinates of a door fall within the left wall region of the target segmented image, the door is determined to be located in the left wall region.

[0140] It can be understood that in the embodiment of the present application, the spatial layout prediction model takes into account features such as segmented areas, boundaries, and boundary corners to predict the spatial layout of the indoor environment, so that the spatial layout prediction model can accurately predict the boundaries of each segmented area and accurately determine the corner points of each segmented area, thereby improving the accuracy of predicting the indoor space layout.

[0141] The term "module" used below refers to a combination of software and / or hardware that can implement a predetermined function. Although the devices described in the following embodiments can be implemented in software, implementation in hardware, or a combination of software and hardware, is also conceivable.

[0142] See also Figure 4 , is a schematic structural diagram of a device for predicting indoor space layout provided in an embodiment of the present application, the device for predicting indoor space layout comprising:

[0143] The first acquisition module 21 is used to acquire an environmental image of the indoor space to be predicted;

[0144] The first input module 22 is used to input the environmental image into the spatial layout prediction model to obtain a target segmentation image of the indoor space to be predicted, where the target segmentation image includes at least two segmentation areas, and the at least two segmentation areas are used to represent the spatial layout of the indoor space to be predicted.

[0145] See also Figure 5 In some embodiments, the device for predicting indoor space layout further includes:

[0146] A second acquisition module 23 is configured to acquire a training sample before inputting the environment image into the spatial layout prediction model, wherein the training sample includes an environment sample image and a standard segmentation image corresponding to the environment sample image;

[0147] A second input module 24 is used to input the environmental sample image and the standard segmentation image into a preset neural network model for training, and use a loss function to calculate the loss cost during training;

[0148] The first determining module 25 is configured to determine the preset neural network model as the spatial layout prediction model when the loss cost satisfies a preset cost condition.

[0149] In some embodiments, if the loss cost does not satisfy the preset cost condition, the preset neural network model is reversely adjusted using the loss cost.

[0150] In some embodiments, the loss function of the spatial layout prediction model is obtained by weighted summation of at least two of a segmentation loss function, a boundary loss function, and a corner loss function.

[0151] In some embodiments, the environment sample image is input into the preset neural network model to obtain a predicted segmented image of the environment sample image; then,

[0152] The first determining module 25 is further configured to:

[0153] The segmentation loss function, the boundary loss function and the corner loss function are determined based on the predicted segmentation image and the standard segmentation image.

[0154] In some embodiments, the first determining module 25 is specifically configured to:

[0155] When determining the segmentation loss function based on the predicted segmentation image and the standard segmentation image, comparing the grayscale values ​​of corresponding pixels of the predicted segmentation image and the standard segmentation image;

[0156] Determine a pixel point in the predicted segmented image whose grayscale value is the same as the grayscale value of the corresponding pixel point in the standard segmented image as a correctly predicted pixel point in the predicted segmented image;

[0157] Counting the first number of correctly predicted pixels and the total number of pixels of the predicted segmented image;

[0158] The segmentation loss function is determined based on the first number and the total number of pixels in the predicted segmented image.

[0159] In some embodiments, the predicted segmented image and the standard segmented image each include at least two segmented regions; then,

[0160] The first determining module 25 is specifically configured to:

[0161] When determining the boundary loss function based on the predicted segmented image and the standard segmented image, a predicted boundary image is determined according to the predicted segmented image, wherein the predicted boundary image is used to represent the boundary of each segmented area in the predicted segmented image;

[0162] Determining a standard boundary image according to the standard segmented image, wherein the standard boundary image is used to represent the boundary of each segmented area in the standard segmented image;

[0163] The boundary loss function is determined based on the predicted boundary image and the standard boundary image.

[0164] In some embodiments, the first determining module 25 is specifically configured to:

[0165] When determining the boundary loss function based on the predicted boundary image and the standard boundary image, comparing the grayscale values ​​of boundary pixels in the predicted boundary image and pixels corresponding to the boundary pixels in the standard boundary image to determine the correctly predicted boundary pixels in the predicted boundary image;

[0166] Counting the second number of the correctly predicted boundary pixels and the total number of boundary pixels of the predicted boundary image;

[0167] The boundary loss function is determined based on the second number and the total number of boundary pixels of the predicted boundary image.

[0168] In some embodiments, the first determining module 25 is specifically configured to:

[0169] When determining the corner loss function based on the predicted segmented image and the standard segmented image, each first corner point is determined in the predicted segmented image, and each second corner point is determined in the standard segmented image, and each first corner point corresponds to each second corner point in a one-to-one manner;

[0170] Calculating an offset parameter of each first corner point according to each corresponding first corner point and second corner point;

[0171] Counting the sum of the offset parameters of all the first corner points and the number of the first corner points;

[0172] The corner point loss function is determined according to the sum of the offset parameters and the number of the first corner points.

[0173] In some embodiments, the first determining module 25 is specifically configured to:

[0174] When calculating the offset parameters of each first corner point according to each corresponding first corner point and second corner point, obtaining the first coordinates of each corresponding first corner point and the second coordinates of each corresponding second corner point;

[0175] Calculating an offset distance of each first corner point according to each corresponding first coordinate and second coordinate;

[0176] Determining the diagonal length of the predicted segmented image;

[0177] An offset parameter of each of the first corner points is calculated according to the diagonal length and the offset distance of each of the first corner points.

[0178] See also Figure 6 In some embodiments, the device for predicting indoor space layout further includes:

[0179] A second determining module 26 is configured to determine position information of a target object contained in the indoor space to be predicted based on the environment image;

[0180] It is also used to determine the segmented area where the target object is located based on the position information of the target object and the target segmented image.

[0181] Since the device embodiment and the method embodiment are based on the same concept, the content of the device embodiment may refer to the method embodiment if the contents do not conflict with each other, and they will not be described one by one here.

[0182] It can be understood that in the embodiment of the present application, the spatial layout prediction model takes into account features such as segmented areas, boundaries, and boundary corners to predict the spatial layout of the indoor environment, so that the spatial layout prediction model can accurately predict the boundaries of each segmented area and accurately determine the corner points of each segmented area, thereby improving the accuracy of predicting the indoor space layout.

[0183] The present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are executed by one or more processors, for example Figure 1 A processor 20 in the embodiment can be used to execute the various steps of a method for predicting an indoor space layout in any of the above-mentioned method embodiments, or to implement the functions of various modules of a device for predicting an indoor space layout in any of the above-mentioned device embodiments.

[0184] The embodiment of the present application further provides a computer program product, the computer program product including a computer program stored on a computer-readable storage medium, the computer program including program instructions, when the program instructions are executed by one or more processors, for example Figure 1 A processor 20 in the computer can enable the computer to execute the various steps of a method for predicting indoor space layout in any of the above method embodiments, or implement the functions of various modules of a device for predicting indoor space layout in any of the above device embodiments.

[0185] The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical units, i.e., they may be located in one place or distributed across multiple network elements. Some or all of these modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0186] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a general hardware platform, and of course, can also be implemented by hardware. Those skilled in the art can understand that all or part of the processes in the above embodiment methods can be completed by hardware related to computer program instructions, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the implementation methods of the above methods. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0187] The above description is merely an embodiment of the present application and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

[0188] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Based on the concept of the present application, the technical features in the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations in different aspects of the present application as described above. For the sake of simplicity, they are not provided in detail. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for predicting indoor space layout, characterized in that: include: Obtain an environmental image of the indoor space to be predicted; Inputting the environmental image into a spatial layout prediction model to obtain a target segmented image of the indoor space to be predicted, wherein the target segmented image includes at least two segmented regions, and the at least two segmented regions are used to represent the spatial layout of the indoor space to be predicted; The spatial layout prediction model is obtained by inputting an environmental sample image and a standard segmentation image corresponding to the environmental sample image into a preset neural network model for training, and a loss function is used to calculate the loss cost during training. The spatial layout prediction model is a preset neural network model when the loss cost meets the preset cost condition, and the loss function is obtained by at least two of a segmentation loss function, a boundary loss function, and a corner loss function; The standard segmented image is used to represent the ideal indoor space layout of the environmental sample image. Each pixel of the standard segmented image is represented by a grayscale value, and the grayscale values ​​of pixels in different segmented areas in the standard segmented image are different.

2. The method according to claim 1, characterized in that The method further comprises: If the loss cost does not meet the preset cost condition, the preset neural network model is reversely adjusted using the loss cost.

3. The method according to claim 1, characterized in that The loss function of the spatial layout prediction model is obtained by weighted summation of at least two of a segmentation loss function, a boundary loss function, and a corner loss function.

4. The method according to claim 3, characterized in that After the environmental sample image is input into the preset neural network model, a predicted segmented image of the environmental sample image is obtained; then, The method further comprises: The segmentation loss function, the boundary loss function and the corner loss function are determined based on the predicted segmentation image and the standard segmentation image.

5. The method according to claim 4, characterized in that The determining the segmentation loss function based on the predicted segmentation image and the standard segmentation image includes: Comparing the grayscale values ​​of corresponding pixels of the predicted segmented image and the standard segmented image; Determine a pixel point in the predicted segmented image whose grayscale value is the same as the grayscale value of the corresponding pixel point in the standard segmented image as a correctly predicted pixel point in the predicted segmented image; Counting the first number of correctly predicted pixels and the total number of pixels of the predicted segmented image; The segmentation loss function is determined according to the first number and the total number of pixels of the predicted segmented image.

6. The method according to claim 4, characterized in that The predicted segmented image and the standard segmented image both include at least two segmented regions; then, The determining the boundary loss function based on the predicted segmentation image and the standard segmentation image includes: Determining a predicted boundary image based on the predicted segmented image, wherein the predicted boundary image is used to represent the boundary of each segmented area in the predicted segmented image; Determining a standard boundary image according to the standard segmented image, wherein the standard boundary image is used to represent the boundary of each segmented area in the standard segmented image; The boundary loss function is determined based on the predicted boundary image and the standard boundary image.

7. The method according to claim 6, characterized in that The determining the boundary loss function based on the predicted boundary image and the standard boundary image includes: Comparing the grayscale values ​​of boundary pixels of the predicted boundary image and pixels corresponding to the boundary pixels in the standard boundary image to determine the correctly predicted boundary pixels in the predicted boundary image; Counting the second number of the correctly predicted boundary pixels and the total number of boundary pixels of the predicted boundary image; The boundary loss function is determined according to the second number and the total number of boundary pixels of the predicted boundary image.

8. The method according to claim 4, characterized in that The determining the corner loss function based on the predicted segmentation image and the standard segmentation image includes: Determining each first corner point in the predicted segmented image and determining each second corner point in the standard segmented image, wherein each first corner point corresponds to each second corner point in a one-to-one manner; Calculating an offset parameter of each first corner point according to each corresponding first corner point and second corner point; Counting the sum of the offset parameters of all the first corner points and the number of the first corner points; The corner point loss function is determined according to the sum of the offset parameters and the number of the first corner points.

9. The method according to claim 8, characterized in that The calculating, according to each corresponding first corner point and second corner point, an offset parameter of each first corner point includes: Obtaining first coordinates of each corresponding first corner point and second coordinates of each corresponding second corner point; Calculating an offset distance of each first corner point according to each corresponding first coordinate and second coordinate; Determining the diagonal length of the predicted segmented image; An offset parameter of each of the first corner points is calculated according to the diagonal length and the offset distance of each of the first corner points.

10. The method according to any one of claims 1 to 9, characterized in that The method further comprises: Determining, based on the environmental image, location information of a target object contained in the indoor space to be predicted; The segmented area where the target object is located is determined according to the position information of the target object and the target segmented image.

11. A device for predicting indoor space layout, characterized in that: include: The first acquisition module is used to acquire an environmental image of the indoor space to be predicted; A first input module is configured to input the environment image into a spatial layout prediction model to obtain a target segmented image of the indoor space to be predicted, wherein the target segmented image includes at least two segmented regions, and the at least two segmented regions are used to represent the spatial layout of the indoor space to be predicted; The spatial layout prediction model is obtained by inputting an environmental sample image and a standard segmentation image corresponding to the environmental sample image into a preset neural network model for training, and a loss function is used to calculate the loss cost during training. The spatial layout prediction model is a preset neural network model when the loss cost meets the preset cost condition, and the loss function is obtained by the weighted sum of at least two of the segmentation loss function, the boundary loss function and the corner loss function; The standard segmented image is used to represent the ideal indoor space layout of the environmental sample image. Each pixel of the standard segmented image is represented by a grayscale value, and the grayscale values ​​of pixels in different segmented areas in the standard segmented image are different.

12. An electronic device, characterized in that: include: at least one processor; as well as A memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor so as to enable the at least one processor to perform the method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used for a processor to execute the method according to any one of claims 1 to 10.

Images