Furniture restoration method, device, electronic device and readable storage medium

Abstract

The embodiments of the present application provide a furniture restoration method, device, electronic device and readable storage medium, the method comprising: obtaining an input image; obtaining camera parameters corresponding to the input image according to the input image; performing instance segmentation on the input image to obtain a sub-region containing furniture; constructing a furniture template data set, wherein the furniture in the furniture template data set is composed of a rectangular parallelepiped; retrieving a model of the furniture in the input image according to the furniture template data set and the sub-region containing furniture; optimizing the shape of the furniture model according to the camera parameters corresponding to the input image. Implementing the above embodiments can quickly generate a model of furniture.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a furniture restoration method, device, electronic device and computer-readable storage medium. Background Art

[0002] In the era of big data, data is the key to all intelligent technologies. In the interior decoration industry, due to patent barriers, there is a lack of 3D data of indoor scenes, but there are many pictures of indoor scenes on the Internet. Therefore, it is very important to infer the content contained in the two-dimensional picture, such as the type and placement of furniture, and restore the algorithm technology of the three-dimensional scene.

[0003] In existing technologies, the predicted shape and 3D position of furniture are inaccurate, resulting in misalignment between the projection into the picture and its actual position in the picture. Simply using the generative network method in deep learning results in poor quality of the generated furniture mesh, with a large number of points and faces, which is not conducive to display on mobile phones. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a furniture restoration method, device, electronic device and computer-readable storage medium, which can accurately generate a three-dimensional model corresponding to a two-dimensional image, and the number of points and surfaces in the generated model is small, which facilitates the display of the three-dimensional model on a mobile phone.

[0005] In a first aspect, an embodiment of the present application provides a furniture restoration method, comprising:

[0006] Get the input image;

[0007] Acquire camera parameters corresponding to the input image according to the input image;

[0008] Performing instance segmentation on the input image to obtain a sub-region containing furniture;

[0009] Constructing a furniture template data set, wherein the furniture in the furniture template data set is composed of a cuboid;

[0010] Retrieving a model of the furniture in the input image according to the furniture template dataset and the sub-region containing the furniture;

[0011] The shape of the furniture model is optimized according to the camera parameters corresponding to the input image.

[0012] In the above implementation process, a furniture template dataset was constructed. The furniture in the furniture template dataset is composed of rectangular blocks, which overcomes the shortcomings of the furniture models generated in the prior art, which have many points and faces and are not conducive to display on mobile phones. At the same time, by performing instance segmentation on the input image, the sub-region containing the furniture is obtained; according to the furniture template dataset and the sub-region containing the furniture, the model of the furniture can be quickly obtained; finally, the shape of the furniture model is optimized through the camera parameters corresponding to the input image, so that the obtained model is more accurate.

[0013] Furthermore, the step of acquiring camera parameters corresponding to the input image according to the input image includes:

[0014] The input image is input into a pre-trained deep residual network ResNet (Residual Network) 34 to obtain camera parameters corresponding to the input image.

[0015] In the above implementation process, the camera parameters corresponding to the input image are obtained through a machine learning algorithm, so that the obtained camera parameters are more accurate and the subsequent optimization effect of the furniture shape is further improved.

[0016] Furthermore, the step of performing instance segmentation on the input image comprises:

[0017] The input image is segmented by using a pre-trained Mask R-CNN to obtain the sub-region containing the furniture and the category of the furniture.

[0018] In the above implementation process, the Mask R-CNN training time is short and the detection accuracy is high. Based on the above implementation method, the sub-area containing furniture can be quickly acquired.

[0019] Furthermore, the step of acquiring the model of the furniture according to the furniture template dataset and the sub-region containing the furniture comprises:

[0020] Constructing a projection dataset according to the furniture template dataset, wherein the projection dataset includes projection templates formed by furniture templates in the furniture template dataset under different camera parameters;

[0021] Using the projection data set to train the deep residual network ResNet34 to obtain a trained deep residual network ResNet34;

[0022] The sub-region containing the furniture is input into the trained deep residual network ResNet34 to obtain an output result, and the output result is input into the classification fully connected layer to obtain a model of the furniture.

[0023] In the above implementation process, the projection of the furniture dataset under different camera perspectives is established, and the deep residual network ResNet34 algorithm is trained using the dataset. The sub-region containing the furniture is input into the trained deep residual network ResNet34 to obtain the output result, and the output result is input into the classification fully connected layer to obtain the furniture model.

[0024] Furthermore, the step of optimizing the shape of the furniture according to the camera parameters corresponding to the input image comprises:

[0025] Extracting the contour of the sub-region containing the furniture, uniformly sampling the contour of the sub-region containing the furniture, and obtaining a contour point set;

[0026] According to the camera parameters, a template point set corresponding to the furniture model in the two-dimensional plane where the contour point set is located is obtained;

[0027] constructing an optimization function, wherein the optimization function includes the distance between the contour point set and the template point set;

[0028] With the goal of minimizing the optimization function, the template point set is iterated using an iterative algorithm to obtain a model of the furniture after shape optimization.

[0029] In the above implementation process, an optimization function is constructed with the goal of minimizing the optimization function, and a template point set formed by projecting the model of the furniture onto a two-dimensional plane is used as an iteration basis to obtain a model of the furniture after shape optimization. The error between the shape of the shape-optimized model and the shape of the furniture in the input image is further reduced.

[0030] Furthermore, the optimization function is:

[0031]

[0032] Among them, P mask is the contour point set, P obj is the template point set, x, y represents P mask and P obj Point in, L(P mask ,P obj ) is P mask To P obj Point-by-point closest distance and P obj To P mask The sum of the point-by-point closest distances; p mask P mask The number of points in , p obj P obj The number of midpoints.

[0033] In the above implementation process, a distance function between the contour subset and the template subset is constructed. The smaller the value of the distance function is, the smaller the difference between the images formed by the two point sets is.

[0034] Furthermore, we use the Pytorch library to minimize L(P mask ,P obj ) as the target, iterate the template point set, wherein the learning rate is set to 0.1, the number of times is 500-600, and the model composed of the projection points of the template point set in three-dimensional space is the model of the furniture after shape optimization.

[0035] In the above implementation process, Python language has efficient data processing capabilities, and using the Pytorch library for iteration can speed up optimization efficiency.

[0036] In a second aspect, the present application provides a furniture restoration device, comprising:

[0037] An image acquisition module, used for acquiring an input image;

[0038] A camera parameter acquisition module, used to acquire camera parameters corresponding to the input image according to the input image;

[0039] An instance segmentation module, used for performing instance segmentation on the input image to obtain a sub-region containing furniture;

[0040] A data set construction module, used to construct a furniture template data set, wherein the furniture in the furniture template data set is composed of a cuboid;

[0041] A model retrieval module, used to retrieve a model of furniture in the input image according to the furniture template dataset and the sub-region containing the furniture;

[0042] The optimization module is used to optimize the shape of the furniture model according to the camera parameters corresponding to the input image.

[0043] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in any one of the first aspects when executing the computer program.

[0044] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which instructions are stored. When the instructions are executed on a computer, the computer executes the method as described in any one of the first aspects.

[0045] Other features and advantages disclosed in the present application will be described in the following description, or some features and advantages can be inferred or determined without doubt from the description, or can be learned by implementing the above-mentioned technology disclosed in the present application.

[0046] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 A schematic diagram of a process for restoring furniture provided in an embodiment of the present application;

[0049] Figure 2 A schematic diagram of obtaining camera parameters provided in an embodiment of the present application;

[0050] Figure 3 The effect diagram of instance segmentation provided in the embodiment of the present application;

[0051] Figure 4 This is a schematic diagram of the structure of a furniture restoration device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0053] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0054] Example 1

[0055] See also Figure 1 , the present application embodiment provides a furniture restoration method, comprising:

[0056] S1: Get input image;

[0057] S2: Obtain camera parameters corresponding to the input image according to the input image;

[0058] S3: Perform instance segmentation on the input image to obtain the sub-region containing furniture;

[0059] S4: Construct a furniture template dataset. The furniture in the furniture template dataset is composed of rectangular blocks.

[0060] S5: retrieve the model of the furniture in the input image based on the furniture template dataset and the sub-region containing the furniture;

[0061] S6: Optimize the shape of the furniture model according to the camera parameters corresponding to the input image.

[0062] In the above embodiment, the input image is a two-dimensional image, and the model of the furniture is a three-dimensional model.

[0063] In the above embodiment, the camera parameters corresponding to the input image refer to the parameters of the camera used when shooting the input image. Specifically, the camera parameters can be expressed by the function R(φ cam ,ψ cam ) represents, where θ cam , ψ cam They represent the angles of rotation around the X-axis and the Y-axis respectively.

[0064] The template data set is a data set composed of template model meshes.

[0065] In the above embodiment, a furniture template dataset is constructed. The furniture in the furniture template dataset is composed of rectangular parallelepipeds, which overcomes the disadvantage that the furniture model generated in the prior art has many points and faces, which is not conducive to display on the mobile phone. At the same time, by performing instance segmentation on the input image, the sub-region containing the furniture is obtained; according to the furniture template dataset and the sub-region containing the furniture, the model of the furniture can be quickly obtained; finally, the shape of the furniture model is optimized through the camera parameters corresponding to the input image, so that the obtained model is more accurate.

[0066] In a possible implementation, S2 includes:

[0067] The input image is input into the pre-trained deep residual network ResNet34 to obtain the camera parameters corresponding to the input image.

[0068] The training set can use the SUN RGB-D dataset, which contains 10,335 indoor scene pictures and corresponding camera parameter labels.

[0069] See also Figure 2, which is a schematic diagram provided in the embodiment of the present application, inputs a picture, passes through ResNet34, obtains a 2048-D (2048-dimension, 2048-dimensional) feature vector, and then passes through a classification fully connected layer to obtain the camera parameter θ cam ,ψ cam They represent the angles of rotation around the X-axis and the Y-axis respectively.

[0070] The camera parameters corresponding to the input image are obtained through machine learning algorithms, making the obtained camera parameters more accurate and further improving the subsequent optimization effect of the furniture shape.

[0071] In a possible implementation, step S3 includes:

[0072] The pre-trained Mask R-CNN is used to perform instance segmentation on the input image to obtain sub-regions containing furniture and the categories of furniture.

[0073] In academia and industry, Mask R-CNN is a very classic and effective deep learning algorithm for instance segmentation tasks. Mask R-CNN has a short training time and high detection accuracy. Based on the above implementation, the sub-region containing furniture can be quickly obtained.

[0074] For example, see Figure 3 , which is the effect of Mask R-CNN on furniture instance segmentation.

[0075] Input the image and use the Mask R-CNN algorithm to obtain a set of sub-regions containing furniture and the corresponding categories of furniture {(I mask1 ,cat mask1 ),(I mask2 ,cat mask2 )...}.

[0076] The parameters of the furniture template dataset include the 3D center position C obj , shape angle θ obj and the size s of each cuboid obj By varying these parameters, a furniture dataset with rich shape variations is established.

[0077] Exemplarily, the embodiment of the present application also provides a method for constructing a furniture template data set, which counts the important furniture commonly seen in indoor scene images, mainly including: bed, four-legged stool without backrest, four-legged stool with backrest, swivel chair, table, TV, wardrobe, bedside table, sofa, coffee table. For these 10 types of furniture, corresponding templates are established: the bed is composed of two connected rectangular blocks, namely the bed body and the backrest; the four-legged stool without backrest is composed of five connected rectangular blocks, namely the seat cushion and four legs; the four-legged stool with backrest is composed of six connected rectangular blocks, namely the seat cushion, the backrest and four legs; the swivel chair is composed of nine connected rectangular blocks, namely the seat cushion, the backrest, two armrests, the support shaft and four legs; the table is composed of five connected rectangular blocks, namely the tabletop and four legs; the TV is composed of a rectangular block; the wardrobe is promoted by a rectangular block; the bedside table is composed of a rectangular block; the sofa is composed of four connected rectangular blocks, namely the seat cushion, the backrest and two armrests; the coffee table is composed of a rectangular block.

[0078] In a possible implementation, S5 includes: constructing a projection dataset according to the furniture template dataset, where the projection dataset includes projection templates formed by furniture templates in the furniture template dataset under different camera parameters;

[0079] The deep residual network ResNet34 is trained using the projection data set to obtain the trained deep residual network ResNet34;

[0080] The sub-region containing the furniture is input into the trained deep residual network ResNet34 to obtain the output result, and the output result is input into a classification fully connected layer to obtain the model of the furniture.

[0081] The parameters of the projection template in the projection dataset (furniture model) include the 3D center position C obj , shape angle θ obj and the size s of each cuboid obj By varying these parameters, a projection dataset (furniture models) containing rich shape variations is constructed.

[0082] Exemplarily, based on the implementation of S4, an embodiment of the present application provides a method for generating a projection data set, setting the camera parameters as follows: on the sphere, the latitude is at (0°, 30°, 45°, 60°, 90°, 120°, 135°, 150°, 180°, -150°, -135°, -120°, -90°, -60°, -45°, -30°), and the longitude is at (0°, 30°, 45°, 60°), simulating different shapes of cameras. For each model in the furniture template data set, OpenGL (OpenGraphics Library, open graphics library or "open graphics library") rendering is performed under the above-mentioned different camera parameters to obtain the rendered model. In this way, we simulate the furniture template data set.

[0083] See also Figure 4 , is a schematic diagram of ResNet34 provided in an embodiment of the present application,

[0084] For the input sub-area containing furniture, the ResNet34 network is used to extract features to obtain a feature vector, and then a classification fully connected layer is passed to obtain the model θ of the furniture. obj .

[0085] Among them, the model of furniture θ obj is a model in the projected dataset.

[0086] In a possible implementation, S5: extracting the contour of the sub-region containing the furniture, uniformly sampling the contour of the sub-region containing the furniture, and obtaining a contour point set; and obtaining, according to the camera parameters, a template point set corresponding to the model of the furniture in the two-dimensional plane where the contour point set is located;

[0087] An optimization function is constructed, and the optimization function includes the distance between the contour point set and the template point set. With the goal of minimizing the optimization function, the template point set is iterated using an iterative algorithm to obtain a model of the furniture after shape optimization.

[0088] Wherein obtaining the template point set corresponding to the model of the furniture on the two-dimensional plane where the contour point set is located comprises:

[0089] Get the coordinates V of the three-dimensional center of the furniture model obj According to the rules of rectangular parallelepiped, C obj ,θ obj ,s obj get.

[0090] Project the 3D center of the furniture model to the contour point set P under the input camera parameters mask In the picture where the projection point is located, the coordinate calculation formula of the projection point is Use OpenCV to extract the convex hull of the projection points, and then evenly sample the points to get the point set P obj .

[0091] Among them, R(θ cam ,ψ cam ) is the camera external parameter, θ cam ,ψ cam is the rotation angle of the camera, K cam is the camera internal parameter.

[0092] In a possible implementation, the optimization function is:

[0093]

[0094] Among them, P mask is the contour point set, P obj is the template point set, x, y represents P mask and P obj Point in, L(P mask ,P obj ) is P mask To P obj Point-by-point closest distance and P obj To P mask The sum of the point-by-point closest distances; p mask P mask The number of points in , p obj P obj The number of midpoints.

[0095] Methods for iterating the template point set using an iterative algorithm to optimize the shape of furniture include:

[0096] Using the Pytorch library, we can minimize L(P mask ,P obj ) is taken as the target, and the template point set is iterated, wherein the learning rate is set to 0.1 and the number of times is 500-600. The model composed of the projection points of the template point set in the three-dimensional space is the model of the furniture after shape optimization.

[0097] Construct a distance function between the contour subset and the template subset. The smaller the value of the distance function, the smaller the difference between the images composed of the two point sets. Python has efficient data processing capabilities, and using the Pytorch library for iteration can speed up optimization efficiency.

[0098] Example 2

[0099] Embodiment 2 of the present application provides a furniture restoration device, comprising:

[0100] Image acquisition module 1, used to acquire input images;

[0101] The camera parameter acquisition module 2 is used to acquire the camera parameters corresponding to the input image according to the input image;

[0102] An instance segmentation module 3 is used to perform instance segmentation on the input image to obtain a sub-region containing furniture;

[0103] The data set construction module 4 is used to construct a furniture template data set, where the furniture in the furniture template data set is composed of a cuboid;

[0104] A model retrieval module 5, configured to retrieve a model of furniture in the input image according to the furniture template dataset and the sub-region containing the furniture;

[0105] The optimization module 6 is used to optimize the shape of the furniture model according to the camera parameters corresponding to the input image.

[0106] In a possible implementation, the camera parameter acquisition module 2 is further used to input the input image into a pre-trained deep residual network ResNet34 to obtain camera parameters corresponding to the input image.

[0107] In a possible implementation, the instance segmentation module 3 is further used to perform instance segmentation on the input image using Mask R-CNN to obtain a sub-region containing furniture.

[0108] In a possible implementation, the model retrieval module 5 is further used to construct a projection dataset according to the furniture template dataset, where the projection dataset includes projection templates formed by the furniture templates in the furniture template dataset under different camera parameters;

[0109] The deep residual network ResNet34 is trained using the projection data set to obtain the trained deep residual network ResNet34;

[0110] The sub-region containing the furniture is input into the trained deep residual network ResNet34 to obtain the output result, and the output result is input into a classification fully connected layer to obtain the model of the furniture.

[0111] In a possible implementation, the optimization module 6 is further used to extract the contour of the sub-region containing the furniture, and uniformly sample the contour of the sub-region containing the furniture to obtain a contour point set;

[0112] The three-dimensional coordinates of the points of the furniture model are obtained; according to the camera parameters and the three-dimensional coordinates of the points constituting the furniture model, the points of the furniture model are projected onto the two-dimensional plane where the contour point set is located to obtain the template point set; an optimization function is constructed, and the optimization function includes the distance between the contour point set and the template point set; with the minimization of the optimization function as the goal, the template point set is iterated using an iterative algorithm to obtain the furniture model after shape optimization.

[0113] In a possible implementation, the optimization function is:

[0114] Among them, P mask is the contour point set, P obj is the template point set, x, y represents P mask and P obj Point in, L(P mask ,P obj ) is P mask To P obj Point-by-point closest distance and P obj To P mask The sum of the point-by-point closest distances; pmask P mask The number of points in , p obj P obj The number of midpoints.

[0115] In a possible implementation, the optimization module 6 is further configured to utilize the Pytorch library to minimize L(P mask ,P obj ) is taken as the target, and the template point set is iterated, wherein the learning rate is set to 0.1 and the number of times is 500-600. The model composed of the projection points of the template point set in the three-dimensional space is the model of the furniture after shape optimization.

[0116] The present application also provides an electronic device, which may include a processor, a communication interface, a memory, and at least one communication bus. The communication bus is used to realize direct connection and communication between these components. The communication interface of the electronic device in the embodiment of the present application is used to communicate signaling or data with other node devices. The processor may be an integrated circuit chip with signal processing capabilities.

[0117] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0118] The memory may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc. The memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the electronic device may execute the various steps involved in the above embodiments.

[0119] Optionally, the electronic device may further include a storage controller and an input / output unit.

[0120] The memory, storage controller, processor, peripheral interface, input / output unit and other components are directly or indirectly electrically connected to each other to achieve data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses. The processor is used to execute executable modules stored in the memory, such as software function modules or computer programs included in the electronic device.

[0121] The input and output unit is used to provide users with the task creation and to create an optional time period or preset execution time for the task to enable interaction between the user and the server. The input and output unit can be, but is not limited to, a mouse and a keyboard.

[0122] It is understood that the structure shown is for illustration only, and the electronic device may include more or fewer components shown, or have a configuration different from that shown. Each component may be implemented by hardware, software, or a combination thereof.

[0123] The embodiment of the present application also provides a computer-readable storage medium, on which instructions are stored. When the instructions are run on a computer, the method of the method embodiment is implemented when the computer program is executed by the processor. To avoid repetition, it will not be described here.

[0124] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and a part of the module, program segment or code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0125] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0126] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program codes.

[0127] The above are only embodiments of the present application and are not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0128] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0129] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

Claims

1. A furniture restoration method, characterized in that: include: Get the input image; According to the input image, the camera parameters corresponding to the input image are obtained, and the camera parameters are obtained by using the function It indicates that, , They represent the angles of rotation around the X-axis and Y-axis respectively; Performing instance segmentation on the input image to obtain a sub-region containing furniture; Constructing a furniture template data set, wherein the furniture in the furniture template data set is composed of a cuboid; Retrieving a model of the furniture in the input image according to the furniture template dataset and the sub-region containing the furniture; Optimizing the shape of the furniture model according to the camera parameters corresponding to the input image specifically includes: Extracting the contour of the sub-region containing the furniture, uniformly sampling the contour of the sub-region containing the furniture, and obtaining a contour point set; Obtaining the three-dimensional coordinates of points constituting the model of the furniture; According to the camera parameters, a template point set corresponding to the model of the furniture in the two-dimensional plane where the contour point set is located is obtained; constructing an optimization function, wherein the optimization function includes the distance between the contour point set and the template point set; With the goal of minimizing the optimization function, the template point set is iterated using an iterative algorithm to obtain a model of the furniture after shape optimization.

2. The furniture restoration method according to claim 1, characterized in that: The step of acquiring the camera parameters corresponding to the input image according to the input image comprises: The input image is input into a pre-trained deep residual network ResNet34 to obtain the camera parameters corresponding to the input image.

3. The furniture restoration method according to claim 2, characterized in that: The step of performing instance segmentation on the input image comprises: The input image is segmented by using a pre-trained Mask R-CNN to obtain the sub-region containing the furniture.

4. The furniture restoration method according to claim 2, characterized in that: The step of retrieving the model of the furniture according to the furniture template data set and the sub-region containing the furniture comprises: Constructing a projection dataset according to the furniture template dataset, wherein the projection dataset includes projection templates formed by furniture templates in the furniture template dataset under different camera parameters; Using the projection data set to train the deep residual network ResNet34 to obtain a trained deep residual network ResNet34; The sub-region containing the furniture is input into the trained deep residual network ResNe34 to obtain an output result, and the output result is input into a classification fully connected layer to obtain a model of the furniture.

5. The furniture restoration method according to claim 1, characterized in that: The optimization function is: in, is the contour point set, is the template point set, express and The point in for arrive Point-by-point closest distance and arrive The sum of the point-by-point closest distances; for The number of points in for The number of midpoints.

6. The furniture restoration method according to claim 5, characterized in that: The method for optimizing the shape of the furniture model according to the camera parameters corresponding to the input image comprises: Using the Pytorch library to minimize As the target, iterate the template point set, wherein the learning rate is set to 0.1, the number of times is 500-600, and the model composed of the projection points of the template point set in three-dimensional space is the model of the furniture after shape optimization.

7. A furniture restoration device, characterized in that: include: An image acquisition module, used for acquiring an input image; A camera parameter acquisition module is used to acquire the camera parameters corresponding to the input image according to the input image. The camera parameters are obtained by using a function It indicates that, , They represent the angles of rotation around the X-axis and Y-axis respectively; An instance segmentation module, used for performing instance segmentation on the input image to obtain a sub-region containing furniture; A data set construction module, used to construct a furniture template data set, wherein the furniture in the furniture template data set is composed of a cuboid; A model retrieval module, used to retrieve a model of furniture in the input image according to the furniture template dataset and the sub-region containing the furniture; The optimization module is used to optimize the shape of the furniture model according to the camera parameters corresponding to the input image, specifically including: Extracting the contour of the sub-region containing the furniture, uniformly sampling the contour of the sub-region containing the furniture, and obtaining a contour point set; Obtaining the three-dimensional coordinates of points constituting the model of the furniture; According to the camera parameters, a template point set corresponding to the model of the furniture in the two-dimensional plane where the contour point set is located is obtained; constructing an optimization function, wherein the optimization function includes the distance between the contour point set and the template point set; With the goal of minimizing the optimization function, the template point set is iterated using an iterative algorithm to obtain a model of the furniture after shape optimization.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on a computer, the computer is enabled to execute the method according to any one of claims 1 to 6.

Images