Display device, furniture layout optimization method and related apparatus
By using display devices to perform panoramic image processing and 3D modeling of home spaces, furniture layout can be optimized, solving the problems of low efficiency and high cost in traditional home decoration layout. It provides intuitive layout optimization solutions, improves user experience, and ensures data security.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHENZHEN TAILIWEI INTELLIGENT TECHNOLOGY CO LTD
- Filing Date
- 2026-03-25
- Publication Date
- 2026-06-23
Smart Images
Figure CN122263202A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of display, and more particularly to a display device, a furniture layout optimization method, and related apparatus. Background Technology
[0002] With the increasing demand for personalized home decoration, users' need for optimized home space layout is becoming more and more prominent. A reasonable home decoration layout that matches the user's own preferences can bring a pleasant living experience.
[0003] Traditional home decoration layouts often rely on the user's subjective judgment or on-site surveys by professional designers, resulting in problems such as low efficiency, high cost, and lack of personalization. For example, when arranging living rooms, bedrooms, or dining rooms, users often encounter problems such as crowded furniture placement, low space utilization, and visual disharmony due to a lack of professional guidance. In some cases, the furniture size may even be mismatched with the space, leading to inconvenience. Summary of the Invention
[0004] This application provides a display device, a furniture layout optimization method, and related apparatus to quickly provide users with furniture layout optimization solutions and improve the efficiency of furniture layout.
[0005] In a first aspect, embodiments of this application provide a display device, including:
[0006] The display screen is configured to display images.
[0007] A controller connected to the display screen is configured to:
[0008] Acquire a panoramic image of the target space and convert the panoramic image into a planar unfolded diagram;
[0009] Based on the planar unfolded diagram, the target space is modeled to obtain a three-dimensional model of the target space;
[0010] The layout of the furniture in the target space is optimized based on the three-dimensional model to obtain an optimized layout scheme, and the display screen is controlled to display the optimized layout scheme; the optimized layout scheme includes a comparison view of the original layout and the optimized layout.
[0011] In some embodiments, the controller is configured to:
[0012] The planar unfolded diagram is cropped into multiple sub-diagrams of target size;
[0013] Furniture identification is performed on each of the sub-graphs to obtain the furniture category and bounding box coordinates in each sub-graph;
[0014] The depth of the furniture is estimated based on the bounding box coordinates, and the size of the furniture is determined based on the metadata of the panoramic image.
[0015] The target space is modeled according to the type and size of the furniture to obtain a three-dimensional model of the target space.
[0016] In some embodiments, the controller is configured to:
[0017] The furniture is segmented based on the bounding box coordinates to determine the pixel outline of the furniture;
[0018] The depth of the furniture is estimated based on the pixel contours to obtain a depth map of the furniture;
[0019] The depth map is transformed based on the metadata to determine the dimensions of the furniture.
[0020] In some embodiments, the controller is configured to:
[0021] In the unfolded plan, the furniture is labeled based on its size and type to obtain a two-dimensional layout plan of the target space;
[0022] Construct a three-dimensional framework of the target space, and convert the furniture into a three-dimensional placeholder model based on the annotation results of the furniture;
[0023] The three-dimensional placeholder model is embedded into the three-dimensional framework of the target space to obtain the three-dimensional model of the target space.
[0024] In some embodiments, the controller is configured to:
[0025] With the goal of maximizing rule compliance and space utilization, the layout of the furniture in the three-dimensional model is optimized to obtain an initial layout optimization scheme;
[0026] Obtain the user's layout preferences, update the initial layout optimization scheme based on the layout preferences, and determine the layout optimization scheme.
[0027] In some embodiments, the controller is configured to:
[0028] Based on the size and style of the target space, and the size and style of the furniture, determine whether the furniture is suitable for the target space;
[0029] If it is determined that the target furniture is not compatible with the target space, a replacement suggestion for the target furniture is output.
[0030] In some embodiments, the panoramic image includes multiple panoramic images of the target space from different viewpoints, and the controller is configured to:
[0031] Scale-invariant feature points and furniture data of the planar unfolded images corresponding to each panoramic image are extracted respectively;
[0032] Based on the scale-invariant feature points, feature matching is performed on each of the planar unfolded diagrams to determine the same feature point in each of the planar unfolded diagrams;
[0033] The pose of the camera corresponding to each panoramic image is determined based on the same feature point.
[0034] Based on the camera pose, feature points of adjacent viewpoints, and furniture data, the target space is reconstructed in three dimensions to determine the three-dimensional model of the target space.
[0035] In some embodiments, the controller is configured to:
[0036] Furniture identification is performed on the unfolded planar diagram to determine the category and bounding box coordinates of the furniture;
[0037] Based on the camera pose and the same feature point, a 3D point cloud of the target space is determined.
[0038] The furniture category and bounding box coordinates are projected onto the 3D point cloud to determine the 3D model of the target space.
[0039] In some embodiments, the controller is configured to:
[0040] In response to user feedback on the layout optimization scheme, the layout optimization scheme is adjusted, and the display screen is controlled to display the adjusted layout optimization scheme.
[0041] Secondly, embodiments of this application provide a method for optimizing furniture layout, including:
[0042] Acquire a panoramic image of the target space and convert the panoramic image into a planar unfolded diagram;
[0043] Based on the planar unfolded diagram, the target space is modeled to obtain a three-dimensional model of the target space;
[0044] The layout of the furniture in the target space is optimized based on the 3D model to obtain an optimized layout scheme, and the optimized layout scheme is displayed; the optimized layout scheme includes a comparison view of the original layout and the optimized layout.
[0045] Thirdly, this application provides an electronic device, including: a memory and a processor;
[0046] The memory is used to store computer instructions; the processor is used to execute the computer instructions stored in the memory to implement the method of any of the second aspects.
[0047] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the method of any of the second aspects.
[0048] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the method of any one of the second aspects.
[0049] The display device, furniture layout optimization method, and related apparatus provided in this application acquire a panoramic image of a target space and convert it into a planar unfolded view. Based on the planar unfolded view, the target space is modeled to obtain a three-dimensional model. The furniture layout in the target space is optimized based on the three-dimensional model to obtain an optimized layout scheme, and the display screen is controlled to show the optimized layout scheme. The optimized layout scheme includes a comparison view of the original layout and the optimized layout. With this solution, users only need to capture a panoramic image of the target space to obtain furniture layout optimization and replacement suggestions that meet their personalized needs, significantly reducing the cost and operational threshold for layout optimization and greatly improving the user experience. Simultaneously, the Bluetooth-based transmission method ensures that image data is transmitted only between local devices, avoiding privacy leaks and network dependence. Attached Figure Description
[0050] Figure 1 This is a schematic diagram of a scenario provided for an embodiment of this application;
[0051] Figure 2 This is a schematic diagram of the structure of a display device provided in an embodiment of this application;
[0052] Figure 3 A flowchart illustrating a furniture layout optimization method provided in this application embodiment. Figure 1 ;
[0053] Figure 4 A flowchart illustrating a furniture layout optimization method provided in this application embodiment. Figure 2 ;
[0054] Figure 5 A schematic diagram of a furniture layout optimization device provided in an embodiment of this application;
[0055] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0056] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0057] In the embodiments of this application, the terms "first" and "second" are used to distinguish identical or similar items with essentially the same function and effect, without limiting their order. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that the terms "first" and "second" do not necessarily imply that they are different.
[0058] It should be noted that, in the embodiments of this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.
[0059] As mentioned earlier, existing home decor layouts may suffer from issues such as cramped furniture placement, low space utilization, and visually uncoordinated effects, leading to inconvenience for users. Furthermore, due to users' limited knowledge, it is difficult for them to quickly and accurately adjust their home decor layout, resulting in a poor living experience.
[0060] To address the aforementioned issues, this application provides a method and related apparatus for optimizing furniture layout on a display device. Through localized image processing and inference technology, combined with Bluetooth transmission and interaction with a large-screen device, it generates intuitive layout optimization schemes and furniture replacement suggestions. This allows users to quickly adjust currently unreasonable layouts, significantly improving the user experience and practicality in home decoration scenarios.
[0061] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of the relevant data all comply with the relevant laws, regulations, and standards of the relevant countries and regions, have taken necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation access points for users to choose to authorize or refuse.
[0062] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.
[0063] Figure 1 This is a schematic diagram of a scenario provided for an embodiment of this application, such as... Figure 1 As shown, when optimizing the layout of furniture, users can take panoramic images of the space to be optimized (e.g., the living room) through a terminal device and upload the panoramic images to the display device.
[0064] After receiving the panoramic image, the display device can identify and analyze it to determine the type and placement of each piece of furniture in the space requiring optimization. Once this information is determined, the display device can optimize the current furniture layout and output a revised layout plan. Users can then adjust the furniture arrangement based on this optimized plan, effectively improving the efficiency and accuracy of furniture layout without requiring a professional designer to conduct on-site surveys, thus reducing the cost of furniture layout.
[0065] In some embodiments, such as Figure 1 As shown, the display device can also suggest furniture replacements for unsuitable furniture, providing users with better furniture layout options and further enhancing the user experience.
[0066] The terminal devices provided in this application can take many forms, such as smartphones, tablets, wearable devices, etc. This application does not limit the type of terminal device.
[0067] Figure 2 This is a schematic diagram of the structure of a display device 20 provided in an embodiment of this application, as shown below. Figure 2 As shown, it includes: display 21 and controller 22.
[0068] Display 21 is configured to display an image.
[0069] The controller 22 is configured to analyze the panoramic image of the target space to be optimized, based on the acquired image, to determine information such as the type and placement of each piece of furniture in the space to be optimized. After determining the furniture information, the display device can optimize the current furniture layout and output the optimized layout scheme.
[0070] The controller 22 is also configured to replace the layout scheme with a video signal and send the video signal to the display screen 21 so that the display screen 21 displays the layout scheme.
[0071] The display device provided in this application can have various implementation forms, such as a smart TV, laser projection device, monitor, electronic bulletin board, electronic table, etc. This application does not limit the type of display device.
[0072] The following is combined Figure 3 The furniture layout optimization method provided in the embodiments of this application is described with the controller as the execution subject.
[0073] Figure 3 This is a flowchart illustrating a furniture layout optimization method provided in an embodiment of this application, as shown below. Figure 3 As shown, it includes:
[0074] S301. Acquire a panoramic image of the target space and convert the panoramic image into a planar unfolded diagram.
[0075] In some embodiments, the target space may refer to the space where furniture layout optimization is to be performed. For example, a living room, bedroom, or other spaces. The target space contains multiple pieces of furniture, such as a coffee table, cabinets, sofa, and dining table.
[0076] A panoramic image can refer to an image that covers a target space in 360°, obtained through stitching together multiple frames or panoramic photography techniques. For example, a panoramic image using an equirectangular projection format.
[0077] In some embodiments, the controller can acquire panoramic images of the target space from an external source. For example, a user can capture a 360° panoramic image of the target space using the image acquisition module of a terminal device (such as the Camera2 API or a third-party SDK).
[0078] The terminal device can transmit the captured panoramic images to the controller via wired or wireless means. For example, the terminal device can transmit panoramic images to the controller via Bluetooth or Wi-Fi.
[0079] In some instances, terminal devices can send panoramic images to the controller via Bluetooth Low Energy (BLE) to reduce transmission power consumption and eliminate the need for the internet in the transmission process, thus effectively improving transmission stability.
[0080] In some embodiments, to further improve transmission efficiency and success rate, and to avoid transmission failures due to panoramic image processing, the terminal device can also segment the panoramic image and transmit the segmented image blocks sequentially.
[0081] For example, using 20KB as the basis, the panoramic image is divided into multiple image blocks, and the image blocks are transmitted sequentially.
[0082] When transmitting each image block, its MD5 value can be calculated, and both the image block and its MD5 value are sent to the controller. Upon receiving an image block, the controller can also calculate its own MD5 value and compare it to the received MD5 value. If they match, it indicates that the complete image block has been received, and a successful reception indication can be sent to the terminal device. Upon receiving this indication, the terminal device can send the next image block in the same manner. This process is repeated until all image blocks have been successfully transmitted.
[0083] If the terminal device receives a reception failure indication from the controller (MS5 value is inconsistent), it can retransmit the image block.
[0084] After receiving all the image blocks, the controller merges them into an original file according to the index order to obtain a panoramic image of the target space.
[0085] In some embodiments, to further improve transmission efficiency, the panoramic image can be compressed before transmission to reduce its size. After receiving the panoramic image, the controller can decompress it.
[0086] In some embodiments, a planar unfolded diagram may refer to a two-dimensional (2D) planar image of the target space.
[0087] After receiving the panoramic image, the controller can use a distortion correction algorithm to eliminate spherical distortion in the panoramic image, thereby converting the panoramic image into a planar unfolded image. For example, OpenCV's undistort function can be used to eliminate spherical distortion in the panoramic image.
[0088] S302. Based on the planar unfolded diagram, model the target space to obtain a three-dimensional model of the target space.
[0089] In some embodiments, after obtaining the planar unfolded drawing, the furniture in the planar unfolded drawing can be identified, and furniture identification, size extraction, spatial modeling, etc. can be completed to obtain a three-dimensional model of the target space.
[0090] For example, modeling the target space to obtain a three-dimensional model of the target space may include the following steps:
[0091] S1: Trim the planar unfolded diagram into multiple sub-diagrams of target size.
[0092] For example, the unfolded planar diagram can be cropped into multiple sub-diagrams of size 640*640, and the position of each sub-diagram in the unfolded planar diagram can be recorded for subsequent mapping and restoration. It should be understood that the size of the sub-diagrams can be set based on actual needs, and this application embodiment does not limit this.
[0093] S2: Perform furniture identification on each sub-graph to obtain the furniture category and bounding box coordinates in each sub-graph.
[0094] In some embodiments, furniture identification can be performed on each sub-graph based on a pre-trained neural network model. For example, an object detection model (such as YOLOv8n) can be used to identify furniture in each sub-graph to obtain the furniture categories and bounding box coordinates included in each sub-graph in the model output.
[0095] By cropping the unfolded planar image into multiple sub-images of target size, and using an object detection model to identify the furniture in each sub-image, the amount of furniture to be identified by the model in a single step can be effectively reduced, thus improving the accuracy of furniture identification.
[0096] S3: Estimate the depth of the furniture based on the bounding box and determine the size of the furniture based on the metadata of the panoramic image.
[0097] In some embodiments, after obtaining the bounding boxes of each piece of furniture, the depth of each piece of furniture can be estimated based on the bounding box coordinates to obtain the corresponding depth map, and combined with reference data, the actual size of each piece of furniture can be obtained.
[0098] For example, the furniture is segmented based on bounding box coordinates to determine the pixel outline of the furniture; the depth of the furniture is estimated based on the pixel outline to obtain the depth map of the furniture; the depth map is transformed based on metadata to determine the size of the furniture.
[0099] For example, bounding box coordinates can be used as cue information. A pre-trained contour segmentation model (such as MobileSAM) can be used to segment the furniture in each bounding box to obtain a binary mask for the furniture, and then the pixel contour corresponding to the furniture can be extracted from the binary mask. In the binary mask, 1 represents an object pixel and 0 represents the background.
[0100] After obtaining the pixel outline of the furniture, a depth estimation model can be used to estimate the depth of the furniture's pixel outline, resulting in a depth map of the furniture output by the model. For example, a monocular depth estimation model (such as the MiDaS model) can be used for depth estimation.
[0101] Since the dimensions in the furniture depth map are relative depths, it is necessary to combine reference data to perform a scale conversion on the relative depths to obtain the absolute dimensions of the furniture.
[0102] For example, the absolute dimensions of furniture can be obtained by performing scale transformation based on the metadata of the panoramic image or the dimensions of known objects.
[0103] Among them, the metadata of the panoramic image can be the spatial scale of the target space (room length, width and height, shooting position) and camera parameters collected by the terminal device when shooting panoramic images.
[0104] For example, one possible metadata might look like this:
[0105] metadata = {
[0106] "camera_height": 1.5, # Camera height above the ground (meters)
[0107] "focal_length": 24, # Focal length (mm)
[0108] "sensor_width": 36, # Sensor width (mm)
[0109] "image_width_px": 640, # Image pixel width
[0110] "room_height": 2.8 # Room ceiling height (meters)
[0111] }
[0112] For example, when calculating the length of a sofa, you can use the metadata to summarize the room height and the pixel height of the room height in the image to calculate the scale factor (i.e., the actual length of each pixel), and then use the scale factor, the pixel width of the sofa, and the depth of the sofa's center point to calculate the actual length of the sofa.
[0113] For example, the pixel width of the sofa in the image: width_px = 400 pixels;
[0114] The depth value at the center point of the sofa: depth = 5.0 meters;
[0115] Scale factor: scale_factor = 2.8 meters / 2800 pixels = 0.001 meters / pixel;
[0116] The actual length of the sofa is: 0.001 * 5 * 400 = 2 meters.
[0117] S4: Model the target space according to the type and size of the furniture to obtain a three-dimensional model of the target space.
[0118] In some embodiments, after obtaining the dimensions and categories of each piece of furniture, annotations can be made on the unfolded plan based on the dimensions and categories of each piece of furniture, and then three-dimensional spatial modeling can be performed based on the annotation results to obtain a three-dimensional model of the target space.
[0119] For example, in the unfolded plan, the furniture is labeled based on its size and category to obtain a two-dimensional layout plan of the target space; a three-dimensional framework of the target space is constructed, and the furniture is converted into a three-dimensional placeholder model based on the labeling results; the three-dimensional placeholder model is embedded into the three-dimensional framework of the target space to obtain a three-dimensional model of the target space.
[0120] For example, in a planar unfolded diagram, based on the furniture outlines identified in each sub-graph and the recorded positions of each sub-graph, the planar unfolded diagram is mapped to coordinates, each piece of furniture is marked in the planar unfolded diagram, and then each piece of furniture is marked based on the furniture identification results (type and size) to obtain a two-dimensional layout planar diagram of the target space.
[0121] Then, the walls, floor, ceiling, doors, and windows are segmented from the 2D layout plan. Wall cubes are created using a 3D modeling library, and door and window models are placed to construct a 3D framework of the target space. For example, pixel-level classification can be obtained through semantic segmentation models, and then 3D walls can be reconstructed using geometric constraints.
[0122] Each piece of furniture is represented by a cube. The cube is resized based on the detected furniture dimensions and categorized to obtain a 3D occupant model for that furniture. Then, based on the coordinates of this 3D occupant model (2D coordinates converted to 3D coordinates), it is filled into the 3D framework of the target space to obtain the 3D model of the target space.
[0123] S303. Optimize the layout of furniture in the target space based on the 3D model to obtain an optimized layout scheme, and control the display screen to show the optimized layout scheme; the optimized layout scheme includes a comparison view of the original layout and the optimized layout.
[0124] In some embodiments, after obtaining the three-dimensional model of the target space, the layout scheme of each piece of furniture in the three-dimensional model can be optimized based on a preset optimization strategy.
[0125] For example, with the goal of maximizing rule compliance and space utilization, the layout of furniture in the 3D model is optimized to obtain an initial layout optimization scheme; the user's layout preferences are obtained, and the initial layout optimization scheme is updated based on the layout preferences to determine the final layout optimization scheme.
[0126] For example, furniture optimization rules could include:
[0127] Ergonomic guidelines: The distance between the sofa and the TV should be ≥3 meters, the distance between the coffee table and the sofa should be 40-60cm, the aisle width should be ≥80cm, and furniture should not block doors, windows, or electrical outlets.
[0128] Rules of spatial aesthetics: symmetrical layout of furniture, centered visual focus, and placement parallel / perpendicular to the wall.
[0129] Based on the above rules, the rule compliance and space utilization of the current furniture layout are calculated using a preset evaluation algorithm. Then, a heuristic search algorithm is used to iteratively optimize the current layout scheme with the goal of maximizing rule compliance and space utilization.
[0130] For example, a genetic algorithm or simulated annealing algorithm can be used to iteratively optimize the current layout scheme to obtain an optimized initial layout scheme.
[0131] Optionally, for some TVs with insufficient computing power, layout optimization can be performed based on the cloud.
[0132] Optionally, on some TVs with insufficient computing power, the layout of each piece of furniture can be directly traversed based on the furniture optimization rules, and the layout of furniture that does not conform to the rules can be adjusted to output an optimized initial layout scheme.
[0133] In some embodiments, although the initial layout scheme may be mathematically optimal, it may not meet the user's aesthetic or special lifestyle habits. Therefore, the initial layout scheme can be adjusted based on the user's layout preferences to make the optimized scheme better meet the user's needs.
[0134] The user's layout preference can be a layout preference determined based on the user's historical layout, or a layout preference collected in real time.
[0135] For example, after obtaining the initial layout scheme, an interactive page can be displayed to obtain the user's layout preferences:
[0136] For example, the interactive page can display the following information:
[0137] "Please select your layout style: Modern Minimalist / Chinese / European";
[0138] "Please select the core of your living room: entertaining guests / audiovisual entertainment / family activities."
[0139] In some embodiments, after obtaining the initial layout scheme, the initial layout scheme can be displayed, and the user can directly modify the current layout scheme in the initial layout scheme, for example, by dragging the sofa to a new position. The user's modified scheme is then recorded as the user's layout preference.
[0140] After obtaining the user's layout preferences, these preferences can be used as new constraints for the next round of optimization.
[0141] For example, if a user selects "modern minimalism", then when calculating the rule compliance, a "minimalist score" can be added (e.g., styles with fewer pieces of furniture and simple lines score higher, or styles with more white space score higher).
[0142] If the user manually drags the sofa, its new position becomes fixed. In subsequent optimizations, the sofa's variable is locked and no longer included in the calculation.
[0143] Using the current layout (including layouts manually adjusted by the user) as the initial high-fitness individuals in the population, a new round of genetic algorithm iterations is performed only on unlocked furniture. In this way, the controller attempts to optimize the remaining space while "preserving user modifications," resulting in a new solution that both meets the user's preferences and satisfies spatial rationality. When the algorithm converges, or the user is satisfied with the current adjustment result and clicks "confirm," the final optimized layout solution is output.
[0144] In some embodiments, when the final layout optimization scheme is obtained, the controller can control the display screen to show a comparison chart of the final layout optimization scheme and the original layout scheme, so that the user can intuitively compare the optimized scheme.
[0145] In some embodiments, during the process of generating layout optimization solutions, if it is detected that a certain piece of furniture does not match the current space, replacement suggestions can be output to help users make better arrangements.
[0146] For example, based on the size and style of the target space, and the size and style of the furniture, determine whether the furniture is suitable for the target space; if it is determined that the target furniture is not suitable for the target space, output a replacement suggestion for the target furniture.
[0147] For example, the controller has a built-in lightweight furniture attribute library, which can include the following types of data:
[0148] Furniture types: sofas, coffee tables, TV cabinets, dining tables, etc.;
[0149] Compatibility parameters: size range, suitable space size, style (modern / Nordic / New Chinese);
[0150] Replacement reason template: such as "too large / incompatible style / limited functionality".
[0151] Based on the above data, if it is determined that the preset matching logic is not met, a replacement recommendation is made based on the recommendation rules.
[0152] For example, recommendation rules can be as follows:
[0153] Size matching: If the identified coffee table size is greater than 1 / 3 of the room width, a narrow coffee table is recommended (e.g., 90×60cm instead of 120×80cm).
[0154] Style matching: Based on the existing furniture styles in the panoramic image (e.g., leather sofa → modern style), recommend furniture of the same style;
[0155] Function optimization: For coffee tables without storage functions, we recommend coffee tables with storage functions; for small apartments, we recommend multi-functional folding furniture.
[0156] In some embodiments, the replacement proposal may be output in the following manner:
[0157] Output format:
[0158] The text suggests: "The current coffee table size is 120×80cm, which is beyond the range suitable for small apartments. It is recommended to replace it with a 90×60cm Nordic-style storage coffee table to increase aisle space."
[0159] Visual comparison: Displays a 3D model comparison between the original furniture and the recommended furniture.
[0160] In some embodiments, to further enhance the user experience, after obtaining the layout optimization solution, it can also be displayed to the user in the following ways:
[0161] Panoramic preview: Use the remote control's directional keys to drag and rotate the panoramic view; identified furniture is highlighted (red box + category label).
[0162] Layout switching: The remote control's confirmation button switches between "Original Layout" and "Optimized Layout," supporting 2D / 3D view switching;
[0163] Furniture Details: Select the furniture (remote control focus) and a details pop-up window will appear (size, type, replacement suggestions);
[0164] Solution saving: Save the optimized layout diagram as an image and store it locally on the display device or transfer it back to the terminal device via Bluetooth.
[0165] The furniture layout optimization method provided in this application involves acquiring a panoramic image of a target space and converting it into a planar unfolded view. Based on the planar unfolded view, the target space is modeled to obtain a three-dimensional model. The furniture layout in the target space is then optimized based on the three-dimensional model to obtain an optimized layout scheme, which is then displayed on a screen. The optimized layout scheme includes a comparison view of the original layout and the optimized layout. With this method, users only need to capture a panoramic image of the target space to obtain furniture layout optimization and replacement suggestions that meet their personalized needs. This significantly reduces the cost and operational threshold for users to perform layout optimization, greatly improving the user experience. Furthermore, the Bluetooth-based transmission method ensures that image data is transmitted only between local devices, avoiding privacy leaks and network dependence.
[0166] Figure 4 A flowchart illustrating another furniture layout optimization method provided in this application embodiment is shown below. Figure 4 As shown, it includes:
[0167] S401. Acquire a panoramic image of the target space.
[0168] In some embodiments, the panoramic image of the target space may include multiple panoramic images of the target space from different viewpoints.
[0169] For example, a panoramic image taken at the center of the target space, a panoramic image taken at corner 1 of the target space, and a panoramic image taken at corner 2 of the target space.
[0170] After obtaining multiple panoramic images, the panoramic images can be converted into corresponding planar unfolded images. The specific implementation method is similar to... Figure 3 Similar to the embodiments shown.
[0171] S402. Extract the scale-invariant feature points and furniture data of the planar unfolded map corresponding to each panoramic image.
[0172] In some embodiments, scale-invariant feature points may refer to scale-invariant feature transform (SIFT) points.
[0173] Furniture data can include the type of furniture and the bounding box coordinates of the furniture in the unfolded plan.
[0174] In some embodiments, the method of extracting furniture data from each floor plan can be similar to... Figure 3 The methods in the illustrated embodiments are similar and will not be described again here.
[0175] In some embodiments, the SIFT algorithm can be used to extract scale-invariant feature points from each planar unfolded image.
[0176] Scale-invariant feature points can include key points in each image and their corresponding descriptive information.
[0177] Keypoints can refer to the location (x, y), size, and angle of salient feature points in an image. A descriptor is a high-dimensional vector (typically 128-dimensional in SIFT) that describes the gradient distribution characteristics of the region surrounding the keypoint, used for subsequent feature alignment.
[0178] S403. Perform feature matching on each planar unfolded diagram based on scale-invariant feature points to determine the same feature point in each planar unfolded diagram.
[0179] In some embodiments, a predefined matcher can be used to determine the same feature point (also known as the feature pair of the same physical point) in the remaining planar unfolded maps based on distance matching.
[0180] For example, for a planar unfolded graph A, a matcher based on the Fast Approximate Nearest Neighbor Search Library (FLANN) can be used to employ the K-nearest neighbor search algorithm to find the two closest candidate matching points m and n in the coplanar unfolded graph for each descriptor in the planar unfolded graph A (also known as a matching pair (m, n), where m is the nearest neighbor and n is the second nearest neighbor).
[0181] Iterate through all matching pairs and calculate the distances D1 and D2 between m and n and the descriptor, respectively. If D1 and D2 satisfy a preset relationship, then the same feature point corresponding to m is selected. For example, the preset relationship can be D1 < 0.7D2.
[0182] S404. Determine the pose of the camera corresponding to each panoramic image based on the same feature point.
[0183] In some embodiments, after determining the same pair of feature points (also known as matching points, such as point a in image A and point m matched in image B), the pixel coordinates of point a and point m can be extracted from the key point lists of image A and image B respectively, and their pixel coordinates can be converted into floating-point arrays.
[0184] The essential matrix of the camera is estimated by processing the floating-point array and the camera's intrinsic parameter matrix using essential matrix-related functions (such as cv2.findEssentialMat). The essential matrix describes the rotation and translation relationship between two camera views. Then, singular value decomposition is performed on the essential matrix to obtain the camera's pose change data. The camera's pose change data includes the rotation matrix R and the translation vector t.
[0185] The pose of the first camera is set as the identity matrix (R=I, t=0), serving as the origin of the world coordinate system. Based on the camera pose change data, the poses of subsequent cameras relative to the first camera are calculated sequentially, resulting in the camera poses for each image and thus constructing the camera motion trajectory.
[0186] S405. Based on the camera pose, the same feature point, and furniture data, perform three-dimensional reconstruction of the target space to determine the three-dimensional model of the target space.
[0187] In some embodiments, after obtaining the camera pose, 3D reconstruction can be performed using triangulation.
[0188] For example, furniture identification is performed on the unfolded planar diagram to determine the furniture category and bounding box coordinates; based on the camera pose and the same feature point, the three-dimensional point cloud of the target space is determined, and the furniture category and bounding box coordinates are projected into the three-dimensional point cloud to determine the three-dimensional model of the target space.
[0189] For example, based on the camera intrinsics and the camera pose estimated in the previous step, a projection matrix between the two cameras is constructed. The projection matrix describes the mapping relationship from 3D world points to 2D image points. Using linear triangulation, the coordinates of the same feature point are projected from 2D to 3D using the projection matrix to obtain the 3D coordinates of the same feature point. Then, these coordinates are converted to standard Euclidean coordinates to obtain the 3D point cloud of the target space.
[0190] Optionally, since pose estimation and triangulation are usually performed in steps, errors accumulate as the number of images increases. To further improve the accuracy of the 3D point cloud, adjustments and optimizations can be made to the 3D point cloud.
[0191] For example, a predefined optimizer is used to project optimized 3D points onto an optimized camera plane. The objective function is to minimize the sum of squared distances between the optimized 3D points and the actual detected 2D feature points. The objective function is then optimized using a nonlinear optimization method, resulting in a high-precision 3D point cloud after global smoothing and optimization.
[0192] In some embodiments, the method for identifying furniture using a planar unfolded diagram and determining the furniture category and bounding box coordinates can refer to... Figure 3 The steps in the illustrated embodiment will not be repeated here.
[0193] After obtaining the 3D point cloud, the bounding box coordinates can be projected onto the 3D point cloud based on the projection matrix. Then, the furniture categories are labeled on the corresponding point clouds, resulting in a point cloud model of the target space with furniture labels. For the point cloud corresponding to each piece of furniture, principal component analysis can be used to find the principal axis direction of the point cloud. The length, width, and height along the principal axis direction are calculated to obtain the dimensions of each piece of furniture. Finally, the point cloud model of the target space is rendered to generate a 3D model of the target space.
[0194] S406. Optimize the layout of furniture in the target space based on the 3D model to obtain an optimized layout scheme, and control the display screen to display the optimized layout scheme.
[0195] S407. In response to user feedback on the layout optimization scheme, adjust the layout optimization scheme and control the display screen to show the adjusted layout optimization scheme.
[0196] The specific implementation methods shown in S406-S407 of this application embodiment are the same as those shown in S406-S407. Figure 3 The specific implementation methods in the illustrated embodiments are similar and will not be described again here.
[0197] The furniture layout optimization method provided in this application uses multiple panoramic images for multi-view geometric reconstruction during the three-dimensional reconstruction of the target space. By using the principle of triangulation, more accurate 3D coordinates are obtained, which effectively improves the accuracy of each piece of furniture in the target space after three-dimensional reconstruction. This improves the accuracy of the subsequently generated optimized layout scheme, thereby providing users with better and faster furniture layout optimization suggestions.
[0198] Based on the above embodiments, this application also provides a furniture layout optimization device.
[0199] Figure 5 This is a schematic diagram of the structure of the furniture layout optimization device provided in the embodiments of this application, as shown below. Figure 5 As shown, it includes:
[0200] The acquisition module 501 is used to acquire a panoramic image of the target space and convert the panoramic image into a planar unfolded image.
[0201] Modeling module 502 is used to model the target space based on the planar unfolded diagram to obtain a three-dimensional model of the target space.
[0202] The optimization module 503 is used to optimize the layout of furniture in the target space based on the 3D model, obtain the layout optimization scheme, and control the display screen to display the layout optimization scheme; the layout optimization scheme includes a comparison view of the original layout and the optimized layout.
[0203] In some embodiments, the modeling module 502 is used to crop the planar unfolded image into multiple sub-images of target size; identify furniture in each sub-image to obtain the furniture category and bounding box coordinates in each sub-image; estimate the depth of the furniture based on the bounding box coordinates and determine the size of the furniture based on the metadata of the panoramic image; and model the target space based on the furniture category and size to obtain a three-dimensional model of the target space.
[0204] In some embodiments, the modeling module 502 is used to perform contour segmentation of the furniture based on bounding box coordinates to determine the pixel contour of the furniture; to perform depth estimation of the furniture based on the pixel contour to obtain a depth map of the furniture; and to transform the depth map based on metadata to determine the size of the furniture.
[0205] In some embodiments, the modeling module 502 is used to annotate the furniture based on its size and category in the unfolded plan to obtain a two-dimensional layout plan of the target space; construct a three-dimensional framework of the target space and convert the furniture into a three-dimensional placeholder model based on the annotation results; and embed the three-dimensional placeholder model into the three-dimensional framework of the target space to obtain a three-dimensional model of the target space.
[0206] In some embodiments, the optimization module 503 is used to optimize the layout of furniture in a 3D model with the goal of maximizing rule compliance and space utilization, to obtain an initial layout optimization scheme; to obtain the user's layout preferences, and to update the initial layout optimization scheme based on the layout preferences, thereby determining the final layout optimization scheme.
[0207] In some embodiments, the optimization module 503 is used to determine whether the furniture is suitable for the target space based on the size and style of the target space, and the size and style of the furniture; if it is determined that the target furniture is not suitable for the target space, it outputs a replacement suggestion for the target furniture.
[0208] In some embodiments, the modeling module 502 is used to extract scale-invariant feature points and furniture data of the planar unfolded images corresponding to each panoramic image; perform feature matching on each planar unfolded image based on the scale-invariant feature points to determine the same feature point in each planar unfolded image; determine the pose of the camera corresponding to each panoramic image based on the same feature point; and perform three-dimensional reconstruction of the target space based on the camera pose, feature points of adjacent viewpoints, and furniture data to determine the three-dimensional model of the target space.
[0209] In some embodiments, the modeling module 502 is used to identify furniture in the unfolded planar diagram, determine the furniture category and bounding box coordinates; determine the three-dimensional point cloud of the target space based on the camera pose and the same feature point, and project the furniture category and bounding box coordinates into the three-dimensional point cloud to determine the three-dimensional model of the target space.
[0210] In some embodiments, the optimization module 503 is used to adjust the layout optimization scheme in response to user feedback on the layout optimization scheme, and control the display screen to display the adjusted layout optimization scheme.
[0211] The furniture layout optimization device provided in this application embodiment can execute the furniture layout optimization method shown in any of the above embodiments. Its specific implementation and technical effects are similar, and will not be described again here.
[0212] This application also provides an electronic device.
[0213] Figure 6 This is a schematic diagram of the structure of the electronic device 60 provided in the embodiments of this application, such as... Figure 6 As shown, the electronic device may include: a transceiver 601, a processor 602, and a memory 603. The electronic device may be a controller as described in any of the above embodiments.
[0214] The processor 602 executes computer execution instructions stored in the memory, causing the processor 602 to perform the scheme in the above embodiments. The processor 602 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.
[0215] The memory 603 is connected to the processor 602 via the system bus and completes communication between them. The memory 603 is used to store computer program instructions.
[0216] Transceiver 601 can perform the functions of receiving and sending data and instructions.
[0217] Optionally, the electronic device 60 may also include a communication interface to communicate and interact with external or internal devices, such as client devices (e.g., mobile phones, tablets). In specific implementations, if the communication interface, memory 603, and processor 602 are implemented independently, they can be interconnected via a bus to complete communication with each other.
[0218] The system bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The system bus can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus. Transceivers are used to enable communication between database access devices and other computers (e.g., clients, read-write libraries, and read-only libraries). Memory may include random access memory (RAM) and may also include non-volatile memory.
[0219] Optionally, in a specific implementation, if the communication interface, memory 603, and processor 602 are integrated on a single chip, then the communication interface, memory 603, and processor 602 can communicate through an internal interface.
[0220] This application also provides a chip for executing instructions, which is used to execute the monitoring method described in the above embodiments.
[0221] This application also provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the technical solution of the above-described monitoring method embodiment. Its implementation principle and technical effect are similar, and will not be repeated here.
[0222] In one possible implementation, a computer-readable medium may include random access memory (RAM), read-only memory (ROM), compact discread-only memory (CD-ROM) or other optical disc storage, disk storage or other magnetic storage devices, or any other medium targeted to carry or to store the required program code in the form of instructions or data structures, and accessible by a computer. Furthermore, any connection is appropriately referred to as a computer-readable medium. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL), or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. As used herein, disks and optical discs include optical discs, laser discs, optical discs, Digital Versatile Discs (DVDs), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs optically reproduce data using lasers. Combinations of the above should also be included within the scope of computer-readable media.
[0223] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the technical solution of the above-described API gateway production verification method embodiment. Its implementation principle and technical effects are similar and will not be repeated here.
[0224] In the specific implementation of the aforementioned terminal device or server, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly manifested as execution by a hardware processor, or execution by a combination of hardware and software modules within the processor.
[0225] Those skilled in the art will understand that all or part of the steps in any of the above method embodiments can be implemented by hardware associated with program instructions. The aforementioned program can be stored in a computer-readable storage medium, and when the program is executed, all or part of the steps in the above method embodiments are performed.
[0226] If the technical solution of this application is implemented in software form and sold or used as a product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the technical solution of this application can be embodied in the form of a software product, which is stored in a storage medium and includes a computer program or several instructions. This computer software product enables a computer device (which may be a personal computer, server, network device, or similar electronic device) to execute all or part of the steps of the methods in the embodiments of this application.
[0227] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.
[0228] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.
[0229] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.
[0230] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.
[0231] When integrated units / modules are implemented in hardware, the hardware can be digital circuits, analog circuits, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, the processor can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, the storage unit can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc.
[0232] If the integrated unit / module is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.
[0233] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.
[0234] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.
Claims
1. A display device, characterized in that, The display device includes: The display screen is configured to display images. A controller connected to the display screen is configured to: Acquire a panoramic image of the target space and convert the panoramic image into a planar unfolded diagram; Based on the planar unfolded diagram, the target space is modeled to obtain a three-dimensional model of the target space; The layout of the furniture in the target space is optimized based on the three-dimensional model to obtain an optimized layout scheme, and the display screen is controlled to display the optimized layout scheme; the optimized layout scheme includes a comparison view of the original layout and the optimized layout.
2. The display device according to claim 1, characterized in that, The controller is configured to: The planar unfolded diagram is cropped into multiple sub-diagrams of target size; Furniture identification is performed on each of the sub-graphs to obtain the furniture category and bounding box coordinates in each sub-graph; The depth of the furniture is estimated based on the bounding box coordinates, and the size of the furniture is determined based on the metadata of the panoramic image. The target space is modeled according to the type and size of the furniture to obtain a three-dimensional model of the target space.
3. The display device according to claim 2, characterized in that, The controller is configured to: The furniture is segmented based on the bounding box coordinates to determine the pixel outline of the furniture; The depth of the furniture is estimated based on the pixel contours to obtain a depth map of the furniture; The depth map is transformed based on the metadata to determine the dimensions of the furniture.
4. The display device according to claim 3, characterized in that, The controller is configured to: In the unfolded plan, the furniture is labeled based on its size and type to obtain a two-dimensional layout plan of the target space; Construct a three-dimensional framework of the target space, and convert the furniture into a three-dimensional placeholder model based on the annotation results of the furniture; The three-dimensional placeholder model is embedded into the three-dimensional framework of the target space to obtain the three-dimensional model of the target space.
5. The display device according to claim 2, characterized in that, The controller is configured to: With the goal of maximizing rule compliance and space utilization, the layout of the furniture in the three-dimensional model is optimized to obtain an initial layout optimization scheme; Obtain the user's layout preferences, update the initial layout optimization scheme based on the layout preferences, and determine the layout optimization scheme.
6. The display device according to claim 5, characterized in that, The controller is configured to: Based on the size and style of the target space, and the size and style of the furniture, determine whether the furniture is suitable for the target space; If it is determined that the target furniture is not compatible with the target space, a replacement suggestion for the target furniture is output.
7. The display device according to claim 1, characterized in that, The panoramic image includes multiple panoramic images of the target space from different viewpoints, and the controller is configured to: Scale-invariant feature points and furniture data of the planar unfolded images corresponding to each panoramic image are extracted respectively; Based on the scale-invariant feature points, feature matching is performed on each of the planar unfolded diagrams to determine the same feature point in each of the planar unfolded diagrams; The pose of the camera corresponding to each panoramic image is determined based on the same feature point. Based on the camera pose, the same feature point, and the furniture data, the target space is reconstructed in three dimensions to determine the three-dimensional model of the target space.
8. The display device according to claim 7, characterized in that, The controller is configured to: Furniture identification is performed on the unfolded planar diagram to determine the category and bounding box coordinates of the furniture; Based on the camera pose and the same feature point, the three-dimensional point cloud of the target space is determined; The furniture category and bounding box coordinates are projected onto the 3D point cloud to determine the 3D model of the target space.
9. The display device according to claim 7 or 8, characterized in that, The controller is configured to: In response to user feedback on the layout optimization scheme, the layout optimization scheme is adjusted, and the display screen is controlled to display the adjusted layout optimization scheme.
10. A method for optimizing furniture layout, characterized in that, include: Acquire a panoramic image of the target space and convert the panoramic image into a planar unfolded diagram; Based on the planar unfolded diagram, the target space is modeled to obtain a three-dimensional model of the target space; The layout of the furniture in the target space is optimized based on the 3D model to obtain an optimized layout scheme, and the optimized layout scheme is displayed; the optimized layout scheme includes a comparison view of the original layout and the optimized layout.
11. A computer-readable storage medium, characterized in that, It stores a computer program, which is executed by a processor to implement the method of claim 10.
12. A computer program product, characterized in that, It includes a computer program that, when executed by the controller, implements the method of claim 10.