Multi-source data fusion-based furniture ar preview space replacement matching system

Abstract

The application discloses a multi-source data fusion furniture AR preview space replacement matching system, which comprises a collection association module, a region division module, a depth splitting and supporting surface recovery module, a model matching module and an occlusion synthesis module. The system collects a current frame RGB image, an aligned original depth map and a camera pose, and associates scene points to a world coordinate system; performs semantic segmentation on the image and divides a furniture region to be replaced, a reserved object region and a background structure region; splits the depth map according to this, fits a ground plane and a wall plane, and generates a recovered ground region and a recovered wall region; then places and matches a virtual furniture three-dimensional model with the recovered region, performs pixel-by-pixel occlusion judgment based on a reserved depth map, and synthesizes and outputs an AR preview picture after furniture replacement.

Description

Technical Field

[0001] This application relates to the field of AR preview technology based on multi-source data fusion, specifically to a furniture AR preview space replacement and matching system based on multi-source data fusion. Background Technology

[0002] With the application of augmented reality (AR) technology in home furnishing scenarios, users can preview the placement of virtual furniture in a real indoor environment via mobile devices. Existing AR furniture previews typically rely on color images, depth maps, and camera position to achieve virtual-real synthesis, and use depth comparison to handle occlusion relationships between real objects and virtual models. However, in furniture replacement preview scenarios, the original furniture to be replaced still exists in the camera image and depth data. If the original depth map is used directly for occlusion determination, the original furniture will continue to occlude the newly inserted virtual furniture, leading to problems such as overlapping of old and new furniture and residual boundaries.

[0003] Furthermore, existing furniture often obscures the floor beneath it, the wall behind it, and corner areas, making it difficult for the system to accurately determine the supporting surface and reference position where virtual furniture should fit. This can easily lead to issues such as virtual furniture floating, sinking into the floor, not fitting properly against the wall, or having unreasonable dimensions. These problems are particularly pronounced for large furniture items such as sofas, beds, and cabinets. Existing solutions mostly focus on placing virtual objects on the visible plane, lacking collaborative processing of the depth information of the replaced object, semantic segmentation results, room structure geometry, and the preservation of object occlusion relationships. This makes it difficult to meet the requirements of real furniture replacement previews for spatial consistency, occlusion accuracy, and visual continuity. Summary of the Invention

[0004] This application provides a furniture AR preview space replacement and matching system based on multi-source data fusion, in order to at least solve some of the technical problems existing in the related technologies described above.

[0005] According to a first aspect of the embodiments of this application, a furniture AR preview space replacement and matching system based on multi-source data fusion is provided, comprising: The acquisition and association module is used to acquire the current frame RGB image, the original depth map aligned with the current frame RGB image, and the current frame camera pose, and associate the scene points in the original depth map with the world coordinate system; The region segmentation module is used to perform semantic segmentation on the current frame RGB image to obtain semantic category labels and instance identifiers. Based on the replacement target furniture, the pixels are divided into three regions, including the furniture to be replaced region, the object to be retained region, and the background structure region. The depth splitting and support surface restoration module is used to split the original depth map based on the three regions, generate a retained depth map and a replacement object depth map, fit the ground plane and wall plane based on the background structure region, the original depth map and the current frame camera pose, select a reference wall, and determine the spatial range of the furniture area to be replaced based on the replacement object depth map, and generate a restored ground area and a restored wall area within the spatial range; The model matching module is used to acquire the virtual furniture 3D model and simplified geometric information, and to match the virtual furniture 3D model with the restored ground area and the restored wall area to obtain the matched virtual furniture 3D model. The occlusion compositing module is used to perform pixel-by-pixel occlusion determination on the 3D model of the placed and matched virtual furniture based on the preserved depth map, and to compose and output the composite image.

[0006] According to a second aspect of the embodiments of this application, a furniture AR preview space replacement and matching method based on multi-source data fusion is also provided, comprising: Acquire the current frame RGB image, the original depth map aligned with the current frame RGB image, and the current frame camera pose, and associate the scene points in the original depth map with the world coordinate system; Semantic segmentation is performed on the current frame RGB image to obtain semantic category labels and instance identifiers. Based on the target furniture to be replaced, the pixels are divided into three regions, including the furniture to be replaced region, the object to be retained region, and the background structure region. The original depth map is split based on the three regions to generate a retained depth map and a replacement object depth map. Based on the background structure region, the original depth map and the current frame camera pose, the ground plane and the wall plane are fitted, a reference wall is selected, and the spatial range of the furniture area to be replaced is determined based on the replacement object depth map. Within the spatial range, the restored ground area and the restored wall area are generated. Obtain a 3D model of virtual furniture and its simplified geometric information, and match the 3D model of virtual furniture with the restored ground area and the restored wall area to obtain a 3D model of virtual furniture after matching. Based on the preserved depth map, pixel-by-pixel occlusion determination is performed on the 3D model of the placed and matched virtual furniture, and the composite image is synthesized and output.

[0007] According to a third aspect of the embodiments of this application, an electronic device is provided, including: a processor; a memory for storing a computer program executable by the processor; wherein the processor is configured to execute the computer program in the memory to implement the method described in the first aspect.

[0008] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided, which, when an executable computer program in the storage medium is executed by a processor, enables the implementation of the method described in the first aspect.

[0009] This application splits the original depth map into a retained depth map and a replacement object depth map according to semantic roles, allowing the depth information of the furniture to be replaced to be excluded from occlusion determination, while retaining its spatial occupancy range for supporting surface restoration, thereby avoiding incorrect occlusion of virtual furniture by the original furniture. By fitting the ground and wall planes based on the background structure region and combining them with the area of ​​the furniture to be replaced to generate restored ground and wall areas, a reliable placement benchmark can be provided for virtual furniture even when the original furniture occludes the ground, walls, or corners. Furthermore, by selecting a reference wall, incorporating optional room structure data verification, and matching simplified geometric information of the model, the bottom, back, and size range of the virtual furniture are kept consistent with the real space. In the compositing stage, occlusion determination is performed pixel-by-pixel based only on the retained depth map, ensuring that the retained object correctly occludes the virtual furniture, and the replaced furniture no longer causes visual persistence. This improves the spatial accuracy, occlusion rationality, and image continuity of the furniture replacement preview, making it particularly suitable for indoor AR replacement preview scenarios for large furniture such as sofas, beds, and cabinets.

[0010] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Furthermore, no embodiment in this disclosure is required to achieve all the effects described above. Attached Figure Description

[0011] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0012] Figure 1 This is a schematic diagram of a furniture AR preview space replacement and matching method based on multi-source data fusion, provided in an embodiment of this disclosure.

[0013] Figure 2 A flowchart illustrating the region division process provided in this embodiment of the disclosure.

[0014] Figure 3 This is a structural block diagram of the panoramic segmentation model provided in the embodiments of this disclosure.

[0015] Figure 4 A flowchart for generating the restored ground area and restored wall area provided in the embodiments of this disclosure.

[0016] Figure 5 A flowchart illustrating the image composition process provided in this embodiment of the disclosure.

[0017] Figure 6This is a schematic diagram of a furniture AR preview space replacement and matching system based on multi-source data fusion, provided in an embodiment of this disclosure.

[0018] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0019] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0020] The furniture AR preview space replacement matching method provided in this disclosure is applicable to indoor furniture replacement preview scenarios. Users, using a mobile terminal equipped with a color camera and depth perception capabilities, can replace one or more existing pieces of furniture with virtual furniture models in a room where furniture has already been placed, to preview the visual effect after replacement. The depth perception capability of the mobile terminal can originate from a LiDAR sensor, a depth estimation interface provided by the operating system, or a depth estimation algorithm based on multi-frame images. This method does not limit the specific method of acquiring depth data, only requiring the terminal to output depth information aligned pixel-by-pixel with the color image. In the replacement scenario, the original furniture still appears in the image captured by the camera and the depth data. If all depth information is used as a whole for occlusion determination, problems such as visual overlap between old and new furniture, incorrect occlusion of virtual furniture by the original furniture, and virtual furniture appearing to float or sink into the ground because it cannot know the position of the occluded ground and walls may occur. This method uses depth data according to semantic roles and performs geometric restoration of the supporting surfaces occluded by the original furniture, enabling the virtual furniture to obtain a reasonable placement reference and correct occlusion relationship at the replacement location.

[0021] The implementation process of the method described in this application will be described in detail below with reference to specific embodiments. It should be noted that this embodiment is only used to explain this application and is not intended to limit the scope of protection of this application. Conventional adjustments or substitutions of each step by those skilled in the art without departing from the concept of this application should be included in the scope of protection of this application.

[0022] Please see Figure 1 , Figure 1 The flowchart illustrates a furniture AR preview space replacement and matching method based on multi-source data fusion, according to an embodiment of the present invention. This method is executed through the multi-source data fusion furniture AR preview space replacement and matching system. Figure 1 As shown, the method includes steps S1-S5: In step S1, the current frame RGB image, the original depth map aligned with the current frame RGB image, and the current frame camera pose are acquired, and the scene points in the original depth map are associated with the world coordinate system.

[0023] In some embodiments, the mobile terminal acquires the current frame RGB image through a color camera and simultaneously obtains a raw depth map pixel-wise aligned with the RGB image through a depth sensing interface. Each pixel in the raw depth map records the depth value of the corresponding scene point along the depth direction in the current camera coordinate system. When the depth sensing interface outputs the distance value from the scene point to the camera's optical center, the system converts it into a camera coordinate system depth value defined with the same depth as the virtual furniture rendering depth. The terminal synchronously runs a Simultaneous Localization and Mapping (SLAM) algorithm to output the current frame camera pose, i.e., the camera's position and orientation in the world coordinate system. The terminal also acquires or calls the camera intrinsic parameters corresponding to the current frame RGB image. These camera intrinsic parameters are used to map the image pixel coordinates to the ray directions in the camera coordinate system. In the following descriptions, backprojection of pixels or determination of camera rays are all based on the camera intrinsic parameters, the depth values ​​in the raw depth map, and the current frame camera pose.

[0024] Using the camera pose of the current frame, the system transforms and associates scene points in the original depth map from the camera coordinate system to the world coordinate system, enabling spatial information acquired from different frames to be processed in a unified three-dimensional coordinate system. In some embodiments, when the terminal is equipped with a LiDAR sensor, the original depth map is directly output by the sensor; when no LiDAR sensor is equipped, it can be obtained by a system-level depth estimation interface or a motion reconstruction structure algorithm based on consecutive multi-frame images.

[0025] In step S2, semantic segmentation is performed on the current frame RGB image to obtain semantic category labels and instance identifiers. Based on the target furniture to be replaced, the pixels are divided into three regions, including the furniture to be replaced region, the object to be retained region, and the background structure region.

[0026] Please see Figure 2 , Figure 2 A flowchart illustrating the region division process provided in an embodiment of this disclosure is shown. Figure 2 As shown in box 201, semantic segmentation is performed on the current frame RGB image to obtain semantic category labels and instance identifiers.

[0027] Specifically, the system inputs the current frame's RGB image into the panoramic segmentation model, which simultaneously outputs instance-level segmentation results and background region segmentation results. Specifically, this embodiment employs a panoramic segmentation model based on the Mask2Former architecture, such as... Figure 3 As shown, Figure 3A block diagram of the panoramic segmentation model structure provided in this embodiment is shown, which includes a backbone network, a pixel decoder, a Transformer decoder, a classification head, and a masking head.

[0028] The backbone network extracts multi-scale feature maps from the input RGB image and outputs feature representations containing different resolution levels. The pixel decoder receives the multi-scale feature maps output by the backbone network, upsamples them layer by layer, and fuses features between adjacent scales to finally generate a pixel-wise feature representation with a resolution similar to the input image. The Transformer decoder maintains a set of learnable query vectors, each corresponding to a candidate segmentation region. The Transformer decoder interacts with each query vector and the pixel-wise feature representation output by the pixel decoder through a cross-attention mechanism, enabling each query vector to aggregate the visual information of its corresponding region. After the interaction, the classification head performs category prediction on each query vector and outputs the semantic category label of the region. The masking head performs a dot product operation on each query vector and the pixel-wise feature representation to generate a binary mask for the region. Pixels with a value of 1 in the binary mask belong to the region, and pixels with a value of 0 do not belong to the region.

[0029] The classification head and masking head both adopt a multilayer perceptron structure. The system assigns an instance identifier to each pixel based on the binary mask of each region. All pixels of the same object share the same instance identifier. Background structure pixels such as ground and walls are labeled according to semantic categories.

[0030] Regarding model training, Mask2Former uses the Hungarian matching algorithm during the training phase to establish a one-to-one correspondence between the predicted results and the real labels. The loss function consists of a weighted sum of classification loss, mask binary cross-entropy loss, and mask dice loss. After the model is pre-trained on a general panoptic segmentation dataset, it can be optionally fine-tuned on a dataset containing indoor furniture scenes to improve the recognition accuracy of furniture categories such as sofas, beds, cabinets, and coffee tables, as well as background structures such as floors, walls, and baseboards.

[0031] In box 202, the pixels are divided into three regions based on the target furniture to be replaced: the furniture to be replaced region, the object to be retained region, and the background structure region.

[0032] Specifically, after the user selects a piece of furniture as the replacement target through the interactive interface, the system divides all pixels in the current frame into three regions based on the semantic category label and instance identifier in the segmentation results. All pixels corresponding to the target furniture are assigned to the furniture to be replaced region; pixels of independent objects in the scene other than the target furniture, including other unselected furniture, people, pets, indoor plants, etc., are assigned to the retained object region; pixels of the floor, walls, ceiling, doors, windows, baseboards, etc., which belong to the fixed structure of the room, are assigned to the background structure region. The three regions cover all pixels of the image, and each pixel is assigned to only one region, with the assignment determined by the semantic category label and instance identifier; objects in the retained object region still exist in the scene after replacement, and their depth information needs to continue to participate in subsequent occlusion determination; the background structure region provides visible geometric information of the floor and walls for subsequent fitting and restoration of supporting surfaces.

[0033] In step S3, the original depth map is split based on the three regions to generate a retained depth map and a replacement object depth map. Based on the background structure region, the original depth map, and the current frame camera pose, the ground plane and the wall plane are fitted, a reference wall is selected, and the spatial range of the furniture area to be replaced is determined based on the replacement object depth map. Within the spatial range, a restored ground region and a restored wall region are generated.

[0034] In existing augmented reality occlusion handling schemes, the system uses the depth values ​​of all pixels in the original depth map for occlusion determination. During rendering, it compares the depth of virtual objects with the depth of the real scene pixel by pixel, displaying the object with the smaller depth value in front. In replacement scenes, this approach can cause existing furniture marked as replacement targets to still occlude virtual furniture. Therefore, this embodiment splits the original depth map of the same frame into two sets of depth data with different purposes, based on three types of regions.

[0035] Please see Figure 4 , Figure 4 A flowchart illustrating the generation process of the restored ground area and restored wall area provided in an embodiment of this disclosure is shown. Figure 4 As shown in box 401, the original depth map is split based on the three regions to generate a retained depth map and a replacement object depth map.

[0036] Specifically, for pixels belonging to the object region to be retained, the depth value of that pixel in the original depth map is recorded at the corresponding position in the retained depth map; for pixels belonging to the background structure region, the depth value of the original depth map is also recorded at the corresponding position in the retained depth map; for pixels belonging to the furniture to be replaced region, an invalid value is recorded at the corresponding position in the retained depth map. Therefore, the retained depth map only contains the depth information of the real objects and background structures that should still participate in the occlusion determination after the replacement, and the depth information of the furniture to be replaced has been removed from it.

[0037] For pixels belonging to the area of ​​the furniture to be replaced, the depth value of the pixel in the original depth map is recorded at the corresponding position in the depth map of the replacement object; for pixels not belonging to the area of ​​the furniture to be replaced, an invalid value is recorded at the corresponding position in the depth map of the replacement object; the depth map of the replacement object does not participate in the occupancy determination in the subsequent rendering stage, but its role is to enter the process of determining the spatial range, and to provide information on the spatial range occupied by the original furniture for the geometric restoration of the supporting surface in the following text.

[0038] Through the above-described splitting process, the depth values ​​of the original furniture are removed from the occlusion relationship, preventing them from incorrectly occluding the virtual furniture. Simultaneously, the spatial extent of the original furniture is preserved, allowing subsequent steps to infer the location of the occluded floor and walls. This embodiment's approach of classifying depth data according to semantic roles differs from existing solutions that use all depth values ​​for occlusion. This ensures that the replaced object not only exits the visual occlusion relationship but its spatial information can also serve as a basis for reconstructing the occluded area.

[0039] In box 402, the ground plane and wall plane are fitted based on the background structure region, the original depth map, and the current frame camera pose.

[0040] The system extracts pixels belonging to the ground category from the background structure region. Based on the depth values ​​of these pixels in the original depth map and the current frame camera pose, it backprojects them from the image coordinate system to ground 3D points in the world coordinate system. The Random Sample Consensus (RANSAC) plane fitting algorithm is then applied to these ground 3D points. The specific process is as follows: In each iteration, three non-collinear points are randomly selected from the ground 3D point set to determine a candidate ground plane; the distances from all ground 3D points to this candidate ground plane are calculated, and points with distances less than a preset threshold are counted as inliers; after multiple iterations, the candidate ground plane with the most inliers is selected as the fitting result and determined as the ground plane. The basic principle of the RANSAC algorithm is to fit a model consistent with the majority of data points from a dataset containing outliers through repeated random sampling and voting, exhibiting strong tolerance to noise and outliers. The preset threshold is adapted to the depth measurement accuracy and can be configured according to the measurement accuracy of the terminal depth sensor in actual implementation. For example, it can be configured as a small value on the order of millimeters for LiDAR depth data and appropriately relaxed for estimated depth data.

[0041] The system extracts pixels belonging to the wall category from the background structure region. Based on the depth values ​​of these pixels in the original depth map and the current frame camera pose, it back-projects them into 3D wall points in the world coordinate system. RANSAC plane fitting is then performed on these 3D wall points to obtain a set of wall planes. The system calculates the centroid position based on the 3D points obtained by back-projecting the corresponding pixels of the furniture area to be replaced in the original depth map and the current frame camera pose. When only one visible wall exists in the scene, the set of wall planes contains one wall plane, which is directly determined as the reference wall for attachment. When multiple visible walls exist in the scene, the set of wall planes contains multiple wall planes. The system determines the wall plane with the smallest distance from the centroid position as the reference wall for attachment, i.e., selecting the wall closest to the back of the furniture to be replaced as the reference surface for the virtual furniture to attach to.

[0042] In box 403, select the reference wall and determine the spatial range of the furniture area to be replaced based on the depth map of the replacement object. Within the spatial range, generate the restored floor area and the restored wall area.

[0043] In some embodiments, after the original furniture is marked as the replacement target, the ground beneath it and the wall behind it are obscured in the camera view. The virtual furniture needs to be placed on these invisible supporting surfaces to present a reasonable positional relationship. The system determines the spatial occupancy range of the area to be replaced by the furniture based on the depth map of the replacement object: the depth values ​​corresponding to the effective pixels in the depth map of the replacement object, combined with the set of three-dimensional points obtained by back-projection of the camera pose in the current frame, define the spatial range occupied by the original furniture in the world coordinate system. The system projects the spatial range onto the ground plane and the reference wall surface respectively to obtain the effective boundaries of the restored ground area and the restored wall area; the subsequent intersections of the camera ray with the ground plane and the reference wall surface are used to constitute the restored ground area or the restored wall area only when they fall into the corresponding effective boundaries.

[0044] Within this spatial range, the system generates a restored ground area and a restored wall area. Specifically, for each pixel within the area of ​​the furniture to be replaced, the system determines the camera ray originating from the camera's optical center and passing through that pixel based on the current frame's camera pose and camera intrinsic parameters. It then calculates the intersection point of this camera ray with the ground plane obtained from the previous steps, i.e., the ground intersection point. Simultaneously, it calculates the intersection point of this camera ray with the previously selected reference wall, i.e., the wall intersection point. The area formed by the ground intersection points within the projection range of the spatial range onto the ground plane is defined as the restored ground area, and the area formed by the wall intersection points within the projection range of the spatial range onto the reference wall is defined as the restored wall area. The restored ground area represents the location of the ground beneath the original furniture that was previously covered, and the restored wall area represents the location of the wall behind the original furniture that was previously covered. Together, they constitute the supporting surface for placing virtual furniture, providing a geometric reference for subsequent placement matching.

[0045] The above-mentioned support surface restoration process utilizes the spatial range information of the replaced object to infer the position of the invisible structural surface. Existing solutions typically only place virtual objects on visible planes, and lack effective position inference methods when the target placement position is covered by the original object. In this embodiment, the replaced object is used both as an object that needs to be removed from the occlusion relationship and as a reference source for restoring the hidden spatial structure, so that the bottom and back of the virtual furniture can establish a geometric alignment relationship with the restored ground and wall surfaces.

[0046] In some embodiments, optionally, the intersection line of the ground plane and the reference wall may be at least partially or entirely within the occlusion area of ​​the furniture to be replaced, resulting in the corner position being invisible in the image. When the intersection line is at least partially within the occlusion area of ​​the furniture to be replaced, and there is a visible intersection line segment outside the boundary of the furniture to be replaced, the system extracts the visible intersection line segment outside the boundary of the furniture to be replaced, performs line fitting on the visible intersection line segment, and extends the line fitting result into the interior of the furniture to be replaced to obtain a corner position estimate. This corner position estimate is incorporated into the input data of placement matching to determine the position where the bottom of the back of the virtual furniture simultaneously contacts the ground and the wall, improving the accuracy of aligning the bottom edge of large furniture with the corner when placed against the wall. When there is no visible intersection line segment outside the boundary of the furniture to be replaced, the system does not perform the corner position extension recovery described in this section, and placement matching still uses the recovered ground area and the recovered wall area as input.

[0047] Optionally, when the system can acquire Building Information Modeling (BIM) data or structured room geometry data provided by the terminal operating system, the system reads the recorded ground plane parameters and wall plane parameters and compares them with the planes obtained through real-time fitting as described above. Specifically, the comparison is performed by calculating the difference in the normal vector angle and the difference in plane offset between the ground plane parameters in the BIM data and the fitted ground plane, and the difference in the normal vector angle and the difference in plane offset between the wall plane parameters in the BIM data and each wall plane in the wall plane set.

[0048] When both the difference in the normal vector angle and the difference in the plane offset fall within the preset tolerance range, it indicates that the BIM data and the real-time fitting results are highly consistent on the plane. The system will incorporate the plane information located inside the area of ​​the furniture to be replaced from the BIM data or structured room geometry data into the restored ground area or restored wall area to supplement the parts of the real-time fitting results that have low accuracy due to insufficient visible points. When the difference in the normal vector angle or the difference in the plane offset exceeds the preset tolerance range, it indicates that there is a deviation between the BIM data and the current actual scene. At this time, the restored ground area and restored wall area obtained by fitting the current frame are retained, and the corresponding information in the BIM data is not used. This tolerance range can be pre-configured according to the accuracy level and depth measurement accuracy of the BIM data.

[0049] Before the comparison, the system uniformly represents the planes to be compared as standard plane equations. ,in For two planes to be compared and The difference in angle between the normal vectors is The difference in plane offset is The preset tolerance range includes angle tolerance. and offset tolerance ,when and When the condition is met, the corresponding plane is determined to satisfy the merging condition.

[0050] In step S5, a virtual furniture 3D model and simplified geometric information are obtained, and the virtual furniture 3D model is matched with the restored ground area and the restored wall area to obtain the matched virtual furniture 3D model.

[0051] The system retrieves the user-selected virtual furniture 3D model and its simplified geometric information from the furniture model library. The simplified geometric information includes the bottom contact surface, the back contact surface, and the outer bounding box. The bottom contact surface describes the planar area where the bottom of the furniture contacts the ground and the distribution of the legs. The back contact surface describes the planar position and orientation of the back of the furniture facing the wall. The outer bounding box records the maximum dimensions of the furniture in three orthogonal directions. The above information is extracted from the geometric data of the 3D model by the model processing tool or manually labeled when the furniture model is added to the library.

[0052] The specific process of placement matching is as follows: align the bottom contact surface of the virtual furniture 3D model with the ground plane in the restored ground area, so that the bottom contact surface fits against the ground plane; align the back contact surface with the reference wall, maintaining a preset distance from the reference wall. This preset distance can be configured according to the type of furniture; for example, the back of sofa furniture usually maintains a small gap with the wall; the system projects the centroid position calculated in the previous steps onto the ground plane as the initial horizontal position of the virtual furniture 3D model on the ground plane, and forms the placement matching result by defining the area occupied by the bounding box at this initial horizontal position; optionally, the user can fine-tune the horizontal position and orientation of the virtual furniture through the interactive interface.

[0053] In some embodiments, the simplified geometry information also includes a permissible scaling range, representing the maximum and minimum scaling ratios allowed for the furniture model; the system generates an available replacement space based on the extent of the restored ground area, the position against the reference wall, and the positions of adjacent retained objects obtained by back-projecting pixels in the retained object area onto the original depth map and the current frame camera pose; the available replacement space describes the maximum extent that the virtual furniture can occupy in each direction.

[0054] Compare the dimensions of the outer bounding box in each direction with the corresponding dimensions of the available replacement space in the same direction; assuming the outer bounding box is in the direction... The dimensions on are The available replacement space has a size of [size] in the corresponding direction. ,in Then the excess in that direction is Preset tolerance value Pre-configured based on the measurement error of the original depth map and the dimensioning error of the virtual furniture 3D model; when the size of the outer bounding box in any direction exceeds the size of the available replacement space in the corresponding direction, and the corresponding excess amount... Greater than the preset tolerance value When this happens, the system will output a size mismatch warning message to the user and may optionally recommend similar furniture models with more suitable sizes.

[0055] When the system performs proportional scaling adaptation, the scaling ratio is adjusted to... and the permitted scaling range Compare; when At this time, the system does not perform corresponding scaling operations on the virtual furniture 3D model to avoid scale distortion.

[0056] In step S5, based on the preserved depth map, pixel-by-pixel occlusion determination is performed on the 3D model of the placed and matched virtual furniture, and the composite image is synthesized and output.

[0057] Please see Figure 5 , Figure 5 A flowchart illustrating the image composition process provided in an embodiment of this disclosure is shown. Figure 5 As shown in box 501, pixel-by-pixel occlusion determination is performed on the 3D model of the placed and matched virtual furniture based on the preserved depth map.

[0058] The system renders the matched 3D model of the virtual furniture into the image space based on the camera pose of the current frame, obtaining the rendering result of the virtual furniture and its pixel-by-pixel rendering depth value. The rendering depth value and the preserved depth value in the preserved depth map use the same camera coordinate system, the same depth direction, and the same unit of length. Occlusion determination is based on the previously generated preserved depth map, comparing pixel by pixel; for each pixel position covered by the virtual furniture rendering result, the system obtains the preserved depth value of that position in the preserved depth map and the rendering depth value of the virtual furniture rendering result in the corresponding pixel.

[0059] When the preserved depth value is valid and less than the rendering depth value, it indicates that there is a real object closer to the camera at that location. This pixel represents the original RGB image of the current frame, and the virtual furniture is occluded at this location. When the preserved depth value is invalid or not less than the rendering depth value, this pixel represents the rendered result of the virtual furniture. Since the pixels corresponding to the area of ​​the furniture to be replaced are all invalid values ​​in the preserved depth map, the original furniture will not occlude the virtual furniture. People, pets, and other unreplaced furniture in the scene can occlude the virtual furniture according to the correct foreground-background relationship.

[0060] In box 502, based on the geometric position information provided by the restored ground area and restored wall area, the background of the eliminated image is filled in using an image inpainting algorithm. For pixels in the furniture area to be replaced that are not covered by the virtual furniture rendering result, the original furniture image needs to be eliminated to avoid visual residue; the system uses the geometric position information provided by the restored ground area and restored wall area, combined with the background pixels around the area, to fill in the background of the above pixels using an image inpainting algorithm.

[0061] In some embodiments, the image inpainting algorithm may employ a diffusion-based inpainting method or a traditional inpainting method based on partial differential equations, using known pixels around the area to be inpainted as a reference to generate a fill result that transitions continuously with the surrounding background in color and texture. Since the area to be inpainted is mostly covered by virtual furniture in the final image, with only a small visible inpainting area outside the edges of the virtual furniture, the background inpainting result is mainly used to fill pixels in the area of ​​the furniture to be replaced that are not covered by the rendering result of the virtual furniture.

[0062] In box 503, the background completion result, the current frame RGB image, and the virtual furniture rendering result are composited pixel by pixel into the final composite image and output to the terminal screen. Specifically, for pixels covered by the virtual furniture rendering result, the system selects pixel values ​​from the current frame RGB image and the virtual furniture rendering result according to the occlusion determination results of the aforementioned steps; for pixels in the area of ​​the furniture to be replaced that are not covered by the virtual furniture rendering result, the system selects the background completion result; for pixels outside the area of ​​the furniture to be replaced that are not covered by the virtual furniture rendering result, the system selects the current frame RGB image.

[0063] The composite image consists of three parts: the rendered image of the virtual furniture is located in the place matching determined, the object and background structure are preserved to maintain the real appearance in the original image and participate in occlusion according to the preserved depth map, and the area filled by the background replaces the original furniture's residual traces in the image.

[0064] In some embodiments, when semantic segmentation is performed independently frame by frame in consecutive video frames, the segmentation boundary may jitter between adjacent frames. To address this, the system backprojects the segmentation result of the previous frame, combined with the depth value of the corresponding pixel in the previous frame, into a 3D point in the world coordinate system. Then, based on the camera pose and camera intrinsic parameters of the current frame, the 3D point is projected onto the image coordinate system of the current frame to obtain the segmentation prediction result. For pixels whose prediction result matches the independent segmentation result of the current frame, the current frame result is directly used. For pixels whose prediction result does not match, in a narrow band region near the segmentation boundary, the system refers to the depth gradient of the current frame depth map at that location for determination: when there is a significant jump in depth value, it indicates that the location is at the edge of an object contour, and the segmentation result of the current frame is used; when the depth value changes gradually, the segmentation result projected from the previous frame is used.

[0065] In some embodiments, the user selects multiple pieces of furniture to replace simultaneously, and the system generates a corresponding area for each piece of furniture to be replaced. The generation of the depth map in the aforementioned steps is adjusted accordingly: the depth values ​​of all pixels corresponding to the areas of the furniture to be replaced are set to invalid values; the supporting surface restoration and placement matching are performed separately for each piece of furniture to be replaced in the aforementioned steps. The system also checks for spatial conflicts between multiple virtual furniture pieces, and prompts the user with conflict information when the bounding boxes of the virtual furniture pieces overlap in their placement positions; the occlusion relationship between the virtual furniture pieces is determined using a conventional depth buffer based on their respective rendering depth values.

[0066] Optionally, the system performs lighting adaptation on the rendering results of the virtual furniture; the system estimates the ambient lighting information of the scene from the brightness distribution of diffuse surfaces such as the ground and walls in the current frame RGB image, and uses the estimated lighting parameters to perform shading calculations when rendering the virtual furniture, and renders the projected shadow on the ground according to the positional relationship between the virtual furniture and the ground plane.

[0067] Therefore, this invention splits the original depth map into a retained depth map and a replacement object depth map according to semantic roles, removing the depth information of the furniture to be replaced from the occlusion determination. Simultaneously, it uses the spatial occupancy range of the furniture, combined with the fitted ground and wall planes, to reconstruct the occluded supporting surface, providing a placement benchmark for virtual furniture that includes ground alignment and wall contact relationships. In the occlusion determination stage, only the retained depth map is used for pixel-by-pixel comparison; the retained object correctly occludes the virtual furniture, while the original furniture no longer causes erroneous occlusion. This alleviates problems such as visual overlap between old and new furniture, virtual furniture floating or sinking into the ground, and inaccurate wall contact relationships in replacement scenes, and is particularly effective for previewing replacements of large furniture such as sofas, beds, and cabinets.

[0068] Please see Figure 6 , Figure 6This is a schematic diagram of a furniture AR preview space replacement and matching system based on multi-source data fusion, as provided in an embodiment of this application. As shown in the figure, the system includes: The acquisition and association module 601 is used to acquire the current frame RGB image, the original depth map aligned with the current frame RGB image, and the current frame camera pose, and associate the scene points in the original depth map with the world coordinate system; The region segmentation module 602 is used to perform semantic segmentation on the current frame RGB image to obtain semantic category labels and instance identifiers, and divide the pixels into three regions based on the replacement target furniture, including the furniture to be replaced region, the object to be retained region, and the background structure region; The depth splitting and support surface restoration module 603 is used to split the original depth map based on the three regions, generate a retained depth map and a replacement object depth map, fit the ground plane and wall plane based on the background structure region, the original depth map and the current frame camera pose, select a reference wall, and determine the spatial range of the furniture area to be replaced based on the replacement object depth map, and generate a restored ground area and a restored wall area within the spatial range; The model matching module 604 is used to acquire the virtual furniture 3D model and simplified geometric information, and to match the virtual furniture 3D model with the restored ground area and the restored wall area to obtain the matched virtual furniture 3D model. The occlusion compositing module 605 is used to perform pixel-by-pixel occlusion determination on the placed and matched virtual furniture 3D model based on the preserved depth map, and to compose and output the composite image.

[0069] Optionally, the region segmentation module is used to input the current frame RGB image into the panoramic segmentation model, extract multi-scale feature maps by the backbone network, and upsample and fuse the multi-scale feature maps layer by layer to generate a pixel-by-pixel feature representation. The Transformer decoder interacts with the query vector and the pixel-by-pixel feature representation, the classification head outputs semantic category labels, the mask head outputs a binary mask, and an instance identifier is generated based on the binary mask.

[0070] Optionally, the region division module is used to assign all pixels corresponding to the target furniture to be replaced to the furniture to be replaced region, assign pixels corresponding to independent objects other than the target furniture to the retained object region, and assign pixels corresponding to the fixed structure of the room to the background structure region, with each pixel being assigned to only one region.

[0071] Optionally, the depth splitting and support surface restoration module is used to record the corresponding positions of the preserved object area and the background structure area as the depth values ​​of the original depth map in the preserved depth map, and record the corresponding positions of the furniture area to be replaced as invalid values; in the replacement object depth map, the corresponding positions of the furniture area to be replaced are recorded as the depth values ​​of the original depth map, and the remaining positions are recorded as invalid values; the preserved depth map enters the pixel-by-pixel occlusion determination, and the replacement object depth map enters the spatial range determination process.

[0072] Optionally, the depth splitting and support surface restoration module is used to extract ground category pixels in the background structure region, and back-project the ground category pixels into ground 3D points in the world coordinate system according to the depth value of the ground category pixels in the original depth map and the current frame camera pose; perform random sampling consistency plane fitting on the ground 3D points, determine the three non-collinear ground 3D points selected in each iteration as candidate ground planes, count the ground 3D points with a distance less than a preset threshold as interior points, and determine the candidate ground plane with the most interior points as the ground plane.

[0073] Optionally, the depth splitting and support surface restoration module is used to extract wall category pixels in the background structure area, and back-project the wall category pixels into three-dimensional wall points in the world coordinate system according to the depth value of the wall category pixels in the original depth map and the current frame camera pose; perform random sampling consistency plane fitting on the three-dimensional wall points to obtain a set of wall planes; when the set of wall planes contains one wall plane, the wall plane is determined as the reference wall; when the set of wall planes contains multiple wall planes, the centroid position is calculated according to the three-dimensional points corresponding to the furniture area to be replaced, and the wall plane with the smallest distance from the centroid position is determined as the reference wall.

[0074] Optionally, the depth splitting and support surface restoration module is used to determine the camera ray for each pixel in the area of ​​the furniture to be replaced based on the current frame camera pose, calculate the ground intersection point of the camera ray and the ground plane, and calculate the wall intersection point of the camera ray and the reference wall; determine the area formed by all the ground intersection points as the restored ground area, and determine the area formed by all the wall intersection points as the restored wall area.

[0075] Optionally, the depth splitting and support surface restoration module is further configured to extract the visible intersection line segment outside the boundary of the furniture to be replaced when the intersection line of the ground plane and the reference wall is located within the area of ​​the furniture to be replaced, perform straight line fitting on the visible intersection line segment, extend the straight line fitting result into the interior of the furniture to be replaced to obtain the corner position estimate, and incorporate the corner position estimate into the placement matching input data.

[0076] Optionally, the depth splitting and support surface restoration module is further configured to, when acquiring BIM data or structured room geometry data, read the ground plane parameters and wall plane parameters therein, compare the ground plane parameters with the difference in the normal vector angle and the difference in plane offset of the ground plane, and compare the wall plane parameters with the difference in the normal vector angle and the difference in plane offset of the wall planes in the wall plane set; when both the difference in the normal vector angle and the difference in plane offset fall within a preset tolerance range, the plane information located inside the area of ​​the furniture to be replaced in the BIM data or structured room geometry data is incorporated into the restored ground area or the restored wall area; otherwise, the restored ground area and the restored wall area obtained by fitting according to the current frame are retained.

[0077] Each processing unit and / or module in the embodiments of this application can be implemented by an analog circuit that implements the functions described in the embodiments of this application, or by software that executes the functions described in the embodiments of this application.

[0078] Please see Figure 7 It shows a schematic diagram of the structure of an electronic device according to an embodiment of this application, which can be used to implement... Figure 1 The method in the illustrated embodiment. (As shown) Figure 7 As shown, the electronic device may include: The system includes at least one processor 701, at least one network interface 704, a user interface 703, a memory 705, and at least one communication bus 702. The communication bus 702 is used to enable connection and communication between the components. The user interface 703 may include buttons, and optionally include a standard wired or wireless interface. The network interface 704 may include, but is not limited to, a Bluetooth module, an NFC module, a Wi-Fi module, etc.

[0079] The processor 701 may include one or more processing cores and connect to various parts within the electronic device via various interfaces and lines. It implements various functions and data processing of the electronic device by running or executing instructions, programs, code sets, or instruction sets stored in the memory 705, and by accessing data in the memory 705. Optionally, the processor 701 may be implemented using at least one hardware form of DSP, FPGA, or PLA. The processor 701 may also integrate one or more combinations of CPU, GPU, and modem.

[0080] The memory 705 may include random access memory (RAM) or read-only memory (ROM). Optionally, the memory 705 includes a non-transitory computer-readable medium for storing instructions, programs, code, code sets, or instruction sets. The memory 705 may be divided into a program storage area and a data storage area, wherein the program storage area can be used to store instructions for implementing an operating system and instructions for implementing the foregoing method embodiments; the data storage area can be used to store data related to the relevant method embodiments. The memory 705 may also be at least one storage device located remotely from the processor 701. Figure 7 As shown, the memory 705, which serves as a computer storage medium, may contain an operating system, a network communication module, a user interface module, and program instructions.

[0081] In particular, the methods and / or embodiments in this application can be implemented as computer software programs. For example, the embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowchart. When the computer program is executed by processor 701, the functions defined in the methods of this application are performed.

[0082] Another embodiment of this application provides a storage medium storing computer program instructions thereon, which can be executed by a processor to implement the methods and / or technical solutions of any one or more embodiments of this application.

[0083] In the above embodiments, the descriptions of each embodiment have different focuses. Parts not described in detail in a certain embodiment can be referred to in the relevant descriptions of other embodiments. The above descriptions are merely preferred embodiments of this application and explanations of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to the technical solutions formed by specific combinations of the above technical features, but should also cover other technical solutions formed by arbitrary combinations of the above technical features or their equivalent features without departing from the inventive concept.

Claims

1. A furniture AR preview space replacement and matching system based on multi-source data fusion, characterized in that, include: The acquisition and association module is used to acquire the current frame RGB image, the original depth map aligned with the current frame RGB image, and the current frame camera pose, and associate the scene points in the original depth map with the world coordinate system; The region segmentation module is used to perform semantic segmentation on the current frame RGB image to obtain semantic category labels and instance identifiers. Based on the replacement target furniture, the pixels are divided into three regions, including the furniture to be replaced region, the object to be retained region, and the background structure region. The depth splitting and support surface restoration module is used to split the original depth map based on the three regions, generate a retained depth map and a replacement object depth map, fit the ground plane and wall plane based on the background structure region, the original depth map and the current frame camera pose, select a reference wall, and determine the spatial range of the furniture area to be replaced based on the replacement object depth map, and generate a restored ground area and a restored wall area within the spatial range; The model matching module is used to acquire the virtual furniture 3D model and simplified geometric information, and to match the virtual furniture 3D model with the restored ground area and the restored wall area to obtain the matched virtual furniture 3D model. The occlusion compositing module is used to perform pixel-by-pixel occlusion determination on the 3D model of the placed and matched virtual furniture based on the preserved depth map, and to compose and output the composite image.

2. The system according to claim 1, characterized in that, The region segmentation module is used to input the current frame RGB image into the panoramic segmentation model, where the backbone network extracts multi-scale feature maps, and the pixel decoder upsamples and fuses the multi-scale feature maps layer by layer to generate pixel-by-pixel feature representations. The Transformer decoder interacts with the query vector and the pixel-by-pixel feature representation. The classification head outputs a semantic category label, the mask head outputs a binary mask, and an instance identifier is generated based on the binary mask.

3. The system according to claim 1, characterized in that, The region division module is used to assign all pixels corresponding to the target furniture to be replaced to the furniture to be replaced region, assign pixels corresponding to independent objects other than the target furniture to the retained object region, and assign pixels corresponding to the fixed structure of the room to the background structure region, with each pixel being assigned to only one region.

4. The system according to claim 3, characterized in that, The depth splitting and support surface restoration module is used to record the corresponding positions of the retained object area and the background structure area as the depth values ​​of the original depth map in the retained depth map, and to record the corresponding positions of the furniture area to be replaced as invalid values. In the depth map of the replacement object, the position corresponding to the area of ​​the furniture to be replaced is recorded as the depth value of the original depth map, and the remaining positions are recorded as invalid values; the retained depth map enters the pixel-by-pixel occlusion determination, and the depth map of the replacement object enters the process of determining the spatial range.

5. The system according to claim 4, characterized in that, The depth splitting and support surface restoration module is used to extract ground category pixels in the background structure region. Based on the depth value of the ground category pixels in the original depth map and the current frame camera pose, the ground category pixels are back-projected into ground 3D points in the world coordinate system. Random sampling consistency plane fitting is performed on the ground 3D points. The three non-collinear ground 3D points selected in each iteration are determined as candidate ground planes. Ground 3D points with a distance less than a preset threshold are counted as interior points. The candidate ground plane with the most interior points is determined as the ground plane.

6. The system according to claim 5, characterized in that, The depth splitting and support surface restoration module is used to extract wall category pixels in the background structure area, and back-project the wall category pixels into 3D wall points in the world coordinate system based on the depth value of the wall category pixels in the original depth map and the current frame camera pose. Random sampling consistency plane fitting is performed on the 3D wall points to obtain a set of wall planes. When the set of wall planes contains one wall plane, the wall plane is determined as the reference wall. When the set of wall planes contains multiple wall planes, the centroid position is calculated based on the 3D points corresponding to the furniture area to be replaced, and the wall plane with the smallest distance from the centroid position is determined as the reference wall.

7. The system according to claim 6, characterized in that, The depth splitting and support surface restoration module is used to determine the camera ray for each pixel in the area of ​​the furniture to be replaced based on the current frame camera pose, calculate the intersection point of the camera ray with the ground plane, and calculate the intersection point of the camera ray with the wall surface of the reference wall. The area formed by all the ground intersections is defined as the restored ground area, and the area formed by all the wall intersections is defined as the restored wall area.

8. The system according to claim 7, characterized in that, The depth splitting and support surface restoration module is also used to extract the visible intersection line segment outside the boundary of the furniture to be replaced when the intersection line of the ground plane and the reference wall is located in the area of ​​the furniture to be replaced, perform straight line fitting on the visible intersection line segment, extend the straight line fitting result into the interior of the furniture to be replaced to obtain the corner position estimate, and incorporate the corner position estimate into the placement matching input data.

9. The system according to claim 7, characterized in that, The depth splitting and support surface restoration module is also used to read the ground plane parameters and wall plane parameters when acquiring BIM data or structured room geometry data, compare the ground plane parameters with the difference in the angle between the normal vectors of the ground plane and the difference in the plane offset, and compare the wall plane parameters with the difference in the angle between the normal vectors of the wall planes in the wall plane set and the difference in the plane offset. When both the difference in the angle between the normal vectors and the difference in the plane offset fall within the preset tolerance range, the plane information located inside the area of ​​the furniture to be replaced in the BIM data or the structured room geometry data is incorporated into the restored floor area or the restored wall area. Otherwise, retain the restored ground area and the restored wall area obtained by fitting the current frame.

10. A method for space replacement and matching of furniture AR preview using multi-source data fusion, executed by the system described in any one of claims 1-9, characterized in that, include: Acquire the current frame RGB image, the original depth map aligned with the current frame RGB image, and the current frame camera pose, and associate the scene points in the original depth map with the world coordinate system; Semantic segmentation is performed on the current frame RGB image to obtain semantic category labels and instance identifiers. Based on the target furniture to be replaced, the pixels are divided into three regions, including the furniture to be replaced region, the object to be retained region, and the background structure region. The original depth map is split based on the three regions to generate a retained depth map and a replacement object depth map. Based on the background structure region, the original depth map and the current frame camera pose, the ground plane and the wall plane are fitted, a reference wall is selected, and the spatial range of the furniture area to be replaced is determined based on the replacement object depth map. Within the spatial range, the restored ground area and the restored wall area are generated. Obtain a 3D model of virtual furniture and its simplified geometric information, and match the 3D model of virtual furniture with the restored ground area and the restored wall area to obtain a 3D model of virtual furniture after matching. Based on the preserved depth map, pixel-by-pixel occlusion determination is performed on the 3D model of the placed and matched virtual furniture, and the composite image is synthesized and output.

Images