Object placement point determination method and augmented reality (AR) device

Abstract

This specification provides an object placement point determination method, including generating an initial 3D point cloud corresponding to the segmentation mask based on the segmentation mask and depth image of the target image; generating a 3D virtual plane based on the initial 3D point cloud and the pixels in the segmentation mask; determining the target placement plane of the target virtual object from the 3D virtual plane based on the attribute information of the target virtual object; and automatically determining the target placement point of the target virtual object on the target placement plane based on the trajectory points of the target virtual object and the planar point set of the target placement plane. This method performs 3D geometric information analysis on the target image and quickly and accurately determines the target placement point in the 3D virtual plane corresponding to the target image based on the 3D geometric information analysis, avoiding manual interaction and saving costs.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of computer technology, and in particular to a method for determining the placement point of an object. Background Technology

[0002] Currently, there are still some shortcomings in the integration of virtual and real elements in AR (Augmented Reality) scenarios, mainly in that virtual objects cannot be well integrated into the real scene. Since the real scene seen by AR devices through the camera is often displayed as a two-dimensional image, when a three-dimensional virtual object is placed on top of an object in a two-dimensional image, it is usually done through manual interaction. This can be done by clicking to specify the placement point of the three-dimensional virtual object in the two-dimensional image, dragging to place the three-dimensional virtual object on a certain object in the two-dimensional image, or directly using the center point of the two-dimensional image as the placement point of the three-dimensional virtual object.

[0003] Determining the object placement point using the methods described above, and directly placing the 3D virtual object onto the object placement point for fusion with the 2D image, has several drawbacks. First, determining the object placement point requires the virtual-real fusion device to have human-computer interaction capabilities, increasing costs. Second, directly placing the 3D virtual object onto the 2D image results in the 3D virtual object floating directly in the 2D image, leading to extremely poor fusion between the 3D virtual object and the 2D image. Summary of the Invention

[0004] In view of this, embodiments of this specification provide two methods for determining object placement points. One or more embodiments of this specification also relate to an object placement point determination device, an augmented reality (AR) device, a computing device, a computer-readable storage medium, and a computer program, to address the technical deficiencies existing in the prior art.

[0005] According to a first aspect of the embodiments of this specification, a method for determining the placement point of an object is provided, comprising:

[0006] Based on the segmentation mask and depth image of the target image, an initial 3D point cloud corresponding to the segmentation mask is generated;

[0007] A three-dimensional virtual plane is generated based on the initial three-dimensional point cloud and the pixels in the segmentation mask;

[0008] Based on the attribute information of the target virtual object, determine the target placement plane of the target virtual object from the three-dimensional virtual plane;

[0009] Based on the trajectory points of the target virtual object and the set of planar points on the target placement plane, the target placement point of the target virtual object on the target placement plane is determined.

[0010] According to a second aspect of the embodiments of this specification, an object placement point determination device is provided, comprising:

[0011] The 3D point cloud generation module is configured to generate an initial 3D point cloud corresponding to the segmentation mask based on the segmentation mask and depth image of the target image;

[0012] The 3D virtual plane generation module is configured to generate a 3D virtual plane based on the initial 3D point cloud and the pixels in the segmentation mask;

[0013] The target placement plane determination module is configured to determine the target placement plane of the target virtual object from the three-dimensional virtual plane based on the attribute information of the target virtual object;

[0014] The target placement point determination module is configured to determine the target placement point of the target virtual object on the target placement plane based on the trajectory points of the target virtual object and the set of planar points of the target placement plane.

[0015] According to a third aspect of the embodiments of this specification, a method for determining the placement point of an object is provided, comprising:

[0016] Define the target virtual scene;

[0017] In response to a user's first interactive operation on a target object in the target virtual scene, the three-dimensional virtual plane of the target object is determined;

[0018] Based on the user's second interactive operation on the target virtual object in the target virtual scene, the target placement plane of the target virtual object is determined from the three-dimensional virtual plane;

[0019] Based on the set of planar points of the target placement plane, determine the target placement point of the target virtual object on the target placement plane.

[0020] According to a fourth aspect of the embodiments of this specification, an augmented reality (AR) device is provided, comprising:

[0021] Memory and processor;

[0022] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, they implement the steps of the above-described method for determining the object placement point.

[0023] According to a fifth aspect of the embodiments of this specification, a computing device is provided, comprising:

[0024] Memory and processor;

[0025] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, they implement the steps of the above-described method for determining the object placement point.

[0026] According to a sixth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer-executable instructions, which, when executed by a processor, implement the steps of the above-described object placement point determination method.

[0027] According to a seventh aspect of the embodiments of this specification, a computer program is provided, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the object placement point determination method described above.

[0028] One embodiment of this specification implements a method for determining object placement points, including generating an initial 3D point cloud corresponding to the segmentation mask based on the segmentation mask and depth image of the target image; generating a 3D virtual plane based on the initial 3D point cloud and the pixels in the segmentation mask; determining the target placement plane of the target virtual object from the 3D virtual plane based on the attribute information of the target virtual object; and automatically determining the target placement point of the target virtual object on the target placement plane based on the trajectory points of the target virtual object and the planar point set of the target placement plane. This method performs 3D geometric information analysis on the target image and quickly and accurately determines the target placement point in the 3D virtual plane corresponding to the target image based on the 3D geometric information analysis, avoiding manual interaction and saving costs; at the same time, placing the target virtual object at the target placement point in the 3D virtual plane corresponding to the target image achieves a more realistic fusion with the target image through spatial coordinates of the same dimension. Attached Figure Description

[0029] Figure 1 This is a schematic diagram illustrating a specific application scenario of an object placement point determination method provided in one embodiment of this specification;

[0030] Figure 2 This is a flowchart illustrating the process of determining the placement point of an object according to one embodiment of this specification.

[0031] Figure 3 This is a schematic diagram illustrating the applicable scenarios of the RANSACN algorithm and the plane fitting algorithm based on depth images in an embodiment of the object placement point determination method provided in this specification.

[0032] Figure 4 This is a schematic diagram illustrating a scenario for determining a candidate placement point in an object placement point determination method provided in one embodiment of this specification.

[0033] Figure 5This is a schematic diagram illustrating a scenario for determining another candidate placement point in an object placement point determination method provided in one embodiment of this specification;

[0034] Figure 6 This is a flowchart illustrating the process of determining the placement point of an object according to one embodiment of this specification.

[0035] Figure 7 This is a schematic diagram of the structure of an object placement point determination device provided in one embodiment of this specification;

[0036] Figure 8 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0037] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0038] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0039] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0040] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0041] Mask: An image of the same size as the original image, marking each pixel as belonging to the foreground or background.

[0042] Panoramic segmentation: Segmenting target instance regions in a target image, such as tables and chairs, as well as common semantic regions, such as the ground and sky.

[0043] Encoding network: A neural network structure in the field of deep learning that extracts deep features from images.

[0044] Mesh surface: A method used in 3D reconstruction to represent 3D objects.

[0045] RANSACN algorithm: short for "RANdom Sampling Consensus", which estimates the parameters of a mathematical model iteratively from a set of observation datasets containing "outsiders".

[0046] Clustering algorithms employ a bottom-up strategy, first treating each object as a separate cluster, then merging these clusters according to a certain distance metric to form increasingly larger clusters, until all objects are in one cluster (the top level of the hierarchy), or until a termination condition is met.

[0047] This specification provides two methods for determining object placement points. One or more embodiments of this specification also relate to an object placement point determination device, an augmented reality (AR) device, a computing device, a computer-readable storage medium, and a computer program, which will be described in detail in the following embodiments.

[0048] See Figure 1 , Figure 1 The diagram illustrates a specific application scenario of a method for determining the placement point of an object according to an embodiment of this specification.

[0049] Figure 1 The system includes a cloud-side device 102 and an edge-side device 104. The cloud-side device 102 can be understood as a cloud server. Of course, in feasible solutions, the cloud-side device 102 can also be replaced by a physical server. The edge-side device 104 includes, but is not limited to, any virtual reality (VR) device (e.g., VR glasses) or augmented reality (AR) device. For ease of understanding, in the embodiments of this specification, the cloud-side device 102 is used as a cloud server and the edge-side device 104 is used as VR glasses for detailed description.

[0050] This specification provides a detailed description of the object placement point determination method provided in the embodiments, applied to a fusion scenario of a three-dimensional virtual object and an RGB (R(red), G(green), B(blue) color image) background image.

[0051] In specific implementation, the cloud-side device 102 receives the target image (such as...) sent by the end-side device 104. Figure 1 RGB indoor images in the image) and target virtual objects (such as Figure 1 (Digital humans in the text).

[0052] In the cloud-side device 102, the target image is input into a pre-trained image segmentation model and a depth estimation model to obtain the segmentation mask of the target image output by the image segmentation model and the depth image of the target image output by the depth estimation model. Then, based on the segmentation mask and the depth image of the target image, an initial 3D point cloud corresponding to the segmentation mask is generated. Based on the initial 3D point cloud and the pixels in the segmentation mask, a 3D virtual plane is generated. Based on the attribute information of the target virtual object, the target placement plane of the target virtual object is determined from the 3D virtual plane. Based on the trajectory points of the target virtual object and the plane point set of the target placement plane, the target placement point of the target virtual object on the target placement plane is determined.

[0053] Finally, the target virtual object is placed on the target placement plane according to the target placement point, and then rendered to generate a fused image of the target virtual object and the target image (e.g., Figure 1 The image is a fused image of a digital human standing in an indoor environment, and the fused image is sent to the end device 102 for display.

[0054] In practical applications, if the computing resources and capabilities of the edge device 104 are sufficient, the specific implementation process of determining the target placement point of the target virtual object on the target placement plane in the cloud device 102 can also be deployed on the edge device 104. The specific deployment implementation depends on the actual application and is not limited here.

[0055] The object placement point determination method provided in the embodiments of this specification establishes an integrated process from target image segmentation, 3D geometric information analysis of the target image (initial 3D point cloud and 3D virtual plane generation), plane extraction (target placement plane determination), and object placement point recommendation in the plane (target placement point of the target virtual object on the target placement plane), providing automated intelligent placement point recommendation capabilities for the virtual-real fusion of target image and target virtual object. Furthermore, this method automatically generates the 3D virtual plane of the target image and the target placement point of the target virtual object on that 3D virtual plane, providing an automated solution for downstream camera pose estimation and 3D (three-dimensional) fusion rendering services, greatly improving the efficiency and effect of virtual-real fusion.

[0056] See Figure 2 , Figure 2 A flowchart of a method for determining the placement point of an object according to an embodiment of this specification is shown, which specifically includes the following steps.

[0057] Step 202: Generate an initial 3D point cloud corresponding to the segmentation mask based on the segmentation mask and depth image of the target image.

[0058] Specifically, the method for determining the object placement point provided in the embodiments of this specification will be described as an example of its implementation on a cloud-based device.

[0059] The target image can be understood as a monocular RGB image of any type, format, resolution, and containing any content; for example, the target image can be an image containing a table in the above embodiment, or an image of a building, an interior image of a room, etc.; and the target image can be captured by any end-side device and uploaded to the cloud-side device, such as being captured by a camera and uploaded to the cloud-side device, or being captured by the camera device of a VR device and uploaded to the cloud-side device; it can also be obtained from the image database of other cloud-side devices; or it can be uploaded by the user through the end-side device.

[0060] For ease of understanding, the embodiments in this specification all use an indoor image of a room as the target image and a digital human as the target virtual object for detailed description.

[0061] In practice, the segmentation mask and depth image of the target image can be obtained quickly and accurately using a pre-trained neural network model. The specific implementation method is as follows:

[0062] Before generating the initial 3D point cloud corresponding to the segmentation mask based on the segmentation mask and depth image of the target image, the method further includes:

[0063] Identify the target image;

[0064] The target image is input into an image segmentation model to obtain a segmentation mask for the target image and the category of the pixels in the segmentation mask;

[0065] The target image is input into the depth estimation model to obtain the depth image of the target image.

[0066] Both the image segmentation model and the depth estimation model can be understood as pre-trained neural network models.

[0067] Let's take an indoor image of a room as an example, with the target image being the interior image of the room.

[0068] In practical applications, a target image is determined, and the target image is input into an image segmentation model to obtain a segmentation mask for the target image, as well as the category of each pixel in the segmentation mask. The target image is then input into a depth estimation model to obtain a depth image of the target image. This can be understood as acquiring or receiving an interior image of a target room, and then inputting the interior image of the target room into an image segmentation model to obtain a segmentation mask after segmentation of the interior image of the target room, as well as the category and corresponding instance of each pixel in the segmentation mask, such as whether each pixel belongs to the category of table, chair, wall, or carpet in the interior image of the target room, or which table each pixel belongs to if the interior image of the target room includes multiple tables.

[0069] In the segmentation mask, pixels of different categories are labeled with different numbers. For example, 0 represents pixels of category 1 (table), 1 represents pixels of category 1 (chair), and 2 represents pixels of category 1 (blanket). The size of the segmentation mask is the same as the size of the target image.

[0070] Simultaneously, the interior image of the target room is input into the depth estimation model to obtain the depth image of the interior image of the target room, wherein the depth image includes the depth value of each pixel.

[0071] After determining the segmentation mask and depth image of the target image, an initial 3D point cloud corresponding to the segmentation mask can be generated based on the segmentation mask and depth image. This enables subsequent 3D geometric information analysis of the target image, allowing for accurate determination of the target placement point in 3D space. The specific implementation method is as follows:

[0072] The step of generating an initial 3D point cloud corresponding to the segmentation mask based on the segmentation mask and depth image of the target image includes:

[0073] Determine the depth value of the pixel in the segmentation mask based on the depth image;

[0074] An initial 3D point cloud corresponding to the segmentation mask is generated based on the coordinate values ​​of the pixels in the segmentation mask, the depth value, and the camera intrinsic parameters corresponding to the target image.

[0075] The camera intrinsic parameters of the target image can be understood as the camera intrinsic parameters used to capture the target image, or the camera intrinsic parameters preset for the target image.

[0076] Specifically, based on the depth values ​​of pixels in the depth image, the depth values ​​of pixels in the corresponding segmentation mask are determined; then, based on the coordinate values ​​and depth values ​​of pixels in the segmentation mask and the camera intrinsic parameters corresponding to the target image, the initial 3D point cloud corresponding to the segmentation mask is generated, that is, the initial 3D coordinate point set corresponding to the segmentation mask.

[0077] Assume (x) i ,y i Let ) represent the coordinates of pixel i within the segmentation mask, and z represent the coordinates of pixel i within the segmentation mask i Let K be the depth value of the pixel, and K be the camera intrinsic parameter. Then, the coordinates (X, Y, Z) of pixel i within the segmentation mask in three-dimensional space are... i ,Y i Z i The calculation of ) is as described in Formula 1:

[0078] [X i Y i Z i ] T =K -1 *[x i *z i y i *z i , z i ] T Formula 1

[0079] As mentioned above, the coordinates of all pixels within the segmentation mask in three-dimensional space can be calculated using Formula 1. The coordinates of all pixels within the segmentation mask in three-dimensional space can be understood as the initial three-dimensional point cloud corresponding to the segmentation mask.

[0080] Step 204: Generate a three-dimensional virtual plane based on the initial three-dimensional point cloud and the pixels in the segmentation mask.

[0081] Specifically, given the initial 3D point cloud corresponding to the segmentation mask, a 3D virtual plane can be generated based on this initial 3D point cloud and the pixels in the segmentation mask. The specific implementation method is as follows:

[0082] The step of generating a three-dimensional virtual plane based on the initial three-dimensional point cloud and the pixels in the segmentation mask includes:

[0083] The initial 3D point cloud is fitted to a plane according to a preset plane fitting algorithm to obtain a fitting plane.

[0084] Determine the distance from the pixel in the segmentation mask to the fitting plane, and project the pixel in the fitting plane that is less than a first preset distance threshold to form a set of planar points on the fitting plane;

[0085] A three-dimensional virtual plane is generated based on the set of plane points of the fitted plane.

[0086] The preset plane fitting algorithm includes, but is not limited to, the RANSACN algorithm (RANdom Sampling Consensus) or the clustering algorithm. The clustering algorithm can be understood as calculating the normal image through the depth image, and then using the spherical clustering method to find the plane normal [a,b,c] and the plane offset d = -median(ax + by + cz) for the normal of the point set (pixels in the initial 3D point cloud). The plane is then fitted based on the plane normal and the plane offset. The first preset distance threshold can be set according to the actual application.

[0087] See Figure 3 , Figure 3 The diagram illustrates the applicable scenarios of the RANSACN algorithm and the depth image-based plane fitting algorithm in an object placement point determination method provided in one embodiment of this specification.

[0088] Depend on Figure 3 It can be seen that the RANSACN algorithm performs better when performing plane fitting in scenes with accurate segmentation and uneven ground, while the clustering algorithm performs better when performing plane fitting in scenes with inaccurate segmentation, object occlusion, or accurate segmentation and hierarchical planes.

[0089] In practice, the choice of which plane fitting algorithm to use for plane fitting can be made according to the actual application scenario, and the embodiments in this specification do not impose any limitations.

[0090] In addition, to further improve robustness, you can first use a clustering algorithm that clusters the points in the normal direction to remove noise, and then use the RANSACN algorithm to perform plane fitting on the remaining points.

[0091] Taking the RANSACN algorithm as the preset plane fitting algorithm as an example.

[0092] Specifically, the initial 3D point cloud is fitted with a plane according to the RANSACN algorithm to obtain the plane parameters [a,b,c,d] of the segmentation mask containing the plane. The fitting plane is determined based on these plane parameters. Here, the vector (a,b,c) is the plane normal of the fitting plane, and d is the suffix that satisfies all points on the plane, which plays a localization role.

[0093] Then, the distance from each pixel in the segmentation mask to the fitting plane is calculated, and pixels whose distance is less than the first preset distance threshold t are projected onto the fitting plane to form a set of planar points of the fitting plane. Then, the planar mesh can be calculated based on the set of planar points of the fitting plane (i.e. the projected pixels) to generate a three-dimensional virtual plane.

[0094] The object placement point determination method provided in the embodiments of this specification performs plane fitting on the initial three-dimensional point cloud according to a preset plane fitting algorithm to obtain a fitting plane, and determines the plane point set of the fitting plane according to the distance from the pixel point in the segmentation mask to the fitting plane, and generates a three-dimensional virtual plane according to the plane point set of the fitting plane, so that the target placement plane of the target virtual object can be determined in the three-dimensional virtual plane, so as to accurately find the target placement point of the target virtual object in the target placement plane.

[0095] After generating the 3D virtual plane, the set of planar points in this 3D virtual plane can be projected onto the target image to obtain a planar segmentation mask for the target image, enabling accurate rendering of the subsequently virtual-fused image. Specifically, the calculation of this planar segmentation mask is as described in Formula 2:

[0096] z′ i *[x′ i y′ i ,1] T =K*[X′ i ,Y′ i Z′ i ] T Formula 2

[0097] Where, X′ i ,Y′ i ,Z′ i Let (x′) be a coordinate point in a three-dimensional virtual plane. i ,y′ i ) represents the pixel coordinates of a point within the target image, and K represents the camera intrinsic parameters.

[0098] Step 206: Determine the target placement plane of the target virtual object from the three-dimensional virtual plane based on the attribute information of the target virtual object.

[0099] The target virtual object can be a three-dimensional virtual object implemented by any device, such as a digital human or a three-dimensional vase; and the attribute information of the target virtual object includes, but is not limited to, the category and volume of the target virtual object.

[0100] Specifically, after identifying the target virtual object, the target placement plane of the target virtual object can be quickly determined from the 3D virtual plane based on the category information of the target virtual object. The specific implementation method is as follows:

[0101] Determining the target placement plane of the target virtual object from the three-dimensional virtual plane based on the attribute information of the target virtual object includes:

[0102] The category of the three-dimensional virtual plane is determined based on the category of the pixels in the segmentation mask;

[0103] Based on the matching relationship between the category of the target virtual object and the category of the three-dimensional virtual plane, the target placement plane of the target virtual object is determined from the three-dimensional virtual plane.

[0104] The set of planar points in the 3D virtual plane is obtained by projecting the pixels in the segmentation mask. Therefore, if the pixels in the segmentation mask have corresponding categories, the planar points in the set of planar points in the 3D virtual plane also have corresponding category attributes. For example, if the category of a pixel in the segmentation mask is "table", the category of the planar points in the set of planar points in the 3D virtual plane is also "table".

[0105] The matching relationship between the category of the target virtual object and the category of the set of points in the three-dimensional virtual plane can be understood as being pre-set. For example, when the category of the target virtual object is a small object such as a cup, book, or pen, the corresponding category of the three-dimensional virtual plane can be a table; when the category of the target virtual object is a photograph or painting, the corresponding category of the three-dimensional virtual plane can be a wall; when the category of the target virtual object is a person or statue, the corresponding category of the three-dimensional virtual plane can be a ground, carpet, sidewalk, sports field, etc.

[0106] Take the target virtual object as an example.

[0107] Specifically, based on the category of pixels in the segmentation mask, the category of each plane in the 3D virtual plane is determined (the category of each plane in the 3D virtual plane is determined based on the category of each plane point in the plane point set of the 3D virtual plane, that is, plane points with the same category and the same instance are regarded as a plane, and the category of the plane is determined by the category of its corresponding plane point); based on the matching relationship between the category of the target virtual object and the category of the 3D virtual plane, for example, the category of the vase in the 3D virtual plane is table; then the plane in the 3D virtual plane with the category of table can be used as the target placement plane of the target virtual object: vase.

[0108] In practical applications, if there are two tables in the three-dimensional virtual plane, the plane corresponding to the table with the larger area can be determined as the target placement plane for the vase. Of course, the more suitable plane for placing the vase can also be determined as the target placement plane based on the proportional relationship between the volume of the vase and the area of ​​the table.

[0109] In practice, since the target virtual object is not placed on the boundary of the target placement plane, to save computation when calculating the target placement point of the target virtual object on the target placement plane, the boundary plane points of the target placement plane can be excluded from the calculation. After determining the target placement plane of the target virtual object, the boundary plane points of the target placement plane can be eliminated based on the projection relationship between the target virtual object and the target placement plane, thereby improving the overall computational efficiency. The specific implementation method is as follows:

[0110] After determining the target placement plane of the target virtual object from the three-dimensional virtual plane based on the attribute information of the target virtual object, the method further includes:

[0111] The target virtual object is projected onto the target placement plane, and the projection shape of the target virtual object on the target placement plane is determined;

[0112] The target radius of the projection surface of the virtual object on the target placement plane is determined based on the distance from the outline point of the projection shape to the centroid point of the projection shape.

[0113] The target segmentation mask is obtained by filtering the segmentation mask corresponding to the target placement plane based on the target radius.

[0114] The centroid can be considered roughly similar to the concept of the center of gravity (COG). If we assume that each element of the selected item (object, face, vertex, etc.) has the same mass, the centroid will be located at the equilibrium point of the selected item's center of gravity (COG).

[0115] Specifically, based on the volume of the target virtual object, the spatial points of the target virtual object are projected onto the target placement plane in the spatial coordinate system. The maximum distance from the contour points of the projection shape to the centroid of the projection shape is determined as the target radius of the projection surface of the target virtual object on the target placement plane. Then, an erosion operation is performed on the segmentation mask based on this target radius. This erosion operation can be understood as filtering in image processing. After determining the target radius, the size of the filter kernel can be determined based on the target radius. Then, a filtering operation is performed on the segmentation mask corresponding to the target placement plane based on this filter to achieve the erosion effect and obtain the filtered target segmentation mask.

[0116] After obtaining the target segmentation mask, the set of planar points of the fitting plane can be updated based on the pixels in the target segmentation mask. Then, based on the correspondence between the updated set of planar points of the fitting plane and the 3D virtual plane, the accurate set of planar points in the target placement plane can be determined. This reduces the computational workload of calculating the trajectory points of the target virtual object and the set of planar points of the target placement plane, thus improving computational efficiency. The specific implementation method is as follows:

[0117] After obtaining the target segmentation mask, the process further includes:

[0118] Determine the distance from the pixel in the target segmentation mask to the fitting plane, project the pixels whose distance is less than the first preset distance threshold onto the fitting plane, and update the plane point set of the fitting plane;

[0119] Based on the correspondence between the updated fitted plane point set and the three-dimensional virtual plane, and the correspondence between the three-dimensional virtual plane and the target placement plane, the plane point set of the target placement plane is determined.

[0120] Specifically, the distance from each pixel in the target segmentation mask to the fitting plane is calculated. Pixels whose distance is less than a first preset distance threshold are projected onto the fitting plane, and the set of points on the fitting plane is updated. Finally, the correspondence between the updated set of points on the fitting plane and the three-dimensional virtual plane is established, and the three-dimensional virtual plane is updated. Then, based on the correspondence between the three-dimensional virtual plane and the target placement plane, the set of points on the target placement plane is determined from the updated three-dimensional virtual plane.

[0121] The object placement point determination method provided in the embodiments of this specification, after obtaining the target segmentation mask after filtering the segmentation mask corresponding to the target placement plane, can update the set of planar points of the fitted plane according to the target segmentation mask, and determine the set of planar points of the target placement plane after filtering and eliminating noise according to the correspondence between the three-dimensional virtual plane and the target placement plane. This allows for improved calculation accuracy and reduced computational load when calculating the target placement point based on the set of planar points of the target placement plane and the trajectory points of the target virtual object.

[0122] Step 208: Determine the target placement point of the target virtual object on the target placement plane based on the trajectory points of the target virtual object and the set of planar points of the target placement plane.

[0123] Once the set of planar points on the target placement plane is determined, the target placement point of the target virtual object on the target placement plane can be determined based on the trajectory points of the target virtual object and the set of planar points on the target placement plane.

[0124] In practical applications, target virtual objects include two types: dynamic virtual objects and static virtual objects. Dynamic virtual objects can be understood as virtual objects that move over time, such as a walking virtual person or a jumping animal, and the trajectory points of a dynamic virtual object include at least two or more. Static virtual objects can be understood as a standing virtual person or various static ornaments (fans, vases, etc.), and the trajectory points of a static virtual object include only one.

[0125] The following section details how the target virtual object is determined on the target placement plane based on its trajectory points and the set of planar points on the target placement plane, when the target virtual object is either a dynamic virtual object or a static virtual object.

[0126] Specifically, when the target virtual object is a dynamic virtual object, the target placement point of the target virtual object on the target placement plane is determined based on the trajectory points of the target virtual object and the set of planar points on the target placement plane. The specific implementation method is as follows:

[0127] Determining the target placement point of the target virtual object on the target placement plane based on the trajectory points of the target virtual object and the set of planar points on the target placement plane includes:

[0128] The target plane point set is determined by adjusting the plane point set of the target placement plane according to the world coordinate system;

[0129] If at least two trajectory points of the target virtual object are determined, the candidate placement point of the target virtual object on the target placement plane is determined based on the target distance between the trajectory points of the target virtual object and the target plane point set plane point.

[0130] The candidate placement points are clustered, and the target placement point of the target virtual object on the target placement plane is determined based on the clustering results.

[0131] When the target virtual object has two or more trajectory points, it can be determined that the target virtual object is a dynamic virtual object, that is, a virtual object that moves with time frames.

[0132] In practical applications, since the target virtual object needs to be rendered in the rendering software with the target image to achieve virtual-real combination, the three-dimensional virtual plane corresponding to the target image is generated in the camera coordinate system. The coordinates of the three-dimensional virtual plane are not aligned with the world coordinate system in the rendering software. Therefore, in order to achieve virtual-real combination and rendering in the rendering software, the target placement plane needs to be adjusted to be aligned with the world coordinate system first.

[0133] Specifically, the specific implementation of adjusting the set of planar points on the target placement plane according to the world coordinate system to determine the target planar point set is as follows:

[0134] First, rotate the set of points on the target plane to the ground plane. Using the Blender coordinate system as a reference, take the xy plane as the ground plane, i.e., the ground normal is [0,0,1]. Calculate the angle theta between the plane normal [a,b,c] and the ground normal [0,0,1]. Based on this angle theta, the corresponding rotation matrix R can be obtained using the Rodrigues rotation formula. The Rodrigues rotation formula is a formula for calculating the new vector obtained by rotating a vector around a rotation axis by a given angle in three-dimensional space.

[0135] For example, let P denote the set of points in the plane before rotation, P xy Let P represent the set of points in the plane rotated to the xy plane. xy The calculation formula is as follows:

[0136] P xy =RP Formula 3

[0137] After determining the target plane point set, the candidate placement points of the target virtual object on the target placement plane can be determined based on the target distance between the trajectory points of the target virtual object and the plane points in the target plane point set. Subsequently, the candidate placement points can be clustered, and the target placement point of the target virtual object on the target placement plane can be accurately determined based on the clustering results.

[0138] In specific implementation, when the target virtual object is a dynamic virtual object, the method for accurately determining the candidate placement point of the target virtual object on the target placement plane based on the target distance between the trajectory points of the target virtual object and the target plane point set points is as follows:

[0139] The step of determining the candidate placement point of the target virtual object on the target placement plane based on the target distance between the trajectory points of the target virtual object and the target plane point set plane points includes:

[0140] S2. Sequentially take each plane point in the target plane point set as the starting point, move the trajectory point of the target virtual object to the target plane point set according to the starting point, and calculate the target distance from the trajectory point of the target virtual object to the plane point in the target plane point set;

[0141] S4. If the target distance is determined to be less than or equal to a second preset distance threshold, the starting point is selected as a candidate placement point; or

[0142] If it is determined that the target distances are all greater than the second preset distance threshold, the trajectory points of the target virtual object are rotated according to a preset angle, and step S2 is continued.

[0143] Specifically, each plane point in the target plane point set is taken as the starting point in turn. The trajectory points of the target virtual object are moved to the target plane point set based on the starting point. That is, the first trajectory point of the target virtual object is moved to coincide with the starting point. Then, the distance between each trajectory point of the target virtual object and each plane point in the target plane point set is calculated. Based on the distance between each trajectory point of the target virtual object and each plane point in the target plane point set, the target distance from the trajectory points of the target virtual object to the plane points in the target plane point set is determined.

[0144] Taking the trajectory points of the target virtual object, including the first trajectory point a, the second trajectory point b, and the third trajectory point c, and plane point 1 in the target plane point set as the starting point, the first trajectory point a of the target virtual object is moved to coincide with plane point 1. Then, the distance between the first trajectory point a of the target virtual object and each plane point in the target plane point set is calculated, and the minimum distance is selected as the first distance from the first trajectory point a to the target placement plane. Similarly, the second distance from the second trajectory point b to the target placement plane and the third distance from the third trajectory point a to the target placement plane are obtained. Then, the first distance, the second distance, and the third distance are added together, and the result is taken as the target distance from the trajectory points of the target virtual object to the plane points in the target plane point set when plane point 1 is the starting point.

[0145] In another implementation, the largest distance among the first, second, and third distances is selected as the target distance from the trajectory point of the target virtual object to the target plane point set, with plane point 1 as the starting point.

[0146] If the target distance is determined to be less than or equal to the second preset distance threshold, the starting point is selected as a candidate placement point, i.e., the starting point: plane point 1 is selected as the candidate placement point; the above operation is repeated to traverse the plane points in the target plane point set to determine the candidate placement point. The second preset distance threshold can be set according to actual needs, and this manual does not impose any restrictions on it.

[0147] See Figure 4 , Figure 4 This diagram illustrates a scenario for determining a candidate placement point in an object placement point determination method provided in one embodiment of this specification.

[0148] Figure 4 The trajectory points in the text can be understood as the trajectory points of the target virtual object. Figure 4The target virtual object has 6 trajectory points; the planar sampling points can be understood as the planar points in the target planar point set; the sampling points that meet the planar conditions can be understood as the candidate placement points determined by calculating the trajectory points and planar sampling points through any of the above methods.

[0149] Furthermore, after traversing the plane points in the target plane point set using the above method, if the calculated target distances are all greater than the second preset distance threshold, then the trajectory points of the target virtual object are rotated according to a preset angle, and after the rotation, the above method is continued to search for candidate placement points in the target plane point set.

[0150] For example, if the preset angle is 15 degrees, then the trajectory point of the target virtual object is rotated 15 degrees from its current position. Then, using the current angle of the trajectory point of the target virtual object, each plane point in the target plane point set is taken as the starting point in turn. The trajectory point of the target virtual object is moved to the target plane point set according to the starting point, and the target distance from the trajectory point of the target virtual object to the plane point in the target plane point set is calculated. If the target distance is determined to be less than or equal to the second preset distance threshold, the starting point is taken as a candidate placement point.

[0151] Specifically, after rotating the trajectory points of the target virtual object, the method for determining the candidate placement points of the target virtual object on the target placement plane based on the target distance between the trajectory points of the target virtual object and the target plane point set plane points is as described above, and will not be repeated here.

[0152] See Figure 5 , Figure 5 This diagram illustrates a scenario for determining another candidate placement point in an object placement point determination method provided in one embodiment of this specification.

[0153] Figure 5 The trajectory points in the text can be understood as the trajectory points of the target virtual object. Figure 5 The target virtual object has 6 trajectory points; the planar sampling points can be understood as the planar points in the target planar point set.

[0154] Specifically, the trajectory point is rotated by 15 degrees, and the target distance is calculated between the trajectory point of the target virtual object after the 15-degree rotation and the point in the plane sampling point to determine the candidate placement point.

[0155] in, Figure 5 The sampling points that meet the conditions can be understood as: after rotating the trajectory points of the target virtual object by 15 degrees, the candidate placement points are determined by calculating the trajectory points and plane sampling points using any of the above methods, and the rotation angle of the trajectory points of the target virtual object is recorded so that the target virtual object can be placed on the target placement point on the target placement plane with this rotation angle.

[0156] If a candidate placement point is still not determined after rotating the trajectory point of the target virtual object by 15 degrees, it can be rotated another 15 degrees, and the same method can be used to determine a candidate placement point. If a candidate placement point is still not found after rotating the trajectory point of the target virtual object a full circle, a second preset distance threshold can be adjusted to determine a candidate placement point in order to ensure the target virtual object is placed on the target placement plane. The specific implementation method is as follows:

[0157] The method further includes:

[0158] If the trajectory point of the target virtual object is rotated according to the preset angle and the preset conditions are met, but there is still no candidate placement point, the trajectory point of the target virtual object is rotated according to the preset angle, and the minimum target distance from the trajectory point of the target virtual object to the target plane point set plane point is calculated.

[0159] The second preset distance threshold is adjusted based on the minimum target distance to obtain the third preset distance threshold;

[0160] Based on the third preset distance threshold and the preset conditions for rotating the trajectory point of the target virtual object according to the preset angle, the target distances from the trajectory point of the target virtual object to the target plane point set plane points are calculated, and candidate placement points are determined.

[0161] The preset conditions can be understood as the trajectory point of the target virtual object having been rotated 360 degrees according to the preset angle; or the trajectory point of the target virtual object having been rotated a preset number of times (such as 10 times) according to the preset angle.

[0162] Taking the example of rotating the trajectory point of the target virtual object 10 times according to the preset conditions and the preset angle.

[0163] Specifically, if the trajectory point of the target virtual object is rotated 10 times according to the preset angle and no candidate placement point is found, the minimum target distance from the trajectory point of the target virtual object to the target plane point set plane point is determined. That is, the minimum target distance among all target distances from the trajectory point of the target virtual object to the target plane point set plane point is calculated within 10 rotations of the trajectory point of the target virtual object according to the preset angle.

[0164] Then, the minimum target distance is multiplied by a preset coefficient (e.g., 1.5) to adjust the second preset distance threshold. That is, the value obtained by multiplying the minimum target distance by the preset coefficient is used to replace the second preset distance threshold to form the third preset distance threshold.

[0165] Then, based on the third preset distance threshold, the plane points corresponding to the target plane points in the target plane point set, whose target plane distances are less than or equal to the third preset distance threshold, are selected as candidate placement points.

[0166] For example, if the trajectory point of the target virtual object is rotated 10 times according to a preset angle, and there are 10 target distances from the trajectory point of the target virtual object to the target plane point set plane point that are less than or equal to the third preset distance threshold, then the plane point with the calculated target distance will be used as the candidate placement point.

[0167] When the target virtual object is a static virtual object, all plane points in the target plane point set are considered as candidate placement points. These candidate placement points are then clustered, and the target placement point of the target virtual object on the target placement plane is determined based on the clustering results. The specific implementation method is as follows:

[0168] Determining the target placement point of the target virtual object on the target placement plane based on the trajectory points of the target virtual object and the set of planar points on the target placement plane includes:

[0169] The target plane point set is determined by adjusting the plane point set of the target placement plane according to the world coordinate system;

[0170] If the trajectory point of the target virtual object is determined to be one, all plane points in the target plane point set are determined as candidate placement points of the target virtual object on the target placement plane.

[0171] The candidate placement points are clustered, and the target placement point of the target virtual object on the target placement plane is determined based on the clustering results.

[0172] The specific implementation method for adjusting the set of planar points on the target placement plane according to the world coordinate system to determine the target planar point set is as described above and will not be repeated here.

[0173] Furthermore, when the trajectory point of the target virtual object is only one, it can be understood that the trajectory of the target virtual object remains unchanged, and it is a static virtual object.

[0174] The object placement point determination method provided in the embodiments of this specification, when the target virtual object is a static virtual object, firstly adjusts the set of planar points on the target placement plane according to the world coordinate system to determine the target planar point set; then, when the trajectory point of the target virtual object is determined to be one, all planar points in the target planar point set are determined as candidate placement points of the target virtual object on the target placement plane; then, the candidate placement points are clustered, and the target placement point of the target virtual object on the target placement plane is determined according to the clustering results, so as to ensure the accuracy of the subsequent determination of the target placement point of the target virtual object on the target placement plane based on the clustering results.

[0175] Therefore, regardless of whether the target virtual object is dynamic or static, multiple candidate placement points can be determined from the target placement plane. These candidate points can then be clustered to obtain the final target placement point of the virtual object on the target placement plane. This clustering method selects a more suitable target placement point from the candidate points, improving the combination effect between the virtual object and the target placement plane and ensuring the rationality and realism of the virtual object's placement within the target image. The specific implementation method is as follows:

[0176] The step of clustering the candidate placement points and determining the target placement point of the target virtual object on the target placement plane based on the clustering results includes:

[0177] The candidate placement points are clustered according to a preset clustering algorithm to obtain a preset number of cluster center points;

[0178] Project each cluster center point onto the target image, and calculate the distance between each cluster center point and the center point of the target image;

[0179] The target placement point of the virtual object on the target placement plane is determined based on the distance between each cluster center point and the center point of the target image.

[0180] The preset clustering algorithms include, but are not limited to, KMeans clustering (which is a partitioning-based clustering algorithm that initializes k cluster centers, calculates the distance between a sample and its center point to summarize the samples belonging to each cluster, and iteratively achieves the goal of minimizing the distance between a sample and its cluster center) or Mean-Shift clustering (the Mean-Shift algorithm is a hill-climbing algorithm based on kernel density estimation, which can be used for clustering, image segmentation, tracking, etc.); the preset number can also be set according to the actual application, for example, the preset number is 4 or 5.

[0181] This article will provide a detailed introduction using the KMeans clustering algorithm and a preset number of clusters of 4 as an example.

[0182] Specifically, candidate placement points are clustered using KMeans clustering to obtain four cluster centers. Each cluster center is projected onto the target image, and the distance between each cluster center and the center point of the target image is calculated. That is, the pixel coordinates of each of the four cluster centers projected onto the target image are calculated, and the Euclidean distance between each cluster center and the center point of the target image is calculated based on the pixel coordinates of each cluster center projected onto the target image. Finally, the cluster center with the smallest Euclidean distance is selected as the target placement point of the virtual object on the target placement plane.

[0183] This clustering method identifies precise target placement points for virtual objects from similar candidate placement points, improving the subsequent integration effect between the virtual object and the target placement plane.

[0184] The object placement point determination method provided in the embodiments of this specification includes generating an initial 3D point cloud corresponding to the segmentation mask based on the segmentation mask and depth image of the target image; generating a 3D virtual plane based on the initial 3D point cloud and the pixels in the segmentation mask; determining the target placement plane of the target virtual object from the 3D virtual plane based on the attribute information of the target virtual object; and automatically determining the target placement point of the target virtual object on the target placement plane based on the trajectory points of the target virtual object and the planar point set of the target placement plane. This method performs 3D geometric information analysis on the target image and quickly and accurately determines the target placement point in the 3D virtual plane corresponding to the target image based on the 3D geometric information analysis, avoiding manual interaction and saving costs. At the same time, placing the target virtual object at the target placement point in the 3D virtual plane corresponding to the target image achieves a more realistic fusion with the target image through spatial coordinates of the same dimension.

[0185] After determining the target placement point of the virtual object on the target placement plane, the target placement point on the target placement plane can be aligned with the virtual object in the world coordinate system. This achieves the effect of placing the virtual object at the target placement point on the target placement plane, thus realizing a combination of virtual and real elements. The specific implementation method is as follows:

[0186] After determining the target placement point of the target virtual object on the target placement plane, the method further includes:

[0187] Align the target placement point of the target placement plane with the target virtual object in the world coordinate system, and render the target virtual object and the target placement plane.

[0188] First, the normal of the target placement point is rotated to the positive Z-axis (rotation center is the world coordinate system origin (0, 0, 0)) to obtain the rotation matrix R. Then, the rotated target placement point is moved to the world coordinate system origin to obtain the offset T. [R, T] then represents the camera extrinsic parameters in the world coordinate system, with the target placement point as the world coordinate origin and the plane containing the target placement point as the XY plane. By automatically calculating the camera extrinsic parameters based on the target placement point, the target virtual object and the target placement plane can be rendered using these camera extrinsic parameters and the aforementioned plane segmentation mask, providing a technical foundation for downstream 3D fusion rendering services.

[0189] The object placement point determination method provided in the embodiments of this specification introduces panoramic segmentation (i.e., image segmentation model) into the process to achieve intelligent matching between virtual objects and placement planes; it designs a scheme of plane normal clustering and plane fitting using the RANSACN algorithm to improve the robustness of plane fitting; and the designed intelligent matching method of trajectory points and plane points intelligently recommends target placement points based on the attributes of virtual objects, image semantics, and image 3D geometric structure, avoiding cumbersome manual interaction steps, which is simple and effective, and can be applied to intelligent target placement point recommendation for both static virtual objects and dynamic virtual objects with moving trajectories, greatly improving the efficiency of virtual-environment fusion services; at the same time, based on the placement point coordinates and normals, simple rotation and translation operations can be used to obtain camera extrinsic parameters, which can be effectively applied to downstream 3D fusion rendering services.

[0190] The following is in conjunction with the appendix Figure 6 Taking the object placement point determination method provided in this specification in a virtual-real fusion scene as an example, the method will be further explained. Figure 6 The present specification illustrates a process flowchart of an object placement point determination method according to an embodiment, which specifically includes the following steps.

[0191] Step 602: Determine the target RGB image.

[0192] Step 604: Input the target RGB image into the image segmentation model to obtain the segmentation mask of the target RGB image.

[0193] Step 606: Input the target RGB image into the depth estimation model to obtain the depth image of the target RGB image.

[0194] Step 608: Perform semantic analysis on the segmentation mask of the target RGB image.

[0195] Specifically, semantic analysis of the segmentation mask of the target RGB image can be understood as determining the category and instance information of each pixel in the segmentation mask of the target RGB image. For example, if the category of a pixel is "table" and the instance information it belongs to is "table1".

[0196] Step 610: Determine the segmentation mask for the target RGB image with planar semantics.

[0197] Specifically, a segmentation mask for a target RGB image with planar semantics can be understood as a segmentation mask that carries the category and instance information of each pixel after semantic analysis of the segmentation mask of the target RGB image.

[0198] Step 612: Generate a 3D point cloud corresponding to the segmentation mask based on the segmentation mask of the target RGB image with planar semantics and the depth image of the target RGB image.

[0199] Step 614: Generate a three-dimensional virtual plane based on the segmentation mask and its corresponding three-dimensional point cloud.

[0200] Step 616: Determine the attribute information of the virtual placed object.

[0201] The attribute information of the virtual placed object includes, but is not limited to, the category of the virtual placed object.

[0202] Step 618: Based on the attribute information of the virtual object and the three-dimensional virtual plane, calculate the coordinates of the target placement point of the virtual object on the three-dimensional virtual plane.

[0203] Specifically, the detailed implementation steps for generating the 3D point cloud, the 3D virtual plane, and obtaining the target placement point in this method can be found in the above embodiments, and will not be repeated here.

[0204] The object placement point determination method provided in the embodiments of this specification introduces an image segmentation model and a depth estimation model into the method to achieve intelligent matching between the virtual object and the target RGB image; and through the intelligent matching method of the trajectory points of the virtual object and the plane points in the corresponding three-dimensional virtual plane of the target RGB image, intelligent recommendation of the target placement point is performed, avoiding cumbersome manual interaction steps, which is simple and effective.

[0205] Corresponding to the above method embodiments, this specification also provides embodiments of an object placement point determination device. Figure 7 A schematic diagram of an object placement point determination device according to one embodiment of this specification is shown. Figure 7 As shown, the device includes:

[0206] The 3D point cloud generation module 702 is configured to generate an initial 3D point cloud corresponding to the segmentation mask based on the segmentation mask and depth image of the target image.

[0207] The 3D virtual plane generation module 704 is configured to generate a 3D virtual plane based on the initial 3D point cloud and the pixels in the segmentation mask.

[0208] The target placement plane determination module 706 is configured to determine the target placement plane of the target virtual object from the three-dimensional virtual plane based on the attribute information of the target virtual object;

[0209] The target placement point determination module 708 is configured to determine the target placement point of the target virtual object on the target placement plane based on the trajectory points of the target virtual object and the set of planar points of the target placement plane.

[0210] Optionally, the method further includes:

[0211] The preprocessing module is configured as follows:

[0212] Identify the target image;

[0213] The target image is input into an image segmentation model to obtain a segmentation mask for the target image and the category of the pixels in the segmentation mask;

[0214] The target image is input into the depth estimation model to obtain the depth image of the target image.

[0215] Optionally, the 3D point cloud generation module 702 is further configured to:

[0216] Determine the depth value of the pixel in the segmentation mask based on the depth image;

[0217] An initial 3D point cloud corresponding to the segmentation mask is generated based on the coordinate values ​​of the pixels in the segmentation mask, the depth value, and the camera intrinsic parameters corresponding to the target image.

[0218] Optionally, the three-dimensional virtual plane generation module 704 is further configured to:

[0219] The initial 3D point cloud is fitted to a plane according to a preset plane fitting algorithm to obtain a fitting plane.

[0220] Determine the distance from the pixel in the segmentation mask to the fitting plane, and project the pixel in the fitting plane that is less than a first preset distance threshold to form a set of planar points on the fitting plane;

[0221] A three-dimensional virtual plane is generated based on the set of plane points of the fitted plane.

[0222] Optionally, the target placement plane determination module 706 is further configured to:

[0223] The category of the three-dimensional virtual plane is determined based on the category of the pixels in the segmentation mask;

[0224] Based on the matching relationship between the category of the target virtual object and the category of the three-dimensional virtual plane, the target placement plane of the target virtual object is determined from the three-dimensional virtual plane.

[0225] Optionally, the device further includes:

[0226] The filtering module is configured as follows:

[0227] The target virtual object is projected onto the target placement plane, and the projection shape of the target virtual object on the target placement plane is determined;

[0228] The target radius of the projection surface of the virtual object on the target placement plane is determined based on the distance from the outline point of the projection shape to the centroid point of the projection shape.

[0229] The target segmentation mask is obtained by filtering the segmentation mask corresponding to the target placement plane based on the target radius.

[0230] Optionally, the device further includes:

[0231] The planar point set determination module is configured as follows:

[0232] Determine the distance from the pixel in the target segmentation mask to the fitting plane, project the pixels whose distance is less than the first preset distance threshold onto the fitting plane, and update the plane point set of the fitting plane;

[0233] Based on the correspondence between the updated fitted plane point set and the three-dimensional virtual plane, and the correspondence between the three-dimensional virtual plane and the target placement plane, the plane point set of the target placement plane is determined.

[0234] Optionally, the target placement point determination module 708 is further configured to:

[0235] The target plane point set is determined by adjusting the plane point set of the target placement plane according to the world coordinate system;

[0236] If at least two trajectory points of the target virtual object are determined, the candidate placement point of the target virtual object on the target placement plane is determined based on the target distance between the trajectory points of the target virtual object and the target plane point set plane point.

[0237] The candidate placement points are clustered, and the target placement point of the target virtual object on the target placement plane is determined based on the clustering results.

[0238] Optionally, the target placement point determination module 708 is further configured to:

[0239] S2. Sequentially take each plane point in the target plane point set as the starting point, move the trajectory point of the target virtual object to the target plane point set according to the starting point, and calculate the target distance from the trajectory point of the target virtual object to the plane point in the target plane point set;

[0240] S4. If the target distance is determined to be less than or equal to a second preset distance threshold, the starting point is selected as a candidate placement point; or

[0241] If it is determined that the target distances are all greater than the second preset distance threshold, the trajectory points of the target virtual object are rotated according to a preset angle, and step S2 is continued.

[0242] Optionally, the device further includes:

[0243] The candidate placement point determination module is configured as follows:

[0244] If the trajectory point of the target virtual object is rotated according to the preset angle and the preset conditions are met, but there is still no candidate placement point, the trajectory point of the target virtual object is rotated according to the preset angle, and the minimum target distance from the trajectory point of the target virtual object to the target plane point set plane point is calculated.

[0245] The second preset distance threshold is adjusted based on the minimum target distance to obtain the third preset distance threshold;

[0246] Based on the third preset distance threshold and the preset conditions for rotating the trajectory point of the target virtual object according to the preset angle, the target distances from the trajectory point of the target virtual object to the target plane point set plane points are calculated, and candidate placement points are determined.

[0247] Optionally, the target placement point determination module 708 is further configured to:

[0248] The target plane point set is determined by adjusting the plane point set of the target placement plane according to the world coordinate system;

[0249] If the trajectory point of the target virtual object is determined to be one, all plane points in the target plane point set are determined as candidate placement points of the target virtual object on the target placement plane.

[0250] The candidate placement points are clustered, and the target placement point of the target virtual object on the target placement plane is determined based on the clustering results.

[0251] Optionally, the target placement point determination module 708 is further configured to:

[0252] The candidate placement points are clustered according to a preset clustering algorithm to obtain a preset number of cluster center points;

[0253] Project each cluster center point onto the target image, and calculate the distance between each cluster center point and the center point of the target image;

[0254] The target placement point of the virtual object on the target placement plane is determined based on the distance between each cluster center point and the center point of the target image.

[0255] Optionally, the device further includes:

[0256] The rendering module is configured as follows:

[0257] Align the target placement point of the target placement plane with the target virtual object in the world coordinate system, and render the target virtual object and the target placement plane.

[0258] The object placement point determination device provided in the embodiments of this specification includes: generating an initial 3D point cloud corresponding to the segmentation mask based on the segmentation mask and depth image of the target image; generating a 3D virtual plane based on the initial 3D point cloud and the pixels in the segmentation mask; determining the target placement plane of the target virtual object from the 3D virtual plane based on the attribute information of the target virtual object; and automatically determining the target placement point of the target virtual object on the target placement plane based on the trajectory points of the target virtual object and the planar point set of the target placement plane. This method performs 3D geometric information analysis on the target image and quickly and accurately determines the target placement point in the 3D virtual plane corresponding to the target image, avoiding manual interaction and saving costs. Simultaneously, placing the target virtual object at the target placement point in the 3D virtual plane corresponding to the target image achieves a more realistic fusion with the target image through spatial coordinates of the same dimension.

[0259] The above is a schematic scheme of an object placement point determination device according to this embodiment. It should be noted that the technical solution of this object placement point determination device and the technical solution of the object placement point determination method described above belong to the same concept. For details not described in detail in the technical solution of the object placement point determination device, please refer to the description of the technical solution of the object placement point determination method described above.

[0260] This specification also provides another method for determining the placement point of an object, including:

[0261] Define the target virtual scene;

[0262] In response to a user's first interactive operation on a target object in the target virtual scene, the three-dimensional virtual plane of the target object is determined;

[0263] Based on the user's second interactive operation on the target virtual object in the target virtual scene, the target placement plane of the target virtual object is determined from the three-dimensional virtual plane;

[0264] Based on the set of planar points of the target placement plane, determine the target placement point of the target virtual object on the target placement plane.

[0265] Optionally, determining the target placement plane of the target virtual object from the three-dimensional virtual plane based on the user's second interactive operation on the target virtual object in the target virtual scene includes:

[0266] Based on the user's second interactive operation on the target virtual object in the target virtual scene, and based on the attribute information of the target virtual object, the target placement plane of the target virtual object is determined from the three-dimensional virtual plane;

[0267] Accordingly, determining the target placement point of the target virtual object on the target placement plane based on the set of planar points of the target placement plane includes:

[0268] Based on the trajectory points of the target virtual object and the set of planar points on the target placement plane, the target placement point of the target virtual object on the target placement plane is determined.

[0269] The target virtual scene can be understood as any virtual scene, such as a shopping virtual scene, a decoration virtual scene, a 3D object design virtual scene, or an exhibition virtual scene. The target objects and virtual objects within a target virtual scene vary depending on the specific target virtual scene. For example, in a shopping virtual scene, the target object can be understood as a shopping cart image, and the target virtual objects can be understood as various virtual goods. In a decoration virtual scene, the target object can be understood as a window image, and the target virtual objects can be understood as various virtual furniture.

[0270] Taking a shopping virtual scene as an example, in response to the user's first interactive operation on the target object in the target virtual scene, the three-dimensional virtual plane of the target object is determined. This can be understood as, in response to the user's click operation on the shopping cart image in the shopping virtual scene, the three-dimensional virtual plane corresponding to the shopping cart image is determined.

[0271] Based on the user's second interactive operation on the target virtual object in the target virtual scene, and based on the attribute information of the target virtual object, the target placement plane of the target virtual object is determined from the three-dimensional virtual plane. This can be understood as, based on the user's click or drag operation on a virtual product in the shopping virtual scene, and based on the attribute information of the virtual product, the target placement plane of the virtual product is determined from the three-dimensional virtual plane. For example, if the virtual product is a large piece of fruit, then the target placement plane can be the bottom layer of the shopping cart.

[0272] Determining the target placement point of the virtual object on the target placement plane based on the trajectory points of the target virtual object and the set of planar points of the target placement plane can be understood as determining the target placement point of the virtual product on the target placement plane based on the trajectory points of the virtual product and the set of planar points of the target placement plane.

[0273] Based on the specific implementation steps described above and the method for determining the object placement point, in a virtual decoration scene, the target placement points of various virtual furniture can be determined on the three-dimensional virtual plane of the floor; in a virtual scene for designing three-dimensional objects (such as a virtual keyboard design scene), the target placement points of each keyboard key can be determined on the three-dimensional virtual plane of the keyboard base; and in a virtual exhibition scene, the target placement points of exhibits to be exhibited can be determined on the three-dimensional virtual plane of the exhibition counter, etc.

[0274] Furthermore, the three-dimensional virtual plane in this method can be understood as being pre-generated using the RGB image of the target object according to the method described in the above embodiments before the construction of the target virtual scene; and the target placement plane of the target virtual object is determined from the three-dimensional virtual plane based on the attribute information of the target virtual object; the specific implementation steps of determining the target placement point of the target virtual object on the target placement plane based on the trajectory points of the target virtual object and the set of plane points of the target placement plane can also be found in the detailed description of the above embodiments, and will not be repeated here.

[0275] Another method for determining object placement points provided in the embodiments of this specification can determine the three-dimensional virtual plane of the target object and the target virtual object to be combined with the three-dimensional virtual plane in the target virtual scene through user interaction, and accurately determine the target placement point of the target virtual object on the target placement plane of the three-dimensional virtual plane based on the trajectory points of the target virtual object and the set of plane points of the three-dimensional virtual plane.

[0276] See Figure 8 , Figure 8 A structural block diagram of a computing device 800 according to one embodiment of this specification is shown. The components of the computing device 800 include, but are not limited to, a memory 810 and a processor 820. The processor 820 is connected to the memory 810 via a bus 830, and a database 850 is used to store data.

[0277] The computing device 800 also includes an access device 840, which enables the computing device 800 to communicate via one or more networks 860. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 840 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0278] In one embodiment of this specification, the above-described components of the computing device 800 and Figure 8 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 8 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0279] The computing device 800 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 800 can also be a mobile or stationary server.

[0280] The processor 820 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-described object placement point determination method.

[0281] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the object placement point determination method described above belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the object placement point determination method described above.

[0282] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the above-described object placement point determination method.

[0283] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the object placement point determination method described above belong to the same concept. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the object placement point determination method described above.

[0284] An embodiment of this specification also provides a computer program, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the above-described object placement point determination method.

[0285] The above is an illustrative scheme of a computer program according to this embodiment. It should be noted that the technical solution of this computer program and the technical solution of the object placement point determination method described above belong to the same concept. For details not described in detail in the technical solution of the computer program, please refer to the description of the technical solution of the object placement point determination method described above.

[0286] According to a third aspect of the embodiments of this specification, an augmented reality (AR) device is provided, comprising:

[0287] Memory and processor;

[0288] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, they implement the steps of the above-described method for determining the object placement point.

[0289] The above is an illustrative scheme of an augmented reality (AR) device according to this embodiment. It should be noted that the technical solution of this AR device and the technical solution of the object placement point determination method described above belong to the same concept. For details not described in detail in the technical solution of the AR device, please refer to the description of the technical solution of the object placement point determination method described above.

[0290] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0291] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added to or subtracted according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0292] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0293] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0294] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A method for determining the placement point of an object, comprising: Based on the segmentation mask and depth image of the target image, an initial 3D point cloud corresponding to the segmentation mask is generated; A three-dimensional virtual plane is generated based on the initial three-dimensional point cloud and the pixels in the segmentation mask; Based on the attribute information of the target virtual object, the target placement plane of the target virtual object is determined from the three-dimensional virtual plane, wherein the attribute information of the target virtual object includes the category or volume of the target virtual object; Based on the trajectory points of the target virtual object and the set of planar points of the target placement plane, the target placement point of the target virtual object on the target placement plane is determined. This determination includes: adjusting the set of planar points of the target placement plane according to the world coordinate system to determine the target planar point set; when at least two trajectory points of the target virtual object are determined, determining candidate placement points of the target virtual object on the target placement plane based on the target distance between the trajectory points of the target virtual object and the planar points in the target planar point set, and a second preset distance threshold; clustering the candidate placement points; and determining the target placement point of the target virtual object on the target placement plane based on the clustering results.

2. The object placement point determination method according to claim 1, before generating the initial 3D point cloud corresponding to the segmentation mask based on the segmentation mask and depth image of the target image, further comprising: Identify the target image; The target image is input into an image segmentation model to obtain a segmentation mask for the target image and the category of the pixels in the segmentation mask; The target image is input into the depth estimation model to obtain the depth image of the target image.

3. The method for determining object placement points according to claim 1, wherein generating an initial 3D point cloud corresponding to the segmentation mask based on the segmentation mask and depth image of the target image includes: Determine the depth value of the pixel in the segmentation mask based on the depth image; An initial 3D point cloud corresponding to the segmentation mask is generated based on the coordinate values ​​of the pixels in the segmentation mask, the depth value, and the camera intrinsic parameters corresponding to the target image.

4. The method for determining object placement points according to claim 1, wherein generating a three-dimensional virtual plane based on the initial three-dimensional point cloud and the pixels in the segmentation mask includes: The initial 3D point cloud is fitted to a plane according to a preset plane fitting algorithm to obtain a fitting plane. Determine the distance from the pixel in the segmentation mask to the fitting plane, and project the pixel in the fitting plane that is less than a first preset distance threshold to form a set of planar points on the fitting plane; A three-dimensional virtual plane is generated based on the set of plane points of the fitted plane.

5. The method for determining the object placement point according to claim 2, wherein determining the target placement plane of the target virtual object from the three-dimensional virtual plane based on the attribute information of the target virtual object includes: The category of the three-dimensional virtual plane is determined based on the category of the pixels in the segmentation mask; Based on the matching relationship between the category of the target virtual object and the category of the three-dimensional virtual plane, the target placement plane of the target virtual object is determined from the three-dimensional virtual plane.

6. The method for determining the object placement point according to claim 4, further comprising, after determining the target placement plane of the target virtual object from the three-dimensional virtual plane based on the attribute information of the target virtual object: The target virtual object is projected onto the target placement plane, and the projection shape of the target virtual object on the target placement plane is determined; The target radius of the projection surface of the virtual object on the target placement plane is determined based on the distance from the outline point of the projection shape to the centroid point of the projection shape. The target segmentation mask is obtained by filtering the segmentation mask corresponding to the target placement plane based on the target radius.

7. The method for determining object placement points according to claim 6, further comprising, after obtaining the target segmentation mask: Determine the distance from the pixel in the target segmentation mask to the fitting plane, project the pixels whose distance is less than the first preset distance threshold onto the fitting plane, and update the plane point set of the fitting plane; Based on the correspondence between the updated fitted plane point set and the three-dimensional virtual plane, and the correspondence between the three-dimensional virtual plane and the target placement plane, the plane point set of the target placement plane is determined.

8. The method for determining object placement points according to claim 1, wherein determining candidate placement points of the target virtual object on the target placement plane based on the target distance between the trajectory points of the target virtual object and the target plane point set plane points and a second preset distance threshold includes: S2. Sequentially take each plane point in the target plane point set as the starting point, move the trajectory point of the target virtual object to the target plane point set according to the starting point, and calculate the target distance from the trajectory point of the target virtual object to the plane point in the target plane point set; S4. If it is determined that the target distance is less than or equal to the second preset distance threshold, the starting point is selected as a candidate placement point; or If it is determined that the target distances are all greater than the second preset distance threshold, the trajectory points of the target virtual object are rotated according to a preset angle, and step S2 is continued.

9. The method for determining the placement point of an object according to claim 8, the method further comprising: If the trajectory point of the target virtual object is rotated according to the preset angle and the preset conditions are met, but there is still no candidate placement point, the trajectory point of the target virtual object is rotated according to the preset angle, and the minimum target distance from the trajectory point of the target virtual object to the target plane point set plane point is calculated. The second preset distance threshold is adjusted based on the minimum target distance to obtain the third preset distance threshold; Based on the third preset distance threshold and the preset conditions for rotating the trajectory point of the target virtual object according to the preset angle, the target distances from the trajectory point of the target virtual object to the target plane point set plane points are calculated, and candidate placement points are determined.

10. The method for determining the object placement point according to claim 1, wherein determining the target placement point of the target virtual object on the target placement plane based on the trajectory points of the target virtual object and the set of planar points of the target placement plane comprises: The target plane point set is determined by adjusting the plane point set of the target placement plane according to the world coordinate system; If the trajectory point of the target virtual object is determined to be one, all plane points in the target plane point set are determined as candidate placement points of the target virtual object on the target placement plane. The candidate placement points are clustered, and the target placement point of the target virtual object on the target placement plane is determined based on the clustering results.

11. The method for determining the object placement point according to claim 1 or 10, wherein clustering the candidate placement points and determining the target placement point of the target virtual object on the target placement plane based on the clustering results comprises: The candidate placement points are clustered according to a preset clustering algorithm to obtain a preset number of cluster center points; Project each cluster center point onto the target image, and calculate the distance between each cluster center point and the center point of the target image; The target placement point of the virtual object on the target placement plane is determined based on the distance between each cluster center point and the center point of the target image.

12. A method for determining the placement point of an object, comprising: Define the target virtual scene; In response to a user's first interactive operation on a target object in the target virtual scene, the three-dimensional virtual plane of the target object is determined; Based on the user's second interactive operation on the target virtual object in the target virtual scene, the target placement plane of the target virtual object is determined from the three-dimensional virtual plane, wherein the determination of the target placement plane is based on the attribute information of the target virtual object, and the attribute information of the target virtual object includes the category or volume of the target virtual object; Based on the set of planar points of the target placement plane, the target placement point of the target virtual object on the target placement plane is determined. This determination includes: adjusting the set of planar points of the target placement plane according to the world coordinate system to determine the target planar point set; when at least two trajectory points of the target virtual object are determined, candidate placement points of the target virtual object on the target placement plane are determined based on the target distance between the trajectory points of the target virtual object and the planar points in the target planar point set, and a second preset distance threshold; the candidate placement points are clustered; and the target placement point of the target virtual object on the target placement plane is determined based on the clustering results.

13. The method for determining the object placement point according to claim 12, wherein determining the target placement plane of the target virtual object from the three-dimensional virtual plane based on the user's second interactive operation on the target virtual object in the target virtual scene comprises: Based on the user's second interactive operation on the target virtual object in the target virtual scene, and based on the attribute information of the target virtual object, the target placement plane of the target virtual object is determined from the three-dimensional virtual plane; Accordingly, determining the target placement point of the target virtual object on the target placement plane based on the set of planar points of the target placement plane includes: Based on the trajectory points of the target virtual object and the set of planar points on the target placement plane, the target placement point of the target virtual object on the target placement plane is determined.

14. An augmented reality (AR) device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the object placement point determination method according to any one of claims 1 to 13.

Images