Three-dimensional modeling method, device, storage medium and equipment based on augmented reality

Abstract

The application discloses a three-dimensional modeling method and device based on augmented reality, a storage medium and equipment. The three-dimensional modeling method comprises the following steps: generating a real scene point cloud according to a real scene image collected by an augmented reality device, wherein the real scene image contains an image of a target object; constructing a target object rough model according to obtained user hand-drawing action information and the real scene point cloud; segmenting a target object point cloud from the real scene point cloud according to the target object rough model; and adjusting the target object rough model by using the target object point cloud to obtain a target object fine model. The method fully utilizes the scale and detail information of the target object contained in the scene image, and a high-precision model can be obtained under the condition that the user only draws simple strokes, so that the learning cost is reduced and the use comfort is improved.

Description

Technical Field

[0001] This invention belongs to the field of computer graphics technology, specifically, it relates to an augmented reality-based 3D modeling method, a 3D modeling device, a computer-readable storage medium, and a computer device. Background Technology

[0002] 3D modeling is an important research topic in the fields of computer graphics and human-computer interaction. 3D models are widely used in games, manufacturing, and other fields. Traditionally, models are created manually on a computer screen using modeling software such as 3ds Max, which is highly demanding and time-consuming for technicians. With the development of VR (Virtual Reality) and AR (Augmented Reality) hardware and software, users can now work directly in a 3D environment. Products like Google Tilt Brush allow for immersive and free creation in a 3D environment, making the modeling process simpler and more intuitive. AR technology is a technique that overlays and merges virtual information with the real world. It widely utilizes multimodal data, 3D modeling, intelligent interaction, and sensing technologies to simulate and overlay computer-generated images and 3D models onto a real scene. The two types of information complement each other, enhancing the realism of the scene. Displaying virtual objects overlaid on real scenes in an AR environment greatly improves users' 3D spatial perception and facilitates 3D interactive operations. Furthermore, AR provides online data acquisition capabilities, seamlessly integrating data acquisition and modeling processes, eliminating the need for offline data collection before modeling begins.

[0003] The advantage of sketch-based modeling techniques lies in the ability for users to create models of objects, whether real-world or not, using simple sketches, achieving a simple and efficient modeling experience. Model creation based on 2D sketches typically employs interaction in a 2D screen space, using devices like mice or tablets to create 2D strokes, and then inferring 3D information based on prior assumptions to obtain a 3D model. While this interaction method is relatively accurate, the 3D information inferred from prior assumptions is inaccurate due to the lack of depth information in 2D sketches. Furthermore, this interaction is not intuitive and often leads to ambiguity. The advantage of using 3D sketches for model creation is the ability to draw 3D strokes, with an intuitive and concrete interaction method. Existing 3D sketch-based modeling methods require drawing complex and precise 3D strokes. However, drawing with the hand in mid-air often results in inaccurate strokes due to shakiness, making the creation of precise strokes a challenging task. Most current work proposes using auxiliary devices such as tablets and digital pens to improve the accuracy of interaction.

[0004] The problem with existing technologies is that they require detailed 3D strokes to create good model results. Introducing additional hardware to improve interactive accuracy requires a learning curve and reduces user comfort. Current methods do not utilize scene data, which is crucial for modeling; they fail to consider that scene data may also contain information about the approximate shape and scale of objects. Summary of the Invention

[0005] (I) The technical problem to be solved by the present invention

[0006] The technical problem solved by this invention is: how to effectively utilize scene point cloud data to improve the accuracy of 3D models.

[0007] (II) Technical Solution Adopted in this Invention

[0008] An augmented reality-based 3D modeling method, the 3D modeling method comprising:

[0009] A real-scene point cloud is generated based on real-scene images captured by augmented reality devices, wherein the real-scene images include images of the target objects.

[0010] A rough model of the target object is constructed based on the user's hand-drawn action information and the real scene point cloud.

[0011] The target object point cloud is segmented from the real scene point cloud based on the rough model of the target object;

[0012] The coarse model of the target object is adjusted using the point cloud of the target object to obtain a fine model of the target object.

[0013] Preferably, the user's hand-drawn action information includes several contour stroke points and several trajectory stroke points generated in real time based on the recognized user drawing action. The method for generating a rough model of the target object based on the acquired user's hand-drawn action information and the real scene point cloud includes:

[0014] When generating contour stroke points, contour matching points are retrieved from the real scene point cloud, and each contour stroke point is snapped to the contour matching point to form a contour stroke.

[0015] When generating trajectory stroke points, trajectory matching points are retrieved from the real scene point cloud, and each trajectory stroke point is attached to the trajectory matching point to form a trajectory stroke.

[0016] A rough model of the target object is generated based on the outline strokes and the trajectory strokes.

[0017] Preferably, the method for retrieving contour matching points from the real scene point cloud when generating contour stroke points includes:

[0018] Candidate point sets are selected from the real scene point cloud, and the distance between each point in the candidate point set and the contour stroke point is less than a predetermined value.

[0019] Determine whether there are edge points in the candidate point set;

[0020] If it exists, the edge point with the smallest distance from the outline stroke point will be used as the outline matching point;

[0021] If no such point exists, the point in the candidate point set that is closest to the contour stroke point is taken as the contour matching point.

[0022] Preferably, the method for segmenting the target object point cloud from the real scene point cloud based on the target object coarse model includes:

[0023] Candidate spatial regions are determined based on the rough model of the target object, and the target object is located within the candidate spatial regions.

[0024] The point clouds in the candidate spatial region are clustered to form several point cloud clusters;

[0025] The portion of the point cloud clusters whose point cloud quantity exceeds a predetermined quantity are selected as candidate point cloud clusters;

[0026] Calculate the spatial distance between the trajectory stroke and each of the candidate point cloud clusters, wherein the spatial distance is the sum of the nearest distances between each stroke point of the trajectory stroke and the candidate point cloud cluster;

[0027] The candidate point cloud cluster with the smallest spatial distance is selected as the target point cloud.

[0028] Preferably, the method for optimizing the coarse model of the target object using the point cloud of the target object to obtain a fine model of the target object includes:

[0029] The model type of the rough model of the target object is predicted;

[0030] The initial parameters are obtained by parameterizing the rough model of the target object according to the model type.

[0031] Construct a distance objective function based on the initial parameters and the point cloud of the target object;

[0032] The distance objective function is minimized using the least squares method to obtain the optimized parameters;

[0033] A refined model of the target object is constructed based on the optimized parameters.

[0034] Preferably, the method for obtaining the prediction model type of the rough model of the target object includes:

[0035] The formed contour strokes and trajectory strokes are input into a pre-trained prediction model to predict the contour stroke type and trajectory stroke type.

[0036] The model type of the rough model of the target object is predicted based on the outline stroke type and trajectory stroke type.

[0037] Preferably, the three-dimensional modeling method further includes:

[0038] The shape of the outline stroke is optimized based on the predicted outline stroke type.

[0039] This application also discloses an augmented reality-based 3D modeling device, the 3D modeling device comprising:

[0040] The preprocessing unit is used to generate a real scene point cloud based on real scene images acquired by the augmented reality device, wherein the real scene images include images of target objects;

[0041] The model building unit is used to build a rough model of the target object based on the acquired user hand-drawn action information and the real scene point cloud;

[0042] A point cloud segmentation unit is used to segment the target object point cloud from the real scene point cloud based on the target object coarse model;

[0043] The model optimization unit is used to adjust the coarse model of the target object using the point cloud of the target object to obtain a fine model of the target object.

[0044] This application also discloses a computer-readable storage medium storing an augmented reality-based 3D modeling program, which, when executed by a processor, implements the above-described augmented reality-based 3D modeling method.

[0045] This application also discloses a computer device, which includes a computer-readable storage medium, a processor, and an augmented reality-based 3D modeling program stored in the computer-readable storage medium. When the augmented reality-based 3D modeling program is executed by the processor, it implements the above-described augmented reality-based 3D modeling method.

[0046] (III) Beneficial Effects

[0047] This invention discloses a 3D modeling method based on augmented reality, which has the following technical advantages compared to traditional modeling methods:

[0048] This method makes full use of the scale and detail information of the target object contained in the scene image, so that even if the user only draws simple strokes, a high-precision model can be obtained, which reduces the learning cost and improves the user's comfort. Attached Figure Description

[0049] Figure 1 This is a flowchart of an augmented reality-based 3D modeling method according to Embodiment 1 of the present invention;

[0050] Figure 2 This is a schematic diagram illustrating the process of an augmented reality-based 3D modeling method according to Embodiment 1 of the present invention.

[0051] Figure 3 This is a schematic diagram illustrating the generation of a surface scanning model according to Embodiment 1 of the present invention;

[0052] Figure 4 This is a schematic diagram of the contour stroke optimization according to Embodiment 1 of the present invention;

[0053] Figure 5 This is a schematic diagram of the prediction process for the model type in Embodiment 1 of the present invention;

[0054] Figure 6 This is a schematic diagram illustrating the modeling process of different single-component objects according to Embodiment 1 of the present invention;

[0055] Figure 7 This is a schematic diagram illustrating the modeling process of a multi-part object according to Embodiment 1 of the present invention;

[0056] Figure 8 This is a schematic diagram of the augmented reality-based 3D modeling device according to Embodiment 2 of the present invention;

[0057] Figure 9 This is a schematic diagram of a computer device according to Embodiment 4 of the present invention. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0059] Before describing the various embodiments of this application in detail, let me first briefly describe the inventive concept: In the existing process of building 3D sketch models, in order to obtain accurate interactive strokes, it is often necessary to use auxiliary devices such as tablets and digital pens to improve the accuracy of interaction, but this inevitably leads to a higher learning cost and a poorer user experience. Therefore, the augmented reality-based 3D modeling method provided in this application first converts the scene image containing the target object into a real scene point cloud, then constructs a rough model of the target object based on the user's hand-drawn actions, further segments the target object point cloud from the real scene point cloud, and finally optimizes and adjusts the rough model of the target object using the target object point cloud to obtain a refined model of the target object. This application makes full use of the scale and detail information of the target object contained in the scene image, enabling users to obtain a high-precision model even with only simple hand-drawn strokes, reducing the learning cost and improving user experience.

[0060] like Figure 1 As shown, the augmented reality-based 3D modeling method in this embodiment includes the following steps:

[0061] Step S10: Generate a real scene point cloud based on the real scene images acquired by the augmented reality device, wherein the real scene images include images of the target objects;

[0062] Step S20: Construct a rough model of the target object based on the obtained user hand-drawn action information and real scene point cloud;

[0063] Step S30: Segment the target object point cloud from the real scene point cloud based on the rough model of the target object;

[0064] Step S40: Optimize the coarse model of the target object using the point cloud of the target object to obtain the fine model of the target object.

[0065] In step S10, this embodiment uses the HoloLens 2 augmented reality device to acquire depth and color images of a real scene, i.e., real scene images. Point cloud data reconstructed from the depth images is displayed in real-time on the head-mounted display. Whenever a user's acquisition command is received, such as the user's thumb and forefinger quickly overlapping twice, a frame of color point cloud is reconstructed from the current depth and color images. Then, a preset registration method is executed to merge the current frame point cloud with historical point clouds to reduce registration errors caused by pose estimation in the HoloLens 2 augmented reality device. Finally, a density-based denoising algorithm is applied to the acquired point cloud data to obtain a cleaner real scene point cloud. Figure 2 As shown in (a).

[0066] Furthermore, to further improve the accuracy and comfort of the constructed model, in this embodiment, the Random Sampling Consensus (RANSAC) algorithm is used to perform planar feature detection on the real scene point cloud to extract the plane; the Canny algorithm is used to extract the edge of the depth map of the real scene; and the point is labeled with the type of each point in the real scene point cloud according to the extraction results, that is, each point is an edge point or a non-edge point, and the labeling type of the point remains unchanged during the point cloud registration process.

[0067] Furthermore, in step S20, a three-dimensional model is constructed using a scanning-based modeling method. Creating a three-dimensional scanning model requires drawing a contour stroke and a trajectory stroke. The contour stroke lies on a plane in space and sweeps along the trajectory stroke to construct the three-dimensional scanning model. Each stroke consists of a series of three-dimensional vertices. In other words, the user's hand-drawn action information in this embodiment includes several contour stroke points and several trajectory stroke points generated in real time based on the recognized user drawing action.

[0068] Specifically, step S20 includes: when generating contour stroke points, retrieving contour matching points from the real scene point cloud, and attaching each contour stroke point to the contour matching point to form a contour stroke; when generating trajectory stroke points, retrieving contour matching points from the real scene point cloud, and attaching each trajectory stroke point to the trajectory matching point to form a trajectory stroke; generating a rough model of the target object based on the contour strokes and the trajectory strokes, wherein the rough model of the target object is as follows: Figure 2 As shown in (b).

[0069] The method for retrieving contour matching points from the real-world point cloud when generating contour stroke points is as follows: Upon recognizing the user's drawing action and converting it into contour stroke points, the real-world point cloud is searched, and a candidate point set is selected. The distance between each point in the candidate point set and the contour stroke point is less than a predetermined value. Further, it is determined whether there are edge points in the candidate point set. If edge points exist, the edge point with the smallest distance to the contour stroke point is selected as the contour matching point. It should be noted that if only one edge point exists, that edge point is directly selected as the contour matching point; if no edge point exists, the point with the smallest distance to the contour stroke point in the candidate point set is selected as the contour matching point. For trajectory stroke points, the same method is used to retrieve trajectory matching points and perform snapping. The specific process can be found in the description above and will not be repeated here.

[0070] Once the outline strokes are drawn, the least squares algorithm is used to fit a plane P containing the outline strokes as the supporting plane, and the stroke points are projected onto plane P. If plane P is similar to plane F extracted from the scene point cloud, then F is selected as the supporting plane. Since the outline strokes are located on a plane in space, finding a supporting plane can improve the interactive experience and reduce the difficulty of operation for users.

[0071] Furthermore, to improve interactivity, after forming the outline strokes, each outline stroke is input into a pre-trained prediction model to predict the stroke type. Based on the predicted stroke type, the outline stroke shape is optimized to obtain the stroke result expected by the user, and the result is fed back to the user in real time. The prediction model uses an LSTM network, and the predicted and optimized stroke types include straight lines, circles, rectangles, and free curves. For straight lines, circles, and rectangles, the least squares algorithm is used to optimize the strokes. For free curves, a cubic b-spline smoothing algorithm is used to reduce the impact of jitter. Specific optimization results can be found in [reference needed]. Figure 4 As shown.

[0072] like Figure 3 As shown, after obtaining the outline strokes and trajectory strokes, the outline strokes are controlled to sweep along the trajectory strokes to form a rough model of the target object. It should be noted that the outline strokes can be swept after the complete trajectory strokes are formed, or the outline strokes can be swept at some sections of the trajectory strokes or at each trajectory stroke point.

[0073] Furthermore, to optimize the user-created coarse model of the target object, it is necessary to segment the point cloud associated with the coarse model of the target object from the real scene point cloud. To solve this task, this embodiment utilizes the segmentation semantics potentially contained in the coarse model of the target object to assist the point cloud segmentation process, resulting in the target object point cloud, as shown below. Figure 2 As shown in (c).

[0074] Specifically, step S30, which involves segmenting the target object point cloud from the real scene point cloud based on the rough model of the target object, includes the following steps:

[0075] Step S31: Determine candidate spatial regions based on the rough model of the target object, whereby the target object lies within the candidate spatial regions. The user-created rough model of the target object has determined the approximate spatial range of the target object, and this approximate spatial range Γ represents the candidate spatial region. Preferably, the candidate spatial region is selected as 1.5 times the axial bounding box of the rough model of the target object.

[0076] Step S32: Cluster the point cloud data in the candidate spatial region to form several point cloud clusters. Specifically, a clustering algorithm that considers Euclidean distance and surface point continuity is used to segment the point cloud data in the candidate spatial region. For cylindrical and cuboid objects, a constraint is added that the point normal is perpendicular to the principal axis to obtain better results. The clustering results in several point cloud clusters.

[0077] Step S33: Select the point cloud clusters with a number of points greater than a predetermined number as candidate point cloud clusters. The predetermined number needs to be set according to the actual situation. For example, here we can select the three point cloud clusters with the highest number of points as candidate point cloud clusters.

[0078] Step S34: Calculate the spatial distance between the trajectory stroke and each of the candidate point cloud clusters, wherein the spatial distance is the sum of the nearest distances between each stroke point of the trajectory stroke and the candidate point cloud cluster.

[0079] Step S35: Select the candidate point cloud cluster with the smallest spatial distance as the target point cloud.

[0080] Specifically, since trajectory strokes are always anchored to the surface of the object's point cloud, the candidate point cloud cluster with the smallest spatial distance to the trajectory stroke is selected as the segmentation result. The spatial distance from the trajectory stroke to the candidate point cloud cluster is calculated as the sum of the nearest distances from each stroke point in the trajectory stroke to the point cloud cluster. The method for calculating the nearest distance from each stroke point to the candidate point cloud cluster is as follows: calculate the distance between the stroke point and each point in the candidate point cloud cluster, and take the minimum distance as the nearest distance from each stroke point to the candidate point cloud cluster.

[0081] Existing technologies typically employ point cloud density-based clustering and region growing algorithms to segment the entire scene. These methods usually only achieve coarse segmentation into small parts, often yielding inaccurate results and requiring manual user specification of the segmentation outcome. The segmentation method in step S30 of this embodiment is specifically designed for scanned model objects, essentially utilizing the user's semantic information to assist the process. By constraining the target object's category with semantic information, the target point cloud can be segmented more accurately. Simultaneously, the initial stroke position information improves segmentation speed and target object selection, eliminating the need for manual user specification of the target point cloud object.

[0082] Furthermore, in step S40, the method for optimizing the coarse model of the target object using the point cloud of the target object to obtain the fine model of the target object includes the following steps:

[0083] Step S41: Predict the model type of the rough model of the target object.

[0084] Specifically, the formed contour strokes and trajectory strokes are input into a pre-trained prediction model to predict the contour stroke type and trajectory stroke type, respectively. Based on the contour stroke type and trajectory stroke type, the model type of the coarse model of the target object is predicted. The prediction model uses an LSTM model, such as... Figure 5 As shown, different model types are predicted based on different contour stroke types and trajectory stroke types.

[0085] Step S42: Perform parameterization on the rough model of the target object according to the model type to obtain the initial parameters.

[0086] At this point, the rough model of the target object is composed of point cloud data. Based on the model type of the rough model of the target object, parameters that can characterize this model type are determined, and then the rough model of the target object is transformed into these parameters. For example, a cylindrical surface can be parameterized as an axis direction n (unit vector), a point q on the axis, and a radius r. That is, the cylindrical surface can be characterized using three initial parameters n, q, and r.

[0087] Step S43: Construct a distance objective function based on the initial parameters and the point cloud of the target object.

[0088] Step S44: Minimize the distance objective function using the least squares method to obtain the optimized parameters.

[0089] Step S45: Construct a detailed model of the target object based on the optimization parameters.

[0090] The constructed distance objective function represents the distance from the target object point cloud to the rough model surface of the target object. The distance objective function is minimized by adjusting the values ​​of the initial parameters. This indicates that the target object point cloud is closest to the rough model surface of the target object. The adjusted initial parameters are the optimization parameters. Finally, based on the fine model of the target object, such as the optimized n, q, and r, the optimized cylindrical surface can be constructed.

[0091] The following describes the optimization process for coarse models of different types of target objects.

[0092] The target object point cloud segmented from the real scene point cloud in the previous step is P = {p1, p2, ..., p...} N}

[0093] (1) Cylindrical Surface. A cylindrical surface can be parameterized as an axis with direction n (unit vector), a point q on the axis, and radius r. Combining the point cloud of the target object, the distance objective function is constructed as follows:

[0094]

[0095] Where, pi q represents from p i A vector pointing to q. This function is constructed based on the premise that the distance from each point on the cylindrical surface to the axis is r.

[0096] (2) Cone. The cone can be parameterized as vertex a, axis direction n (unit vector), and half-angle θ of the cone. For the cone, the distance objective function is constructed as follows, where the function represents the distance from the point cloud to the cone surface:

[0097]

[0098] (3) Cuboid: A cuboid is always parameterized as orthogonal vectors W, H, L, corresponding scalars w, h, l, and center point c. From these parameters, the planar set T = {t1, t2, t3, t4, t5, t6} of the cuboid can be reconstructed. The objective function for fitting the cuboid is as follows, where the function represents the distance from the point cloud to the surface of the cuboid:

[0099]

[0100] Δ γ =(WH) 2 +(HL) 2 +(WL)2,

[0101] Where distance(p) i ,t k ) represents point p i to plane t k The distance term Δγ is used to ensure that vectors W, H, and L are mutually orthogonal.

[0102] (4) Generalized Cylinder: We represent the generalized cylinder as a whole composed of multiple circular slices, with the centers of all the circles on the axis. Therefore, the optimization objective can be simplified to fitting multiple coaxial three-dimensional circular surfaces, with the distance objective function being...

[0103]

[0104]

[0105] Where K is the number of circular slices, c i r i α and γ are the parameters of the i-th circular slice, respectively. α controls the smoothness of the radius transformation of the circular surface, and γ is used to ensure that the centers of the circular slices are all on the same axis. For the optimization of the i-th circle, the point set S on the circle is first extracted from the point cloud P of the target object. The judgment criterion is that the distance from the point to the plane where circle i is located is less than a certain threshold, which is 0.005 in this embodiment.

[0106]

[0107] Therefore, by minimizing the above-mentioned distance objective functions, the corresponding optimization parameters can be obtained. Finally, the optimized model, i.e., the refined model of the target object, can be constructed using the optimization parameters.

[0108] The augmented reality-based 3D modeling method disclosed in Embodiment 1 does not require users to hand-draw detailed strokes. It combines simple drawing with scene data captured from the real world and adopts a 3D interactive approach, making the model creation process simpler. At the same time, by leveraging augmented reality technology, it overcomes the drawback of 2D sketches drawn on a 2D screen being difficult to define.

[0109] For example, Figure 6 This is a schematic diagram illustrating the process of creating 3D models of different single-component objects. Figure 7 This is a schematic diagram illustrating the process of creating a 3D model of a multi-part object.

[0110] Furthermore, such as Figure 8 As shown in Embodiment 2, an augmented reality-based 3D modeling device is also disclosed. The 3D modeling device includes a preprocessing unit 100, a model building unit 200, a point cloud segmentation unit 300, and a model optimization unit 400. The preprocessing unit 100 generates a real-scene point cloud based on real-scene images acquired by the augmented reality device, wherein the real-scene images include images of the target object. The model building unit 200 constructs a rough model of the target object based on acquired user hand-drawn action information and the real-scene point cloud. The point cloud segmentation unit 300 segments the target object point cloud from the real-scene point cloud based on the rough target object model. The model optimization unit 400 adjusts the rough target object model using the target object point cloud to obtain a refined model of the target object. The specific working processes of the preprocessing unit 100, model building unit 200, point cloud segmentation unit 300, and model optimization unit 400 can be referred to the relevant description in Embodiment 1, and will not be repeated here.

[0111] This third embodiment also discloses a computer-readable storage medium storing an augmented reality-based 3D modeling program, which, when executed by a processor, implements the above-described augmented reality-based 3D modeling method.

[0112] Embodiment 4 of this application also discloses a computer device, at the hardware level, such as Figure 9As shown, the terminal includes a processor 12, an internal bus 13, a network interface 14, and a computer-readable storage medium 11. The processor 12 reads the corresponding computer program from the computer-readable storage medium and runs it, forming a request processing device at the logical level. Of course, besides software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices. The computer-readable storage medium 11 stores an augmented reality-based 3D modeling program, which, when executed by the processor, implements the aforementioned augmented reality-based 3D modeling method.

[0113] Computer-readable storage media include both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer-readable storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0114] The specific embodiments of the present invention have been described in detail above. Although some embodiments have been shown and described, those skilled in the art should understand that modifications and improvements can be made to these embodiments without departing from the principles and spirit of the present invention as defined by the claims and their equivalents, and such modifications and improvements should also be within the protection scope of the present invention.

Claims

1. A method of three-dimensional modeling based on augmented reality, characterized in that, The three-dimensional modeling method comprises: generating a real scene point cloud according to a real scene image collected by an augmented reality device, wherein the real scene image contains an image of a target object; constructing a target object rough model according to obtained user hand-drawing action information and the real scene point cloud; segmenting a target object point cloud from the real scene point cloud according to the target object rough model; adjusting the target object rough model by using the target object point cloud to obtain a target object fine model; the user hand-drawing action information comprises a plurality of contour stroke points and a plurality of trajectory stroke points generated in real time according to recognized user drawing actions, and the method for generating a target object rough model according to obtained user hand-drawing action information and the real scene point cloud comprises: when generating a contour stroke point, retrieving a contour matching point from the real scene point cloud, and adsorbing each contour stroke point to the contour matching point to form a contour stroke; when generating a trajectory stroke point, retrieving a trajectory matching point from the real scene point cloud, and adsorbing each trajectory stroke point to the trajectory matching point to form a trajectory stroke; generating a target object rough model according to the contour stroke and the trajectory stroke.

2. The augmented reality-based three-dimensional modeling method of claim 1, wherein, When generating a contour stroke point, the method for retrieving a contour matching point from the real scene point cloud comprises: screening a candidate point set from the real scene point cloud, wherein each point in the candidate point set is less than a predetermined value from the contour stroke point; judging whether there is an edge point in the candidate point set; if there is, taking the edge point with the smallest distance from the contour stroke point as the contour matching point; if there is not, taking the point with the smallest distance from the contour stroke point in the candidate point set as the contour matching point.

3. The augmented reality-based three-dimensional modeling method of claim 1, wherein, The method for segmenting a target object point cloud from the real scene point cloud according to the target object rough model comprises: determining a candidate space region according to the target object rough model, wherein the target object is in the candidate space region; performing clustering processing on the point cloud in the candidate space region to form a plurality of point cloud clusters; taking a part of the point cloud clusters with a point cloud number greater than a predetermined number as candidate point cloud clusters; calculating a spatial distance between a trajectory stroke and each candidate point cloud cluster, wherein the spatial distance is the sum of the nearest distances between each stroke point of the trajectory stroke and the candidate point cloud cluster; taking the candidate point cloud cluster with the smallest spatial distance as the target object point cloud.

4. The augmented reality-based three-dimensional modeling method of claim 3, wherein, The method for optimizing the target object rough model by using the target object point cloud to obtain a target object fine model comprises: predicting a model type of the target object rough model; performing parameterization processing on the target object rough model according to the model type to obtain initial parameters; constructing a distance objective function according to the initial parameters and the target object point cloud; performing minimum processing on the distance objective function by using a least square method to obtain optimized parameters; constructing a target object fine model according to the optimized parameters.

5. The augmented reality-based three-dimensional modeling method of claim 4, wherein, The method for obtaining the predicted model type of the target object rough model comprises: The generated contour strokes and trajectory strokes are respectively input into a pre-trained prediction model to predict contour stroke types and trajectory stroke types; A model type of a target object rough model is predicted according to the contour stroke types and the trajectory stroke types.

6. The augmented reality-based three-dimensional modeling method of claim 5, wherein, The three-dimensional modeling method further includes: The shape of the contour strokes is optimized according to the predicted contour stroke types.

7. An augmented reality-based three-dimensional modeling apparatus, characterized by comprising: The three-dimensional modeling device includes: A preprocessing unit is configured to generate a real scene point cloud based on a real scene image captured by an augmented reality device, wherein the real scene image contains an image of a target object; A model construction unit is configured to construct a target object rough model based on acquired user hand-drawing action information and the real scene point cloud; A point cloud segmentation unit is configured to segment a target object point cloud from the real scene point cloud based on the target object rough model; A model optimization unit is configured to adjust the target object rough model by using the target object point cloud to obtain a target object fine model; The user hand-drawing action information includes a plurality of contour stroke points and a plurality of trajectory stroke points generated in real time based on recognized user drawing actions, and the method of generating a target object rough model based on acquired user hand-drawing action information and the real scene point cloud includes: When generating contour stroke points, contour matching points are searched from the real scene point cloud, and each contour stroke point is adsorbed to a contour matching point to form a contour stroke; When generating trajectory stroke points, trajectory matching points are searched from the real scene point cloud, and each trajectory stroke point is adsorbed to a trajectory matching point to form a trajectory stroke; A target object rough model is generated based on the contour strokes and the trajectory strokes.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores an augmented reality-based three-dimensional modeling program, and the augmented reality-based three-dimensional modeling program, when executed by a processor, implements the augmented reality-based three-dimensional modeling method of any one of claims 1 to 6.

9. A computer device, comprising: The computer device includes a computer-readable storage medium, a processor, and an augmented reality-based three-dimensional modeling program stored in the computer-readable storage medium, and the augmented reality-based three-dimensional modeling program, when executed by the processor, implements the augmented reality-based three-dimensional modeling method of any one of claims 1 to 6.

Images