A real-time editing method and device for three-dimensional models based on Unity platform and AR technology

Abstract

The present invention discloses a method and device for real-time editing of three-dimensional models based on the Unity platform and AR technology, the method comprising: obtaining a three-dimensional model to be edited in real time; generating alignment mapping information between the three-dimensional model to be edited in real time and the three-dimensional space in which the AR device is located based on preset markers, and generating an AR fusion image in the Unity platform based on the alignment mapping information; determining the difference target information between the three-dimensional model to be edited in real time and the real environment based on the AR fusion image, the difference target information including category information and coordinate information; updating the three-dimensional model to be edited in real time based on the alignment mapping information and the difference target information to generate an updated three-dimensional model. Through the deep integration of Unity and AR, the present invention solves the core problems of traditional 3D editing tools such as lack of spatial sense, inefficient collaboration, and complex cross-platform adaptation, and realizes real-time synchronization between the real space and the three-dimensional model.

Description

Technical Field

[0001] The present invention belongs to the technical field of the intersection of augmented reality (AR) and computer graphics processing, and specifically relates to a method and device for real-time editing of three-dimensional models based on the Unity development platform and AR technology. Background Art

[0002] With the rapid development of augmented reality (AR) technology, real-time 3D model editing systems built on the Unity engine are thriving in numerous fields, demonstrating enormous potential for application. The Unity platform, with its exceptional cross-platform compatibility, easily adapts to a variety of devices, including mobile phones, tablets, and head-mounted displays. Its powerful real-time rendering capabilities allow virtual models to render realistic lighting and shadow effects in real-world scenes. Its extensive development toolchain provides developers with a convenient and efficient creative path, making it a core engine for AR application development.

[0003] While the integration of AR and Unity has achieved some success, enabling basic 3D model display and simple interactions, such as allowing users to translate and rotate models and view them from different perspectives through gestures or device manipulation, current systems lack real-time editing capabilities, a significant shortcoming that needs to be addressed urgently. Summary of the Invention

[0004] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a method and device for real-time editing of three-dimensional models based on the Unity platform and AR technology. To achieve the above-mentioned purpose, the present invention is implemented by adopting the following technical solutions:

[0005] In a first aspect, the present invention provides a method for real-time editing of three-dimensional models based on the Unity platform and AR technology, comprising:

[0006] Obtain the 3D model to be edited in real time;

[0007] Generate alignment mapping information between the 3D model to be edited in real time and the 3D space in which the AR device is located based on the preset markers, and generate an AR fused image in the Unity platform based on the alignment mapping information;

[0008] Determining, based on the AR fused image, difference target information between the three-dimensional model to be edited in real time and the real environment, the difference target information including category information and coordinate information;

[0009] The method includes updating the three-dimensional model to be edited in real time according to the alignment mapping information and the difference target information to generate an updated three-dimensional model. The method includes updating the three-dimensional model to be edited in real time according to the alignment mapping information and the difference target information to generate an updated three-dimensional model, including:

[0010] According to the alignment mapping information and the coordinate information in the difference target information, the local coordinate position of the difference target in the three-dimensional model to be edited in real time is obtained through the Transform component of Unity;

[0011] According to the category information in the difference target information, matching the corresponding three-dimensional model from a preset three-dimensional model library to determine it as the target model to be added;

[0012] Using the Raycast method to perform spatial collision detection, and determining the scaling ratio of the target model to be added in combination with the geometric features of the target model to be added;

[0013] According to the scaling ratio and the local coordinate position, Mesh.RecalculateNormals and Mesh.RecalculateBounds are called to update the mesh topology structure and generate the updated three-dimensional model.

[0014] In combination with the first aspect, optionally, generating alignment mapping information between the three-dimensional model to be edited in real time and the three-dimensional space in which the AR device is located based on the preset marker includes:

[0015] Acquiring visual feature information of a preset marker in the three-dimensional space by an image acquisition module of the AR device, where the preset marker includes at least three non-collinear reference points;

[0016] Extracting the marker spatial coordinate data from the visual feature information, including the relative position parameters and direction vector of the reference point in the AR device coordinate system;

[0017] Performing coordinate correlation matching on the reference point coordinate data of the three-dimensional model to be edited in real time and the marker space coordinate data in the visual feature information to generate a coordinate system transformation matrix;

[0018] Based on the transformation matrix, the vertex coordinates of the three-dimensional model to be edited in real time are mapped to the three-dimensional space coordinate system of the AR device to generate the alignment mapping information.

[0019] In combination with the first aspect, optionally, generating an AR fused image in a Unity platform according to the alignment mapping information includes:

[0020] Receive the alignment mapping information, and convert the alignment mapping information into quaternion pose data in the Unity world coordinate system through Unity's AR Foundation plug-in;

[0021] Creating a virtual object in a Unity scene, and dynamically associating the preset marker with the virtual object;

[0022] Call Unity's rendering pipeline to perform virtual-reality fusion rendering, render the image captured by the AR device to the background layer, and render the virtual object to the foreground layer to generate the AR fusion image.

[0023] In combination with the first aspect, optionally, determining the difference target information between the three-dimensional model to be edited in real time and the real environment according to the AR fusion image includes:

[0024] Using a sliding window to calculate the HSV color difference value and edge gradient difference value in the AR fusion image within each window;

[0025] When the HSV color difference value within the sliding window is greater than a first threshold and the edge gradient difference value is greater than a second threshold, marking the sliding window as a candidate difference region and adding the candidate difference region to a candidate difference region set;

[0026] Merging all pixel values in the candidate difference region set to determine the difference pixel value;

[0027] The difference target information is generated according to the difference pixel value.

[0028] In combination with the first aspect, optionally, generating the difference target information according to the difference pixel value includes:

[0029] Clustering the difference pixel values using a density clustering method to generate a set of difference regions;

[0030] Obtaining a minimum bounding box for each difference area in the difference area set;

[0031] Determine the target category of the difference area using the Yolo model in the minimum bounding box;

[0032] The vertex coordinates of the minimum bounding box and the target category of the difference area are determined as the difference target information.

[0033] In a second aspect, the present invention provides a three-dimensional model real-time editing device based on the Unity platform and AR technology, comprising:

[0034] Model loading module: obtains the 3D model to be edited in real time;

[0035] A fusion image generation module generates alignment mapping information between the 3D model to be edited in real time and the 3D space in which the AR device is located based on preset markers, and generates an AR fusion image in the Unity platform based on the alignment mapping information.

[0036] A difference target determination module is configured to determine difference target information between the three-dimensional model to be edited in real time and the real environment based on the AR fusion image, wherein the difference target information includes category information and coordinate information;

[0037] Model update module: updating the three-dimensional model to be edited in real time according to the alignment mapping information and the difference target information to generate an updated three-dimensional model; updating the three-dimensional model to be edited in real time according to the alignment mapping information and the difference target information to generate an updated three-dimensional model, including: obtaining the local coordinate position of the difference target in the three-dimensional model to be edited in real time through Unity's Transform component according to the alignment mapping information and the coordinate information in the difference target information; matching the corresponding three-dimensional model from a preset three-dimensional model library according to the category information in the difference target information, and determining it as the target model to be added; using the Raycast method to perform spatial collision detection, and determining the scaling ratio of the target model to be added in combination with the geometric features of the target model to be added; calling Mesh.RecalculateNormals and Mesh.RecalculateBounds according to the scaling ratio and the local coordinate position to update the mesh topology structure to generate the updated three-dimensional model.

[0038] Compared with the prior art, the beneficial effects achieved by the method and device for real-time editing of three-dimensional models based on the Unity platform and AR technology provided by the embodiments of the present invention include:

[0039] The present invention obtains a three-dimensional model to be edited in real time; generates alignment mapping information between the three-dimensional model to be edited in real time and the three-dimensional space in which the AR device is located based on preset markers, and generates an AR fusion image in the Unity platform based on the alignment mapping information; determines the difference target information between the three-dimensional model to be edited in real time and the real environment based on the AR fusion image, and the difference target information includes category information and coordinate information; updates the three-dimensional model to be edited in real time based on the alignment mapping information and the difference target information to generate an updated three-dimensional model. Through the deep integration of Unity and AR, the present invention solves the core problems of traditional 3D editing tools such as lack of spatial sense, inefficient collaboration, and complex cross-platform adaptation, and realizes real-time synchronization between the real space and the three-dimensional model. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a flowchart of a method for real-time editing of three-dimensional models based on the Unity platform and AR technology provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0041] It should be noted that, unless there is a conflict, the embodiments and features in the embodiments of this embodiment can be combined with each other. The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0042] Example 1:

[0043] like Figure 1 As shown, an embodiment of the present invention provides a real-time editing method for a three-dimensional model based on the Unity platform and AR technology, including:

[0044] S1: Obtain the 3D model to be edited in real time;

[0045] S2: Generate alignment mapping information between the 3D model to be edited in real time and the 3D space where the AR device is located based on the preset markers, and generate an AR fused image in the Unity platform based on the alignment mapping information;

[0046] Specifically, generating alignment mapping information between the three-dimensional model to be edited in real time and the three-dimensional space in which the AR device is located based on the preset markers, and generating an AR fused image in the Unity platform based on the alignment mapping information includes:

[0047] S2-1: Collecting visual feature information of a preset marker in the three-dimensional space through an image acquisition module of the AR device, where the preset marker includes at least three non-collinear reference points;

[0048] S2-2: Extracting the marker spatial coordinate data from the visual feature information, including the relative position parameters and direction vector of the reference point in the AR device coordinate system;

[0049] S2-3: performing coordinate correlation matching on the reference point coordinate data of the three-dimensional model to be edited in real time and the marker space coordinate data in the visual feature information to generate a coordinate system transformation matrix;

[0050] S2-4: Mapping the vertex coordinates of the three-dimensional model to be edited in real time to the three-dimensional space coordinate system of the AR device based on the transformation matrix to generate the alignment mapping information;

[0051] S2-5: Receive the alignment mapping information, and convert the alignment mapping information into quaternion pose data in the Unity world coordinate system through the Unity AR Foundation plug-in;

[0052] S2-6: creating a virtual object in the Unity scene, and dynamically associating the preset marker with the virtual object;

[0053] S2-7: Calling Unity's rendering pipeline to perform virtual-reality fusion rendering, rendering the image captured by the AR device to the background layer, and rendering the virtual object to the foreground layer, to generate the AR fusion image;

[0054] S3: determining difference target information between the three-dimensional model to be edited in real time and the real environment according to the AR fusion image, where the difference target information includes category information and coordinate information;

[0055] Specifically, determining the difference target information between the three-dimensional model to be edited in real time and the real environment according to the AR fusion image, wherein the difference target information includes category information and coordinate information, includes:

[0056] S3-1: Calculate the HSV color difference value and edge gradient difference value of the AR fusion image in each window using a sliding window;

[0057] S3-2: When the HSV color difference value within the sliding window is greater than a first threshold and the edge gradient difference value is greater than a second threshold, mark the sliding window as a candidate difference region and add the candidate difference region to a set of candidate difference regions;

[0058] S3-3: merging all pixel values in the candidate difference region set to determine a difference pixel value;

[0059] S3-4: Generate the difference target information according to the difference pixel value.

[0060] In a specific embodiment, generating the difference target information according to the difference pixel value includes:

[0061] S3-4-1: clustering the difference pixel values using a density clustering method to generate a set of difference regions;

[0062] S3-4-2: Obtain a minimum bounding box for each different area in the set of different areas;

[0063] S3-4-3: Determine the target category of the difference area using the YOLO model in the minimum bounding box;

[0064] S3-4-4: Determine the vertex coordinates of the minimum bounding box and the target category of the difference area as the difference target information.

[0065] S4: updating the three-dimensional model to be edited in real time according to the alignment mapping information and the difference target information to generate an updated three-dimensional model;

[0066] Specifically, updating the to-be-edited three-dimensional model according to the alignment mapping information and the difference target information to generate an updated three-dimensional model includes:

[0067] S4-1: Based on the alignment mapping information and the coordinate information in the difference target information, obtain the local coordinate position of the difference target in the 3D model to be edited in real time through the Transform component of Unity;

[0068] S4-2: According to the category information in the difference target information, matching a corresponding 3D model from a preset 3D model library to determine it as the target model to be added;

[0069] S4-3: Using the Raycast method to perform spatial collision detection, and determining the scaling ratio of the target model to be added in combination with the geometric features of the target model to be added;

[0070] S4-4: Call Mesh.RecalculateNormals and Mesh.RecalculateBounds to update the mesh topology structure according to the scaling ratio and the local coordinate position, and generate the updated three-dimensional model.

[0071] The present invention provides a real-time editing method for three-dimensional models based on the Unity platform and AR technology. Through the deep integration of Unity and AR, it solves the core problems of traditional 3D editing tools such as lack of spatial sense, inefficient collaboration, and complex cross-platform adaptation, and realizes real-time synchronization between real space and three-dimensional models.

[0072] Example 2:

[0073] An embodiment of the present invention provides a real-time editing device for three-dimensional models based on the Unity platform and AR technology, comprising:

[0074] Model loading module: obtains the 3D model to be edited in real time;

[0075] A fusion image generation module generates alignment mapping information between the 3D model to be edited in real time and the 3D space in which the AR device is located based on preset markers, and generates an AR fusion image in the Unity platform based on the alignment mapping information.

[0076] A difference target determination module is configured to determine difference target information between the three-dimensional model to be edited in real time and the real environment based on the AR fusion image, wherein the difference target information includes category information and coordinate information;

[0077] Model update module: updating the three-dimensional model to be edited in real time according to the alignment mapping information and the difference target information to generate an updated three-dimensional model; updating the three-dimensional model to be edited in real time according to the alignment mapping information and the difference target information to generate an updated three-dimensional model, including: obtaining the local coordinate position of the difference target in the three-dimensional model to be edited in real time through Unity's Transform component according to the alignment mapping information and the coordinate information in the difference target information; matching the corresponding three-dimensional model from a preset three-dimensional model library according to the category information in the difference target information, and determining it as the target model to be added; using the Raycast method to perform spatial collision detection, and determining the scaling ratio of the target model to be added in combination with the geometric features of the target model to be added; calling Mesh.RecalculateNormals and Mesh.RecalculateBounds according to the scaling ratio and the local coordinate position to update the mesh topology structure to generate the updated three-dimensional model.

[0078] The present invention provides a real-time editing device for three-dimensional models based on the Unity platform and AR technology. Through the deep integration of Unity and AR, it solves the core problems of traditional 3D editing tools such as lack of spatial sense, inefficient collaboration, and complex cross-platform adaptation, and realizes real-time synchronization between real space and three-dimensional models.

[0079] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0080] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0081] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0082] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0083] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A real-time editing method for three-dimensional models based on Unity platform and AR technology, characterized in that: include: Obtain the 3D model to be edited in real time; Generate alignment mapping information between the 3D model to be edited in real time and the 3D space in which the AR device is located based on the preset markers, and generate an AR fused image in the Unity platform based on the alignment mapping information; Determining, based on the AR fused image, difference target information between the three-dimensional model to be edited in real time and the real environment, the difference target information including category information and coordinate information; updating the three-dimensional model to be edited in real time according to the alignment mapping information and the difference target information to generate an updated three-dimensional model; The updating, based on the alignment mapping information and the difference target information, in the three-dimensional model to be edited in real time to generate an updated three-dimensional model includes: According to the alignment mapping information and the coordinate information in the difference target information, the local coordinate position of the difference target in the three-dimensional model to be edited in real time is obtained through the Transform component of Unity; According to the category information in the difference target information, matching the corresponding three-dimensional model from a preset three-dimensional model library to determine it as the target model to be added; Using the Raycast method to perform spatial collision detection, and determining the scaling ratio of the target model to be added in combination with the geometric features of the target model to be added; According to the scaling ratio and the local coordinate position, Mesh.RecalculateNormals and Mesh.RecalculateBounds are called to update the mesh topology structure and generate the updated three-dimensional model.

2. The method for real-time editing of three-dimensional models based on Unity platform and AR technology according to claim 1, characterized in that: The step of generating alignment mapping information between the three-dimensional model to be edited in real time and the three-dimensional space in which the AR device is located according to the preset markers includes: Acquiring visual feature information of a preset marker in the three-dimensional space by an image acquisition module of the AR device, where the preset marker includes at least three non-collinear reference points; Extracting the marker spatial coordinate data from the visual feature information, including the relative position parameters and direction vector of the reference point in the AR device coordinate system; Performing coordinate correlation matching on the reference point coordinate data of the three-dimensional model to be edited in real time and the marker space coordinate data in the visual feature information to generate a coordinate system transformation matrix; Based on the transformation matrix, the vertex coordinates of the three-dimensional model to be edited in real time are mapped to the three-dimensional space coordinate system of the AR device to generate the alignment mapping information.

3. The method for real-time editing of three-dimensional models based on Unity platform and AR technology according to claim 1, characterized in that: Generating an AR fusion image in the Unity platform according to the alignment mapping information includes: Receive the alignment mapping information, and convert the alignment mapping information into quaternion pose data in the Unity world coordinate system through Unity's AR Foundation plug-in; Creating a virtual object in a Unity scene, and dynamically associating the preset marker with the virtual object; Call Unity's rendering pipeline to perform virtual-reality fusion rendering, render the image captured by the AR device to the background layer, and render the virtual object to the foreground layer to generate the AR fusion image.

4. The method for real-time editing of three-dimensional models based on Unity platform and AR technology according to claim 1, characterized in that: The determining, based on the AR fusion image, difference target information between the three-dimensional model to be edited in real time and the real environment includes: Using a sliding window to calculate the HSV color difference value and edge gradient difference value in the AR fusion image within each window; When the HSV color difference value within the sliding window is greater than a first threshold and the edge gradient difference value is greater than a second threshold, marking the sliding window as a candidate difference region and adding the candidate difference region to a candidate difference region set; Merging all pixel values in the candidate difference region set to determine the difference pixel value; The difference target information is generated according to the difference pixel value.

5. The method for real-time editing of three-dimensional models based on Unity platform and AR technology according to claim 4, characterized in that: Generating the difference target information according to the difference pixel value includes: Clustering the difference pixel values using a density clustering method to generate a set of difference regions; Obtaining a minimum bounding box for each difference area in the difference area set; Determine the target category of the difference area using the Yolo model in the minimum bounding box; The vertex coordinates of the minimum bounding box and the target category of the difference area are determined as the difference target information.

6. A 3D model real-time editing device based on Unity platform and AR technology, characterized in that: include: Model loading module: obtains the 3D model to be edited in real time; A fusion image generation module generates alignment mapping information between the 3D model to be edited in real time and the 3D space in which the AR device is located based on preset markers, and generates an AR fusion image in the Unity platform based on the alignment mapping information. A difference target determination module is configured to determine difference target information between the three-dimensional model to be edited in real time and the real environment based on the AR fusion image, wherein the difference target information includes category information and coordinate information; Model updating module: updating the three-dimensional model to be edited in real time according to the alignment mapping information and the difference target information to generate an updated three-dimensional model; The updating is performed in the three-dimensional model to be edited in real time according to the alignment mapping information and the difference target information to generate an updated three-dimensional model, including: obtaining the local coordinate position of the difference target in the three-dimensional model to be edited in real time through the Transform component of Unity according to the alignment mapping information and the coordinate information in the difference target information; matching the corresponding three-dimensional model from a preset three-dimensional model library according to the category information in the difference target information to determine it as the target model to be added; using the Raycast method to perform spatial collision detection, and determining the scaling ratio of the target model to be added in combination with the geometric features of the target model to be added; calling Mesh.RecalculateNormals and Mesh.RecalculateBounds according to the scaling ratio and the local coordinate position to update the mesh topology structure to generate the updated three-dimensional model.

Images