Three-dimensional model data processing method, device, equipment, medium and product

Abstract

The present application discloses a three-dimensional model data processing method, device, equipment, medium and product, which relate to the field of image processing. The method comprises: obtaining a global view of a first object, the first object comprising at least two object components; segmenting from the global view to obtain local views corresponding to at least two object components; converting the local views corresponding to at least two object components into component three-dimensional models; combining the component three-dimensional models corresponding to at least two object components to obtain an object three-dimensional model of the first object. The method enables the generated object three-dimensional model to have higher editing flexibility in downstream applications. In the downstream application process, when the object three-dimensional model needs to be locally adjusted, it supports local editing by adjusting the model data corresponding to the component three-dimensional model, thereby reducing the workload of the object three-dimensional model in downstream applications.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a three-dimensional model data processing method, device, equipment, medium and product. Background Art

[0002] With the development of artificial intelligence (AI) technology, AI technology can be used to complete the modeling of three-dimensional models through two-dimensional images, which greatly simplifies the process of creating three-dimensional models.

[0003] In the related art, a two-dimensional image is directly input into a pre-trained AI model, and a three-dimensional model corresponding to the target to be generated in the two-dimensional image is generated by the AI ​​model.

[0004] However, the three-dimensional model generated by the above scheme has the problem of poor model flexibility. Summary of the invention

[0005] The embodiments of the present application provide a three-dimensional model data processing method, device, equipment, medium and product. The technical solution is as follows.

[0006] In one aspect, a method for processing three-dimensional model data is provided, the method comprising:

[0007] Acquire a global view of a first object, the global view comprising a two-dimensional image showing the first object from a first viewing direction, the first object comprising at least two object components;

[0008] Segmenting from the global view to obtain local views corresponding to the at least two object components, respectively, the local views comprising two-dimensional images showing the corresponding object components from the first viewing direction;

[0009] Converting the local views corresponding to the at least two object components into component three-dimensional models;

[0010] The component three-dimensional models respectively corresponding to the at least two object components are combined to obtain an object three-dimensional model of the first object.

[0011] In another aspect, a three-dimensional model data processing device is provided, the device comprising:

[0012] an acquisition module, configured to acquire a global view of a first object, wherein the global view comprises a two-dimensional image showing the first object from a first viewing direction, wherein the first object comprises at least two object components;

[0013] a segmentation module, configured to segment from the global view to obtain local views corresponding to the at least two object components, the local views comprising two-dimensional images showing the corresponding at least two object components from the first viewing angle;

[0014] A generating module, used for converting the local views corresponding to the at least two object components into a component three-dimensional model;

[0015] The combining module is used to combine the component three-dimensional models corresponding to the at least two object components to obtain the object three-dimensional model of the first object.

[0016] In some optional embodiments, the segmentation module includes:

[0017] A first acquisition unit, configured to acquire position distribution information of the at least two object components in the global view, wherein the position distribution information is used to indicate distribution of the at least two object components in the global view;

[0018] A segmentation unit is used to segment the first object in the global view based on the position distribution information to obtain local views corresponding to the at least two object components.

[0019] In some optional embodiments, the first acquisition unit is further used to identify object components existing in the global view based on the image semantics of the global view, and obtain position distribution information corresponding to the at least two object components respectively;

[0020] The first acquisition unit is also used to receive marking operations corresponding to the at least two object components in the global view, and the marking operation is used to mark at least one marking point corresponding to the at least two object components; based on the marking point indicated by the marking operation, generate position distribution information corresponding to the at least two object components.

[0021] In some optional embodiments, the segmentation unit is further used to input the position distribution information corresponding to the global view and the at least two object components into a regional segmentation model to obtain local views corresponding to the at least two object components. The regional segmentation model is a pre-trained machine learning model for segmenting the area where each object component is located from the global view.

[0022] In some optional embodiments, the generation module is also used to input the local views corresponding to the at least two object components into the three-dimensional modeling model to obtain the three-dimensional models of the components corresponding to the at least two object components. The three-dimensional modeling model is a pre-trained machine learning model for outputting the three-dimensional model of the component based on the local views.

[0023] In some optional embodiments, the device further comprises:

[0024] a completion module, configured to input the local views corresponding to the at least two object components into an image generation model to obtain a third view after the occluded parts in the local views are completed, wherein the image generation model is a machine learning model pre-trained to fill the image content of the occluded areas of the object components in the local views;

[0025] The generation module is further used to input the third views corresponding to the at least two object components into the three-dimensional modeling model to obtain the component three-dimensional models corresponding to the at least two object components.

[0026] In some optional embodiments, the combined module includes:

[0027] A second acquisition unit, used to acquire first position information of the at least two object components in the three-dimensional space, where the first position information is used to indicate positions of the at least two object components in the three-dimensional space when they constitute the first object;

[0028] A combining unit is used to combine the component three-dimensional models corresponding to the at least two object components according to the first position information to obtain the object three-dimensional model of the first object.

[0029] In some optional embodiments, the generating module is further used to convert the global view into a registered three-dimensional model corresponding to the first object;

[0030] The second acquisition unit is further used to determine the first position information of the at least two object components according to the mapping relationship between the component three-dimensional models respectively corresponding to the at least two object components and the registered three-dimensional model.

[0031] In some optional embodiments, the combined module further includes:

[0032] a mapping unit, configured to establish a first mapping relationship between the component three-dimensional models respectively corresponding to the at least two object components and a first registration area in the global view, wherein the first registration area is used to indicate the areas where the at least two object components are respectively located in the global view;

[0033] The mapping unit is further used to establish a second mapping relationship between first registration areas corresponding to the at least two object components in the global view and a second registration area of ​​the first object, wherein the second registration area is used to indicate an area where the first object is located in the global view;

[0034] The mapping unit is further used to establish a third mapping relationship between the second registration area of ​​the first object in the global view and the registered three-dimensional model;

[0035] The mapping unit is further used to determine a fourth mapping relationship between the component three-dimensional models respectively corresponding to the at least two object components and the registered three-dimensional model according to a mapping chain formed by the first mapping relationship, the second mapping relationship and the third mapping relationship;

[0036] The second acquisition unit is further used to determine the first position information of the at least two object components according to the fourth mapping relationship.

[0037] In some optional embodiments, the fourth mapping relationship includes at least two first point pair relationships corresponding to the at least two object components, respectively, and the first point pair relationship is used to indicate a mapping relationship between a mapping point pair formed by a first position point in the component three-dimensional model and a second position point in the registration three-dimensional model;

[0038] The second acquisition unit is further used to screen the at least two first point pair relationships according to the mapping accuracy of the at least two first point pair relationships corresponding to the i-th object component to obtain a second point pair relationship, where i is a positive integer;

[0039] The second acquisition unit is further used to determine the first position information according to the second point pair relationship corresponding to the at least two object components.

[0040] In some optional embodiments, the second acquisition unit is further used to sample at least one third point pair relationship from the at least two first point pair relationships corresponding to the i-th object component;

[0041] The second acquisition unit is further configured to use the at least one third point pair relationship as a reference transformation relationship to determine mapping accuracies corresponding to the at least two first point pair relationships respectively;

[0042] The second acquisition unit is further configured to filter the at least two first point pair relationships based on a threshold according to the mapping accuracies respectively corresponding to the at least two first point pair relationships, so as to obtain the second point pair relationship.

[0043] In some optional embodiments, the combined module further includes:

[0044] A registration unit, configured to register the at least two object components with the registration three-dimensional model in a first registration mode based on the first position information, to obtain registration results corresponding to the at least two object components respectively;

[0045] The combining unit is further configured to combine the component three-dimensional models respectively corresponding to the at least two object components based on the registration result to obtain the object three-dimensional model of the first object.

[0046] In some optional embodiments, the registration unit is further used to register the model data of the three-dimensional model of the component in a second registration mode according to the positional relationship between the at least two object components indicated by the first position information, to obtain registration model data corresponding to the at least two object components, respectively, the registration model data being used to indicate the model data obtained after the deformation of the three-dimensional model of the component is completed by the second registration mode;

[0047] The combining unit is further used to combine the registration model data corresponding to the at least two object components respectively according to the registration result to obtain the object three-dimensional model of the first object.

[0048] On the other hand, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement a three-dimensional model data processing method as described in any of the above-mentioned embodiments of the present application.

[0049] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction, at least one program, a code set or an instruction set is stored in the storage medium, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement a three-dimensional model data processing method as described in any of the above-mentioned embodiments of the present application.

[0050] On the other hand, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the three-dimensional model data processing method described in any of the above embodiments.

[0051] The technical solution provided by this application includes at least the following beneficial effects.

[0052] In the process of generating a three-dimensional model from a two-dimensional image, the global view of the first object is divided into at least two local views corresponding to the object components, and the component three-dimensional models corresponding to each object component are generated from the local views, so that the object three-dimensional model of the first object is obtained by combining the component three-dimensional models of at least two object components. By decomposing the first object into at least two object components, the generation granularity of the object three-dimensional model is refined, and an independent model data is generated for each object component, which solves the problem of object component adhesion caused by all components belonging to the same model data in the object three-dimensional model of the first object finally obtained, so that the object three-dimensional model has higher editing flexibility in downstream applications. In the downstream application process, when the object three-dimensional model needs to be locally adjusted, it supports local editing by adjusting the model data corresponding to the component three-dimensional model, which reduces the workload of the object three-dimensional model in downstream applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0054] Figure 1 is a structural block diagram of a computer system provided by an exemplary embodiment of the present application;

[0055] Figure 2 is a flow chart of a three-dimensional model data processing method provided by an exemplary embodiment of the present application;

[0056] Figure 3 are three views of a first object provided by an exemplary embodiment of the present application;

[0057] Figure 4 is a flow chart of a three-dimensional model data processing method provided by another exemplary embodiment of the present application;

[0058] Figure 5 is a schematic diagram of an implementation process of a marking operation provided by an exemplary embodiment of the present application;

[0059] Figure 6 is a schematic diagram of the model structure of a large visual segmentation model provided by an exemplary embodiment of the present application;

[0060] Figure 7 is a schematic diagram of segmentation of object components provided by an exemplary embodiment of the present application;

[0061] Figure 8is a flowchart of a three-dimensional model data processing method provided by another exemplary embodiment of the present application;

[0062] Fig. 9 is a schematic diagram of a model structure provided by an exemplary embodiment of the present application;

[0063] Fig.10 is a schematic diagram of an image completion process provided by an exemplary embodiment of the present application;

[0064] Fig.11 is a flowchart of a three-dimensional model data processing method provided by another exemplary embodiment of the present application;

[0065] Fig.12 is a schematic diagram of a process of combining component three-dimensional models provided by an exemplary embodiment of the present application;

[0066] Fig.13 It is a schematic diagram comparing the effects of directly generating a three-dimensional model through the CraftsMan model and generating a three-dimensional model through the method of the embodiment of the present application;

[0067] Fig.14 is a structural block diagram of a three-dimensional model data processing device provided by an exemplary embodiment of the present application;

[0068] Fig.15 is a structural block diagram of a three-dimensional model data processing device provided by another exemplary embodiment of the present application;

[0069] Fig.16 It is a structural diagram of a computer device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0070] In order to make the purpose, technical solution and advantages of the present application clearer, the implementation mode of the present application will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0071] In this application, the terms "first", "second", etc. are used to distinguish identical items or similar items with substantially the same effects and functions. It should be understood that there is no logical or temporal dependency between "first" and "second", nor are there any limitations on quantity and execution order.

[0072] In order to improve the editing freedom of the 3D model generated by AI, a 3D model data processing method is provided in the embodiment of the present application. Figure 1, which shows a structural block diagram of a computer system provided by an exemplary embodiment of the present application. The computer system is used to provide a hardware environment for the three-dimensional model data processing method provided by an embodiment of the present application. The computer system includes: a terminal 110 and a server 120.

[0073] The terminal 110 has an application program installed and running that supports the generation of a three-dimensional model. The device type of the terminal 110 includes at least one of a game console, a desktop computer, a smart phone, a tablet computer, and a laptop computer.

[0074] The server 120 includes at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. The server 120 is used to provide backend services for applications that support the generation of three-dimensional models. Optionally, the server 120 undertakes the main computing work, and the terminal 110 undertakes the secondary computing work; or, the server 120 undertakes the secondary computing work, and the terminal 110 undertakes the main computing work; or, the server 120 and the terminal 110 use a distributed computing architecture to coordinate computing.

[0075] It is worth noting that the server 120 can be implemented as a physical server or a cloud server in the cloud. In some embodiments, the server 120 can also be implemented as a node in a blockchain system.

[0076] The terminal 110 is connected to the server 120 via a wireless network or a wired network.

[0077] In some embodiments, taking the three-dimensional model data processing method implemented by the terminal 110 and the server 120 in collaboration as an example, the user sends the global view 101 of the first object for which three-dimensional model generation is required to be performed to the server 120 through the terminal 110. After receiving the global view 101 of the first object, the server 120 sends the global view 101 to the three-dimensional model generation service, and obtains at least two local views corresponding to the object components respectively from the global view 101 through the three-dimensional model generation service, converts the local views corresponding to the at least two object components respectively into component three-dimensional models, and combines the component three-dimensional models corresponding to the at least two object components respectively, to obtain the object three-dimensional model 104 of the first object. The server 120 sends the generated object three-dimensional model 104 to the terminal 110, and the terminal 110 displays the object three-dimensional model 104 on the display screen.

[0078] In other embodiments, the 3D model data processing method may also be independently implemented by the terminal 110 or the server 120. Exemplarily, when the 3D model data processing method is independently executed by the terminal 110, the user loads the global view 101 stored in the terminal 110 into an application supporting 3D model generation, and the application obtains local views corresponding to at least two object components from the global view 101, and converts the local views corresponding to at least two object components into component 3D models by calling an AI processing chip and a graphics processing unit (GPU), and finally obtains the object 3D model 104 of the first object by combining the component 3D models corresponding to at least two object components. Exemplarily, when the three-dimensional model data processing method is independently executed by the server 120, when the server 120 detects the need to generate a three-dimensional model of the first object, the server 120 reads the global view 101 from the database and sends the global view 101 to the three-dimensional model generation service. The three-dimensional model generation service segments the global view 101 to obtain local views corresponding to at least two object components, converts the local views corresponding to the at least two object components into component three-dimensional models, and combines the component three-dimensional models corresponding to the at least two object components to obtain the object three-dimensional model 104 of the first object.

[0079] Those skilled in the art will appreciate that the number of the above devices may be more or less. For example, the above device may be only one, or the above devices may be dozens or hundreds, or more. The embodiment of the present application does not limit the number and type of devices.

[0080] In some embodiments, the method provided in the embodiments of the present application can be applied to cloud application scenarios, so that the calculation of the data logic for generating the three-dimensional model is completed through the cloud server, and the terminal 110 is responsible for the display of the three-dimensional model generation process.

[0081] Please refer to Figure 2 , which shows a flowchart of a three-dimensional model data processing method provided by an exemplary embodiment of the present application, wherein the method is Figure 1 Taking the execution of the server 120 in as an example, the method includes the following steps 210 to 240.

[0082] Step 210: Acquire a global view of the first object.

[0083] The first object is an object instance of a three-dimensional model to be generated. A three-dimensional model refers to a model structure constructed on a computer device and expressed in a three-dimensional form. The three-dimensional model can also be implemented as a three-dimensional virtual model, which includes a model expressed as a virtual entity in a three-dimensional virtual scene. Schematically, the three-dimensional virtual model is implemented as a virtual character model, a virtual animal model, a virtual building model, etc. in a virtual scene.

[0084] The first object is implemented as at least one of a person object, a building object, an animal object, a decoration object, a prop object, a container object, etc. Taking the first object as a virtual character object in a game as an example, a three-dimensional model corresponding to the virtual character object is generated according to the virtual character view of the virtual character object.

[0085] In an embodiment of the present application, the first object includes at least two object components, wherein the object components are used to indicate the parts that constitute the first object. In one example, taking the first object as a virtual character as an example, the object components include a head part, a hair accessory part, a hairstyle part, a torso part, a limb part, a clothing part, and a shoe part.

[0086] Optionally, the at least two object components are implemented as a set of all components in the first object, or the at least two object components are implemented as components of a part of the first object.

[0087] The global view is a two-dimensional image for displaying the first object. In the embodiment of the present application, the global view includes a two-dimensional image for displaying the first object from a first viewing angle.

[0088] Optionally, the first viewing angle direction may be implemented as a single viewing angle direction or as at least two viewing angle directions.

[0089] Optionally, the global view includes at least one of a front view, a left view, a right view, a top view, a bottom view, and a rear view for the first object. The front view is a view showing the first object from the front viewing direction; the left view is a view showing the first object from the left viewing direction; the right view is a view showing the first object from the right viewing direction; the top view is a view showing the first object from the top downward viewing direction; the bottom view is a view showing the first object from the bottom upward viewing direction; and the rear view is a view showing the first object from the back viewing direction of the first object.

[0090] When the first viewing direction is implemented as at least two viewing directions, the global view includes views showing the first object from at least two viewing directions, that is, the global view is a multi-view of the first object. Optionally, the multi-view can be implemented as two views, three views, six views, etc., which are not limited here.

[0091] In one example, the global view is implemented as three views of the first object as an example, Figure 3 As shown, it shows three views 300 of a first object provided by an exemplary embodiment of the present application, the three views 300 include a front view 301 showing the first object from a front viewing direction, a side view 302 showing the first object from a side viewing direction, and a rear view 303 showing the first object from a rear viewing direction.

[0092] Optionally, the global view of the first object may be implemented as an image drawn by a computer device; or, the global view of the first object may be implemented as an image obtained by photographing the entity of the first object by a camera device.

[0093] Step 220: Segment the global view to obtain at least two local views corresponding to the object components.

[0094] In an embodiment of the present application, a partial view includes a two-dimensional image showing a corresponding object component from a first viewing angle. Exemplarily, a first object includes n object components, and the i-th partial view includes a two-dimensional image showing the i-th object component from the first viewing angle, where i and n are positive integers, n≥2, i≤n.

[0095] In some embodiments, the global view is segmented by the position distribution of at least two object components in the global view to obtain local views corresponding to the at least two object components. Schematically, the position distribution information of at least two object components in the global view is obtained, wherein the position distribution information is used to indicate the distribution of at least two object components in the global view; the first object in the global view is segmented based on the position distribution information to obtain local views corresponding to the at least two object components. That is, the image area of ​​each object component in the global view is segmented in combination with the acquired position distribution information of each object component in the global view, thereby obtaining the local view, which can improve the segmentation accuracy when segmenting the image area corresponding to each object component in the global view in combination with the position distribution information, which is a priori information.

[0096] Optionally, the position distribution information of at least two object components in the global view may be indicated by manual annotation; or, the position distribution information of at least two object components in the global view may be automatically identified and determined by computer vision technology.

[0097] Optionally, the location distribution information includes fuzzy location indication information or precise location indication information. The fuzzy location indication information can be indicated by a marking point, a marking box, or a location description text, that is, the fuzzy location indication information only indicates the approximate location of the object component in the global view. The precise location indication information can be indicated by a mask indicating clear boundary information or area information, that is, the precise location indication information can indicate the specific location of the object component in the global view.

[0098] Optionally, when the position distribution information is implemented as fuzzy position indication information, the process of segmenting the global view according to the position distribution information is implemented by a pre-trained region segmentation model, wherein the region segmentation model is a pre-trained machine learning model for segmenting the region where each object component is located from the global view. Schematically, the position distribution information and the global view are input into the region segmentation model to obtain local views corresponding to at least two object components.

[0099] Optionally, when the position distribution information is implemented as precise position indication information, the global view is segmented according to the regional edge of the image region to which the object component provided by the position distribution information (precise position indication information) belongs to obtain a local view corresponding to the object component.

[0100] Step 230: convert the local views corresponding to at least two object components into a component three-dimensional model.

[0101] In an embodiment of the present application, after obtaining the local views corresponding to the various object components of the first object, the local views are converted into a three-dimensional model of the component, wherein the three-dimensional model of the component is stored in the form of model data in a computer device. Exemplarily, the local views are converted into model data corresponding to the three-dimensional model of the component.

[0102] The model data is data used to define the attribute information of the three-dimensional model of the object component, so that the object component can be accurately rendered and displayed in computer graphics applications. The attribute information includes at least one of the geometric shape, appearance, and model behavior of the object component.

[0103] Optionally, the model data is implemented as at least one of mesh data, voxel data, and point cloud data.

[0104] Among them, the mesh is a data structure used in three-dimensional computer graphics to represent the surface of a three-dimensional model. The mesh consists of a set of vertices, edges, and faces, which together define the geometric shape of the object. Optionally, the mesh units composed of edges in the mesh can be implemented as polygons such as triangles, quadrilaterals, and pentagons. In the mesh, vertices are the basic points that make up the mesh. They have coordinates (x, y, z) in three-dimensional space. Each vertex represents a position on the mesh and is the smallest unit that constitutes the three-dimensional model; edges are line segments connecting two vertices. Edges define the connection relationship between vertices and are the basis for the formation of faces; faces are polygons surrounded by multiple vertices and edges, that is, the above-mentioned mesh units. In the three-dimensional model, faces define the surface of the object, and multiple faces are combined to form the complete appearance of the object.

[0105] A voxel is a pixel in three-dimensional space, similar to a pixel in a two-dimensional image. Voxels are the basic unit of three-dimensional volume data. Each voxel occupies a cubic volume in three-dimensional space, and its size depends on the resolution of the entire three-dimensional model. Voxels are usually used to represent the volume information of a three-dimensional model in three-dimensional space, and can store a variety of data about the volume, such as geometric occupancy, density, color, etc. Each voxel in a three-dimensional model represented by voxel data has a clear spatial position, defined by its coordinates (x, y, z) in a three-dimensional coordinate system.

[0106] Point cloud is a model data representation method for a 3D model. Point cloud consists of a large number of 3D coordinate points, which together describe the surface of an object or scene. That is, point cloud is a collection of points formed by the spatial coordinates of each sampling point on the surface of a 3D object. Point cloud data includes the 3D coordinates (x, y, z) of the sampling point, as well as at least one additional attribute information of laser reflection intensity (Intensity), color information, and reflection intensity.

[0107] Optionally, the method of converting the local views corresponding to at least two object components into a component three-dimensional model may be implemented as at least one of the following.

[0108] The first one is manual conversion.

[0109] In some embodiments, local views corresponding to at least two object components are sent to at least one modeling terminal, and the modeler completes modeling of the object components according to the local views at the modeling terminal to obtain model data corresponding to the object components.

[0110] The second is conversion through AI technology.

[0111] In some embodiments, local views corresponding to at least two object components are input into a three-dimensional modeling model to obtain component three-dimensional models corresponding to at least two object components, wherein the three-dimensional modeling model is a pre-trained machine learning model for outputting component three-dimensional models based on local views.

[0112] Optionally, the above-mentioned three-dimensional modeling model is implemented as at least one of the Wonder3D model, a diffusion model based on Score Distillation Sampling (SDS), a CraftsMan model, a diffusion model based on the Make-It-3D framework (a framework for generating high-quality 3D models from a single image through diffusion priors), a Unique3D model, a Stable Video 3D (SV3D) model, etc., which is an artificial intelligence model that can convert a two-dimensional image into a three-dimensional model.

[0113] Step 240 , combining the component three-dimensional models corresponding to at least two object components to obtain an object three-dimensional model of the first object.

[0114] Illustratively, after generating component 3D models corresponding to the components of each object, the component 3D models of the components of each object are combined to obtain the object 3D model of the first object. In an embodiment of the present application, model data corresponding to the component 3D models are combined to obtain model data for representing the object 3D model.

[0115] In some embodiments, the component three-dimensional models corresponding to the at least two object components are combined according to the positions of the at least two object components in the three-dimensional space, thereby obtaining the object three-dimensional model. Schematically, first position information of the at least two object components in the three-dimensional space is obtained, wherein the first position information is used to indicate the positions corresponding to the at least two object components when forming the first object in the three-dimensional space; the component three-dimensional models corresponding to the at least two object components are combined according to the first position information to obtain the object three-dimensional model of the first object.

[0116] In some embodiments, the first position information of at least two object components in the three-dimensional space can be converted based on the second position information of the at least two object components in the two-dimensional plane. Schematically, the second position information of the at least two object components in the two-dimensional plane is obtained from the global view, and the second position information is converted into the first position information based on the spatial mapping relationship between the three-dimensional space and the two-dimensional plane.

[0117] In some embodiments, when the global view includes at least two views showing a first object from at least two first viewing directions, second position information of at least two object components in a two-dimensional plane is obtained from the at least two views respectively, and the second position information is converted into first position information based on a spatial mapping relationship between the three-dimensional space and the two-dimensional plane corresponding to each view.

[0118] Illustratively, the first position information includes at least one three-dimensional coordinate point corresponding to the object component in the three-dimensional coordinate system corresponding to the three-dimensional space. The three-dimensional coordinate system is implemented as a coordinate system established with a specified point in the specified three-dimensional space as the origin (for example, the edge point of the lower right corner of the three-dimensional space or the center point of the three-dimensional space), and with the x-axis, y-axis, and z-axis extended in three mutually perpendicular directions as coordinate axes.

[0119] In an illustrative manner, the second position information includes at least one two-dimensional coordinate point corresponding to the object component in the plane coordinate system corresponding to the global view. The plane coordinate system is implemented as a coordinate system established with a specified pixel in the global view as the origin, a first direction in the plane where the global view is located as the x-axis, and a second direction in the plane where the global view is located as the y-axis, wherein the first direction and the second direction are two directions perpendicular to each other.

[0120] There is a spatial mapping relationship between the three-dimensional coordinate point and the two-dimensional coordinate point. Optionally, the at least one two-dimensional coordinate point is implemented as a centroid position, a center position or any other position belonging to the object component.

[0121] Optionally, the mapping method of obtaining the first position information of the three-dimensional space according to the second position information of the two-dimensional plane can be implemented as at least one of perspective projection, orthographic projection, depth information prediction and addition, feature-based mapping, and parametric mapping, which is not limited here.

[0122] In an embodiment of the present application, after obtaining the object three-dimensional model of the first object by combining the component three-dimensional models of at least two object components, the object three-dimensional model can be applied to downstream scenarios such as animation driving, game modeling, medical scene simulation, architectural scene simulation, and cultural relics restoration research simulation.

[0123] In some embodiments, in the downstream application process, when there is a need to adjust the components of a specified object component among at least two object components, the model data corresponding to the object three-dimensional model, the model data of the component three-dimensional model corresponding to the specified object component, and the first position information used in the combination stage of the component three-dimensional models corresponding to at least two object components are obtained, and the model data of the component three-dimensional model corresponding to the object component is adjusted according to the component adjustment demand to obtain the adjusted model data, and the position of the specified object component indicated in the first position information in the three-dimensional space is used as a reference, and the model data of the component three-dimensional model corresponding to the specified object component in the model data corresponding to the object three-dimensional model is replaced with the adjusted model data, so as to obtain the object three-dimensional model after the component adjustment. Optionally, the above-mentioned component adjustment demand includes at least one of model deformation adjustment, model color adjustment, and model position adjustment. Taking the model data as mesh data as an example, the model deformation adjustment or model position adjustment is achieved by adjusting the mesh vertex position of the model data of the component three-dimensional model, and the model color adjustment is achieved by adjusting the color information corresponding to the mesh unit.

[0124] In summary, in the process of generating a three-dimensional model through a two-dimensional image, the global view of the first object is divided into local views corresponding to at least two object components, and the component three-dimensional models corresponding to each object component are generated through the local views, so that the object three-dimensional model of the first object is obtained by combining the component three-dimensional models of at least two object components. By decomposing the first object into at least two object components, the generation granularity of the object three-dimensional model is refined, and an independent model data is generated for each object component, which solves the problem of object component adhesion caused by all components belonging to the same model data in the object three-dimensional model of the first object finally obtained, so that the object three-dimensional model has higher editing flexibility in downstream applications. In the downstream application process, when the object three-dimensional model needs to be locally adjusted, it supports local editing by adjusting the model data corresponding to the component three-dimensional model, which reduces the workload of the object three-dimensional model in downstream applications.

[0125] In an embodiment of the present application, when the object three-dimensional model of the first object is generated through a global view, the following process steps are included.

[0126] Step 1: Object component segmentation.

[0127] In some optional embodiments, a region segmentation model is used to segment at least two local views corresponding to the object components from the global view. Figure 4, which shows a flowchart of a three-dimensional model data processing method provided by an exemplary embodiment of the present application, the method includes steps 221 to 222, wherein steps 221 to 222 are subordinate steps of step 220.

[0128] Step 221: Obtain position distribution information of at least two object components in the global view.

[0129] The position distribution information is used to indicate the distribution of at least two object components in the global view.

[0130] Optionally, the location distribution information may be acquired in at least one of the following ways.

[0131] The first one is manual labeling.

[0132] Illustratively, a user annotates each object component in the global view through a terminal, and the terminal generates location distribution information of at least two object components according to the user's annotation results.

[0133] Optionally, when marking each object component in the global view, the user can mark by indicating at least one marking point. Schematically, a marking operation corresponding to at least two object components in the global view is received, wherein the marking operation is used to mark at least one marking point corresponding to the at least two object components; based on the marking point indicated by the marking operation, position distribution information corresponding to the at least two object components is generated.

[0134] In some embodiments, the marker point location information of the marker points corresponding to the object components is recorded as the location distribution information corresponding to the object components. That is, the computing device (terminal or server) determines the location of each object component according to the marker points.

[0135] In some embodiments, different object components are indicated by markers of different styles. Optionally, the styles of the markers include at least one of marker color, marker shape, marker size, marker transparency, and marker identification. For example, a red marker is used to mark the head of the first object, a green marker is used to mark the clothes of the first object, and a blue marker is used to mark the torso of the first object.

[0136] In one example, a marking operation for each object component indicates three marking points, such as Figure 5As shown, it shows a schematic diagram of the implementation process of the marking operation provided by an exemplary embodiment of the present application. On the global view 500 of the first object provided, the user indicates the headwear part of the first object through the marking point A 510 indicated by the marking operation, indicates the clothing part of the first object through the marking point B 520, and indicates the belt part of the first object through the marking point C 530, thereby realizing the indication of the position distribution information of the object components of the first object through the above-mentioned marking points.

[0137] Optionally, when marking each object component in the global view, the user can mark the region range corresponding to the object component. Schematically, a region range marking operation corresponding to at least two object components in the global view is received, and the region range marking operation is used to mark the image regions corresponding to the at least two object components; based on the image region indicated by the region range marking operation, position distribution information corresponding to the at least two object components is generated.

[0138] In one example, a user uses a virtual brush tool provided by a terminal to draw an area range corresponding to each object component on a global view, and a computing device determines the location of each object component according to the area range drawn by the virtual brush tool.

[0139] That is, by indicating the position distribution of each object component through manual annotation, the segmentation accuracy when segmenting and obtaining the local views corresponding to the object components can be improved, thereby ensuring the accuracy of downstream three-dimensional model generation.

[0140] The second one is automatic detection.

[0141] In some embodiments, based on the image semantics of the global view, object components existing in the global view are identified, and position distribution information corresponding to at least two object components is obtained.

[0142] In some embodiments, the global view is input into a component recognition model, and position distribution information corresponding to at least two object components is output, wherein the component recognition model is a pre-trained model for identifying the positions corresponding to each object component in the global view.

[0143] Optionally, the above-mentioned component recognition model can be implemented by neural network models such as Convolutional Neural Networks (CNN), Feedforward Neural Network (FNN), Residual Network (ResNet), Transformer, etc., which are not specifically limited here.

[0144] In some embodiments, the training process of the component recognition model is implemented as follows: obtaining sample image data and sample position information, wherein the sample image data is a two-dimensional image to be recognized, and the sample position information is information for marking the distribution of each component in the sample image data; inputting the sample image data into the component recognition model to be trained to obtain predicted position information; determining a first loss value between the predicted position information and the sample position information through a first loss function; iteratively adjusting the model parameters of the component recognition model to be trained according to the first loss value until convergence to obtain a component recognition model.

[0145] Optionally, the first loss function may be implemented as at least one of a cross-entropy loss function, a mean squared error loss (MSE) function, a logarithmic loss function, a minimum absolute deviation loss (Least Absolute Deviations Loss, L1 Loss) function, etc., which is not limited here.

[0146] In some embodiments, in order to improve the recognition accuracy of the component recognition model, prompt words corresponding to at least two object components are input to the component recognition model while the global view is input.

[0147] Optionally, the prompt word includes at least one of name information, color information, volume information, and component sequence information corresponding to at least two object components. In one example, taking the prompt word including the name information corresponding to the object component as an example, Figure 1 The prompt words for the input component recognition model are "recognize the hat worn by the character, the clothes worn by the character, and the body torso of the character in the global view".

[0148] That is, the AI ​​model automatically identifies the components of each object in the first image under simple text prompts, and outputs the corresponding position distribution of each object component, which reduces the need for manual operation and improves the efficiency of overall model generation.

[0149] Step 222: input the global view and the position distribution information corresponding to the at least two object components into the region segmentation model to obtain the local views corresponding to the at least two object components.

[0150] In the embodiment of the present application, the region segmentation model is a pre-trained machine learning model for segmenting the regions where each object component is located from the global view. That is, the region segmentation model segments each region in the global view as a complete view to obtain local views corresponding to at least two object components.

[0151] Optionally, the above-mentioned region segmentation model can be implemented by a neural network model such as a convolutional neural network, a feedforward neural network, a residual network, a converter, etc., which is not specifically limited here.

[0152] In some embodiments, when the position distribution information is information obtained through manual annotation, after the global view and the position distribution information are input into the region segmentation model, the region segmentation model identifies the region corresponding to each object component in the global view based on the marking points corresponding to each object component in the position distribution information, and segments the image based on the recognition results to obtain the local view corresponding to each object component.

[0153] In other embodiments, when the position distribution information is obtained through identification of a component recognition model, the component recognition model and the region segmentation model can be implemented as different processing units of the same model, that is, the pre-trained AI model includes a component recognition unit and a region segmentation unit. After the prompt words corresponding to the global view and the object components are input into the above-mentioned AI model, the component recognition unit identifies in the global view the position distribution information corresponding to at least two object components whose image semantics match the text semantics corresponding to the prompt words, and the region segmentation unit segments the global view according to the position distribution information to obtain local views corresponding to at least two object components.

[0154] In some embodiments, the training process of the region segmentation model is implemented as follows: acquiring sample image data and sample annotation information, the sample image data including a sample pair consisting of a sample global view and a sample local view, the sample global view being a two-dimensional image for displaying a sample object, the sample local view being a two-dimensional image for displaying at least two sample components of the sample object, the sample annotation information being manually annotated and used to indicate the position distribution information of each sample component in the sample global view of the sample pair, inputting the sample global data and the sample annotation information into the region segmentation model to be trained to obtain a predicted local view, determining a second loss value between the predicted local view and the sample local view through a second loss function, iteratively adjusting the model parameters of the region segmentation model to be trained according to the first loss value until convergence, and obtaining a region segmentation model.

[0155] Optionally, the second loss function may be implemented as at least one of a cross entropy loss function, a mean square error loss function, a logarithmic loss function, a minimum absolute deviation loss function, and the like, which is not limited here.

[0156] In one example, taking the above-mentioned region segmentation model implemented as a visual segmentation model (Segment Anything Model, SAM) as an example, the image segmentation process of at least two object components is schematically explained.

[0157] SAM is a model that can achieve segmentation through "hints". Figure 6 As shown, it shows a model structure diagram of a visual segmentation model (SAM) 600 provided by an exemplary embodiment of the present application. SAM600 includes three modules: an image encoder (Image Encoder) 610, a prompt encoder (Prompt Encoder) 620 and a mask decoder (Mask Decoder) 630.

[0158] In the embodiment of the present application, the image encoder 610 is used to receive the input global view 601 and perform encoding on the global view 601, and output the image encoding representation (Image Embedding) of the global view 601. The prompt encoder 620 is used to receive the input prompt information 602 and perform encoding on the prompt information 602, and output the prompt encoding representation corresponding to the prompt information 602. The encoded image encoding representation and the prompt encoding representation are input into the mask decoder 630 together, and the mask decoder 630 converts the above two encoding representations into masks (Mask) 603 corresponding to at least two object components respectively, and outputs the local views corresponding to at least two object components respectively can be segmented based on the global view 601.

[0159] In the embodiment of the present application, the prompt information 602 is the acquired position distribution information corresponding to the at least two object components. Optionally, the prompt information 602 can be implemented as at least one of a mark point (point), a mark box (box), a description text (text) or an area range mask (mask) indicating the position distribution of the at least two object components.

[0160] In some embodiments, the image encoder 610 is implemented using a Vision Transformer (ViT) pre-trained using a Masked Autoencoders (MAE) learning method.

[0161] In some embodiments, a prompt encoder 620 of different structures is selected according to the type of prompt information input. Schematically, different prompt information can be divided into sparse prompts and dense prompts, wherein sparse prompts include markers, marker boxes and description texts, and dense prompts include area range masks. For sparse prompts, the prompt encoder 620 implements the encoding of markers and marker boxes through position encoding, and implements the encoding of description texts through text encoding; for dense prompts, the prompt encoder 620 implements the encoding of area range masks through convolutional layers.

[0162] In some embodiments, the mask decoder 630 uses a modified Transformer decoder block and a dynamic mask prediction head. The modified Transformer decoder block uses cue self-attention and cross-attention to update all feature representations in two directions (from cue encoding representation to image encoding representation and from image encoding representation to cue encoding representation). After running the two decoder blocks, the image embedding is upsampled and the upsampled result is mapped to a dynamic linear classifier through a multilayer perceptron (MLP), which then calculates the foreground mask probability of each position in the global view for each object component, and finally outputs the mask corresponding to each object component according to the foreground mask probability.

[0163] In some embodiments, the training process of SAM600 is implemented as follows: obtaining a sample global view, a sample mask and sample annotation information, wherein the sample global view is a two-dimensional image for displaying a sample object, the sample mask is a mask indicating the area where at least two sample components in the sample global view are located, and the sample annotation information is manually annotated for indicating the position distribution information of each sample component in the sample global view of the sample pair, the sample global view and the sample annotation information are input into the SAM to be trained, and the prediction masks corresponding to at least two sample components are obtained respectively, and the second loss value between the prediction mask and the sample mask is determined by a second loss function, and the model parameters of the SAM to be trained are iteratively adjusted according to the first loss value until convergence to obtain SAM600.

[0164] In some embodiments, in order to improve the model training effect of SAM600, during the training process, candidate masks with high confidence are detected from the prediction mask, and the candidate masks are filled into the sample global view to obtain a masked sample global view. Other unannotated parts in the masked sample global view are manually labeled to obtain supplementary labeling information, and the supplementary labeling information is used as sample labeling information to participate in the model training process.

[0165] In one example, the global view is implemented as three views of the first object as an example, Figure 7 As shown, it shows a schematic diagram of the segmentation of object components provided by an exemplary embodiment of the present application. After the three views 710 of the provided first object and the set of manually labeled marking points of each object component are input into SAM, the mask corresponding to each object component is obtained, and the three views 710 of the first object are segmented by the above-mentioned mask to obtain the three views 720 corresponding to each object component.

[0166] In summary, in the process of generating a three-dimensional model through a two-dimensional image, the global view of the first object is divided into local views corresponding to at least two object components, and the component three-dimensional models corresponding to each object component are generated through the local views, so that the object three-dimensional model of the first object is obtained by combining the component three-dimensional models of at least two object components. By decomposing the first object into at least two object components, the generation granularity of the object three-dimensional model is refined, and an independent model data is generated for each object component, which solves the problem of object component adhesion caused by all components belonging to the same model data in the object three-dimensional model of the first object finally obtained, so that the object three-dimensional model has higher editing flexibility in downstream applications. In the downstream application process, when the object three-dimensional model needs to be locally adjusted, it supports local editing by adjusting the model data corresponding to the component three-dimensional model, which reduces the workload of the object three-dimensional model in downstream applications.

[0167] In an embodiment of the present application, efficient and accurate image segmentation is achieved through the pre-trained region segmentation model combined with a small amount of manual annotation information or prompt text on the object components. By ensuring the accuracy of the local view as the segmentation result and the segmentation efficiency, the model accuracy of the subsequent three-dimensional model generation and the execution efficiency of the overall process are improved.

[0168] Step 2: Generate 3D models of components of the object.

[0169] In some optional embodiments, the process of converting the local view into the component three-dimensional model is implemented by a three-dimensional modeling model, please refer to Figure 8 , which shows a flowchart of a three-dimensional model data processing method provided by an exemplary embodiment of the present application, the process includes step 231, wherein step 231 is a subordinate step of step 230.

[0170] Step 231 , inputting the local views corresponding to at least two object components into the three-dimensional modeling model to obtain the component three-dimensional models corresponding to at least two object components.

[0171] In an embodiment of the present application, after obtaining the local views corresponding to the various object components of the first object, the local views are converted into a three-dimensional model of the component, wherein the three-dimensional model of the component is stored in the form of model data in a computer device. Exemplarily, the local views are converted into model data corresponding to the three-dimensional model of the component.

[0172] Model data is data used to define the attribute information of the three-dimensional model of the object component, so that the object component can be accurately rendered and displayed in computer graphics applications. The attribute information includes at least one of the geometric shape, appearance, and model behavior of the object component.

[0173] Optionally, the model data is implemented as at least one of mesh data, voxel data, and point cloud data.

[0174] In the embodiment of the present application, the 3D modeling model is a pre-trained model used to determine the corresponding component 3D model according to the local view and output the model data of the component 3D model. The input of the component 3D model is image data (i.e., local view and / or global view), and the output is the model data of the 3D model (e.g., mesh data, point cloud data).

[0175] In some embodiments, the training process of the three-dimensional modeling model is implemented as follows: obtaining sample view data and sample model data, wherein the sample view data is a two-dimensional image of a sample object to be subjected to three-dimensional model generation, and the sample model data is model data corresponding to the three-dimensional model of the sample object; the sample view data is converted to the three-dimensional modeling model to be trained to obtain predicted model data of the predicted three-dimensional model; a third loss value between the predicted model data and the sample model data is determined by a third loss function; model parameters of the three-dimensional modeling model to be trained are iteratively adjusted according to the third loss value until convergence to obtain a three-dimensional modeling model.

[0176] Optionally, the third loss function may be implemented as at least one of a cross entropy loss function, a mean square error loss function, a logarithmic loss function, a minimum absolute deviation loss function, and the like, which is not limited here.

[0177] Optionally, the above-mentioned three-dimensional modeling model is implemented as at least one of the Wonder3D model, a diffusion model based on Score Distillation Sampling (SDS), a CraftsMan model, a diffusion model based on the Make-It-3D framework (a framework for generating high-quality 3D models from a single image through diffusion priors), a Unique3D model, a Stable Video 3D (SV3D) model, etc., which is an artificial intelligence model that can convert a two-dimensional image into a three-dimensional model.

[0178] In one example, taking the above three-dimensional modeling model implemented as a CraftsMan model as an example, the generation process of the model data of at least two object components is schematically described, such as Fig. 9As shown, it shows a schematic diagram of a model structure provided by an exemplary embodiment of the present application. The CraftsMan model 900 is a model implemented based on a diffusion model. The CraftsMan model 900 includes an encoder 910, a U-Net network structure 920 and a decoder 930, wherein the U-Net network structure 920 includes a forward diffusion network 921 and a reverse diffusion network 922.

[0179] The encoder 910 and the decoder 930 are implemented as a residual network (Residual Network, ResNet). The encoder 910 compresses the image representation of the local view corresponding to the object component into a low-resolution image (downsampling process), and the decoder 930 decodes the low-resolution features into mesh data of the three-dimensional model of the component of the object (upsampling process).

[0180] The U-Net network structure 920 includes a multi-view (MV) diffusion model structure and a normal-based geometry refiner. The MV diffusion model is used to complete the first-stage generation process, that is, to generate a coarse mesh with smooth geometry based on the image features of the input local view. This process operates on the latent space of 3D representation learning based on the potential set, and can generate a rough geometry with a regular mesh topology in a short time; the normal-based geometry refiner is used to complete the second-stage generation process, that is, to refine the coarse mesh of the previous stage using the enhanced multi-view normal map generated by 2D normal diffusion, thereby enhancing the surface details and finally outputting the mesh data of the three-dimensional model of the component.

[0181] During the training process, the training sample data 901 is encoded and input into the CraftsMan model 900 to obtain predicted grid data. The CraftsMan model 900 is trained based on the effect of the predicted grid data, so that the CraftsMan model 900 learns the feature mapping transformation from the two-dimensional image to the three-dimensional model until the loss function converges to the optimal point, thereby obtaining the optimized CraftsMan model 900.

[0182] In the embodiment of the present application, in the process of generating the model data for the i-th object component, the input of the CraftsMan model 900 includes a local view 903 of the i-th object component, the local view 903 is encoded by the encoder 910 to obtain an image encoding representation, and the image encoding representation is input to the U-Net network structure 920. That is, the local view 903 is encoded by the encoder 910 to the latent space z init .

[0183] The function of the forward diffusion network 921 in the CraftsMan model 900 is to sample Gaussian noise from standard Gaussian noise. The feature representation of the sampled Gaussian noise and the image encoding representation obtained by encoding the local view 903 are input to the reverse diffusion network 922, and the 3D model feature representation is obtained by denoising through the reverse diffusion network 922. Schematically, the reverse diffusion network 922 performs iterative denoising with the 3D model of the component corresponding to the local view 903 as the denoising condition until the denoising is completed to obtain the 3D model feature representation. The 3D model feature representation is input to the decoder 930, and the mesh data 906 corresponding to the i-th object component is obtained by decoding by the decoder 930.

[0184] In some optional embodiments, the input of the CraftsMan model 900 further includes prompt words (not shown in the figure), and the prompt words are used to control the model generation effect when the CraftsMan model 900 generates a three-dimensional model based on the local view 903. When the input of the CraftsMan model 900 further includes prompt words, the encoder 910 further includes a text encoder that performs encoding on the prompt words.

[0185] Optionally, the prompt word includes at least one of the following contents: generation style prompt information, used to indicate the model style of the component three-dimensional model generated based on the local view 903, optionally, the generation style prompt information includes at least one of oil painting style, comic style, realistic style, photo style, digital art style, etc.; attribute prompt information, used to indicate the model attributes of the component three-dimensional model generated based on the local view 903, optionally, the above-mentioned model attributes include at least one of resolution, size, model data format, model size (storage space occupied), and model quality.

[0186] In some embodiments, in order to improve the generation effect of model data when generating model data based on a local view, image completion is performed on the local view before generating the model data. Schematically, the local views corresponding to at least two object components are respectively input into the image generation model to obtain a third view after the occluded part in the local view is completed; the third views corresponding to at least two object components are respectively input into the three-dimensional modeling model to obtain model data corresponding to at least two object components.

[0187] The image generation model is a machine learning model that is pre-trained to fill in the image content of the occluded area of ​​the object component in the local view. Optionally, the image generation model can be implemented by a neural network model such as a convolutional neural network, a feedforward neural network, a residual network, a converter, etc., which is not specifically limited here.

[0188] In some embodiments, the training process of the image generation model is implemented as follows: obtaining a sample incomplete view and a sample complete view, wherein the sample incomplete view is a two-dimensional image for displaying a sample object with incomplete areas, and the sample complete view is a two-dimensional image for displaying a complete sample object, and the sample incomplete view is converted to the image generation model to be trained to obtain a predicted complete view, and a fourth loss value between the predicted complete view and the sample complete view is determined by a fourth loss function, and the model parameters of the image generation model to be trained are iteratively adjusted according to the fourth loss value until convergence to obtain the image generation model.

[0189] In some embodiments, the sample incomplete view may be an image obtained by performing random masking on the sample complete view.

[0190] Optionally, the fourth loss function may be implemented as at least one of a cross entropy loss function, a mean square error loss function, a logarithmic loss function, a minimum absolute deviation loss function, and the like, which is not limited here.

[0191] In one example, an image generation model is implemented as a latent diffusion model (LDM). The latent diffusion model is a deep learning model for generating images. It generates images by diffusion in latent space. The latent diffusion model can learn image features by adding noise and denoising the image. Therefore, the latent diffusion model needs to add noise and denoise the image through a forward process and a reverse process. That is, the latent diffusion model performs noise processing on the image through a forward diffusion network, and performs denoising on the noisy image through a reverse diffusion network.

[0192] During the training process of the latent diffusion model, the training data of the model includes sample data pairs formed by occluded views and complete views of training samples, wherein the occluded views can be obtained by randomly masking the complete views.

[0193] like Fig.10 As shown, it shows a schematic diagram of the image completion process provided by an exemplary embodiment of the present application, where the local view 1001 corresponding to each object component obtained from the global view segmentation is input into the LDM 1000 to perform image completion, and obtain the third view 1002.

[0194] After the local view 1001 is input to the LDM 1000, the LDM 1000 performs a noise addition process on the local view 1001 to obtain a noise image. This process can be expressed as: After LDM1000 adds noise to the local view 1001, it performs iterative denoising on the noisy image with the goal of completing the object components in the local view 1001. The denoising process can be expressed as ,in, represents the standard Wiener Process (also known as Brownian motion), for Drift Coefficient, for The diffusion coefficient, for The image sample corresponding to the moment, Indicates time from Backflow to The standard Wiener process when for The data distribution, Represents the data distribution score.

[0195] In the noise addition process of LDM1000 on the local view 1001, the diffusion duration is a continuous time variable. ,exist The data distribution corresponding to the time can be expressed as ,exist The data distribution corresponding to the time can be expressed as , the diffusion time is used to indicate the number of steps required to generate the third view 1002 from the noise image in the iterative denoising process. In order to learn the image features, the computer device needs to sample the real samples through the discretization inverse process. .

[0196] In summary, in the process of generating a three-dimensional model through a two-dimensional image, the global view of the first object is divided into local views corresponding to at least two object components, and the component three-dimensional models corresponding to each object component are generated through the local views, so that the object three-dimensional model of the first object is obtained by combining the component three-dimensional models of at least two object components. By decomposing the first object into at least two object components, the generation granularity of the object three-dimensional model is refined, and an independent model data is generated for each object component, which solves the problem of object component adhesion caused by all components belonging to the same model data in the object three-dimensional model of the first object finally obtained, so that the object three-dimensional model has higher editing flexibility in downstream applications. In the downstream application process, when the object three-dimensional model needs to be locally adjusted, it supports local editing by adjusting the model data corresponding to the component three-dimensional model, which reduces the workload of the object three-dimensional model in downstream applications.

[0197] In an embodiment of the present application, the conversion process of converting a two-dimensional image into a three-dimensional model is implemented through an AI model such as the CraftsMan model, which reduces the need for manual modeling, reduces labor costs, improves the efficiency of generating three-dimensional models, and thereby improves the production efficiency of three-dimensional model-related businesses.

[0198] In an embodiment of the present application, a pre-trained image generation model is used to complete the parts of the object components in the local view obtained by segmentation that are occluded or omitted during segmentation, and then a three-dimensional model is generated based on the completed third view. Since the completed third view has more complete texture information and shape information of the object components, the generated three-dimensional model can be more accurate, thereby improving the model accuracy of the final object three-dimensional model.

[0199] Step 3: Assemble the 3D models of the object's components.

[0200] In some optional embodiments, when combining the component 3D models corresponding to at least two object components, the first position information of the two object components is determined by using the registration 3D model directly generated by the global view, so as to combine the component 3D models according to the first position information to obtain the object 3D model. Fig.11 , which shows a flowchart of a three-dimensional model data processing method provided by an exemplary embodiment of the present application, the process includes steps 241 to 243, wherein steps 241 to 243 are subordinate steps of step 240.

[0201] Step 241 : converting the global view into a registered three-dimensional model corresponding to the first object.

[0202] In an embodiment of the present application, the registered three-dimensional model is a three-dimensional model directly converted based on the global view of the first object. The registered three-dimensional model is used as a reference basis for splicing component three-dimensional models with each other during the process of combining component three-dimensional models, so as to ensure the accuracy of the component three-dimensional model combination process.

[0203] In some embodiments, the global view is input into the three-dimensional modeling model to obtain model data corresponding to the registered three-dimensional model, wherein the three-dimensional modeling model is a pre-trained model used to determine the corresponding registered three-dimensional model according to the global view and output the model data of the registered three-dimensional model.

[0204] Optionally, the above-mentioned three-dimensional modeling model is implemented as at least one of the Wonder3D model, a diffusion model based on Score Distillation Sampling (SDS), a CraftsMan model, a diffusion model based on the Make-It-3D framework (a framework for generating high-quality 3D models from a single image through diffusion priors), a Unique3D model, a Stable Video 3D (SV3D) model, etc., which is an artificial intelligence model that can convert a two-dimensional image into a three-dimensional model.

[0205] In the embodiment of the present application, the method of converting the global view into the registered three-dimensional model is the same as the method of converting the local view (or third view) of the object component into the component three-dimensional model, which will not be repeated here.

[0206] Step 242: determining first position information of at least two object components according to a mapping relationship between the component three-dimensional models corresponding to the at least two object components and the registered three-dimensional model.

[0207] In the embodiment of the present application, since there is a 2D component-3D component mapping relationship between the object component in the global view and the component three-dimensional model, there is a 2D component-2D whole mapping relationship between the object component in the global view and the object whole corresponding to the first object, and there is a 2D whole-3D whole mapping relationship between the object whole corresponding to the first object in the global view and the registered three-dimensional model, therefore, by combining the mapping chain formed by the above three mapping relationships, it is possible to obtain a 3D component-3D whole mapping relationship between the component three-dimensional model of the object component and the registered three-dimensional model of the object whole representing the first object.

[0208] Illustratively, a first mapping relationship is established between component three-dimensional models corresponding to at least two object components and a first registration area in a global view, wherein the first registration area is used to indicate areas where the at least two object components are located in the global view; a second mapping relationship is established between the first registration areas corresponding to at least two object components in the global view and a second registration area of ​​the first object, wherein the second registration area is used to indicate an area where the first object is located in the global view; a third mapping relationship is established between the second registration area of ​​the first object in the global view and the registration three-dimensional model; a fourth mapping relationship is determined between the component three-dimensional models corresponding to the at least two object components and the registration three-dimensional model based on a mapping chain formed by the first mapping relationship, the second mapping relationship and the third mapping relationship; and first position information of the at least two object components is determined based on the fourth mapping relationship. That is, after obtaining the first mapping relationship, the second mapping relationship and the third mapping relationship, the mapping relationship between the component three-dimensional model and the second registration area can be obtained through the first mapping relationship and the second mapping relationship, and the mapping relationship between the component three-dimensional model and the registration three-dimensional model can be obtained through the mapping relationship between the component three-dimensional model and the second registration area and the third mapping relationship, that is, the fourth mapping relationship is obtained, thereby realizing the establishment of a mapping relationship between the component three-dimensional model and the registration three-dimensional model in three-dimensional space through a two-dimensional image.

[0209] In one example, the fourth mapping relationship is used as the first position information indicating the position of at least two object components in the three-dimensional space.

[0210] For example, Fig.12 As shown, it shows a schematic diagram of the combination process of component three-dimensional models provided by an exemplary embodiment of the present application, the global view 1201 generates a registration three-dimensional model 1202 through a three-dimensional modeling model 1210, and the local view 1203 corresponding to each object component generates a component three-dimensional model 1204 corresponding to each object component through the three-dimensional modeling model 1210, and a mapping relationship 1205 from the midpoint of the component three-dimensional model 1204 to the midpoint of the registration three-dimensional model 1202 is obtained through the above-mentioned mapping chain, and the first position information of the object component is indicated by the mapping relationship 1205.

[0211] In some embodiments, since there may be some erroneous mapping relationships in the actual calculation process, these erroneous mapping relationships may have a negative impact on the final combination result. Therefore, the fourth mapping relationship is screened to ensure the accuracy of the mapping relationship used in the combination. Schematically, the fourth mapping relationship includes at least two first point pair relationships corresponding to at least two object components, wherein the first point pair relationship is used to indicate the mapping relationship of the mapping point pair formed by the first position point in the component three-dimensional model and the second position point in the registration three-dimensional model; according to the mapping accuracy of the at least two first point pair relationships corresponding to the i-th object component, at least two first point pair relationships are screened to obtain the second point pair relationship, where i is a positive integer; and the first position information is determined according to the second point pair relationships corresponding to the at least two object components.

[0212] In some embodiments, a Random Sample Consensus (RANSAC) algorithm is used to remove incorrect mapping relationships. Schematically, at least one third point pair relationship is sampled from at least two first point pair relationships corresponding to the i-th object component; at least one third point pair relationship is used as a reference transformation relationship to determine the mapping accuracy corresponding to at least two first point pair relationships; according to the mapping accuracy corresponding to at least two first point pair relationships, at least two first point pair relationships are screened based on a threshold to obtain a second point pair relationship.

[0213] In some embodiments, when screening at least two first point pair relationships, first point pair relationships with mapping accuracy higher than or equal to a threshold are retained, first point pair relationships with mapping accuracy lower than the threshold are screened out, and the retained first point pair relationship is determined as the second point pair relationship.

[0214] Exemplarily, the above threshold is a pre-configured threshold.

[0215] For example, a certain number of mapping point pairs are randomly sampled and their mapping relationships are calculated, and then all mapping point pairs are applied with the mapping relationship, and the number of mapping point pairs within a certain error range is counted, and this process is repeated multiple times, and the mapping relationship with the best fit is selected as the final result. In this way, the wrong mapping relationship can be effectively eliminated and the accuracy of the three-dimensional model of the combined component can be improved.

[0216] In some embodiments, the first position information corresponding to at least two object components is composed of second point pair relationships corresponding to at least two object components, that is, the combination of component three-dimensional models can be achieved through the above second point pair relationships.

[0217] Step 243 , combining the component three-dimensional models corresponding to at least two object components respectively according to the first position information to obtain an object three-dimensional model of the first object.

[0218] In an embodiment of the present application, the first position information includes a second point pair relationship corresponding to at least two object components. When the component three-dimensional models corresponding to the at least two object components are combined according to the first position information, the registered three-dimensional model is used as a reference, and the position of the component three-dimensional model in the registered three-dimensional model is determined by the second point pair relationship. That is, the component three-dimensional models of the at least two object components are sequentially combined onto the registered three-dimensional model according to the positions in the registered three-dimensional model indicated by the second point pair relationship, thereby realizing the combination process of the component three-dimensional models.

[0219] In some embodiments, at least two object components are aligned with the three-dimensional model in a first alignment mode based on the first position information to obtain alignment results corresponding to the at least two object components respectively; based on the alignment results, the three-dimensional models of the components corresponding to the at least two object components are combined to obtain the object three-dimensional model of the first object.

[0220] Optionally, the above-mentioned first registration mode can be implemented as a rigid registration (Rigid Registration) mode, a fast point feature histogram (Fast Point Feature Histograms, FPFH) mode, a directional histogram feature descriptor (Signature of Histograms of OrienTations, SHOT), an iterative closest point (Iterative Closest Point, ICP) algorithm mode, a normal distribution transform (Normal Distributions Transform, NDT) algorithm mode, etc.

[0221] In one example, the first registration mode is implemented as a rigid registration mode, that is, the combination process of the component three-dimensional models of at least two object components is implemented by rigid registration. Schematically, according to the first position information, the at least two object components are respectively rigidly registered with the registration three-dimensional model to obtain the registration results corresponding to the at least two object components; according to the registration results, the model data corresponding to the at least two object components are combined to obtain the object three-dimensional model of the first object.

[0222] Among them, rigid registration is a technology applied in 3D point clouds or meshes. Its goal is to find an optimal rigid transformation so that a point (source) can coincide with another point (target) as much as possible after this transformation. Mathematically, rigid registration can be expressed as minimizing a metric function that measures the difference between the transformed point cloud and the target point cloud. Rigid transformations include rotations and translations, but not scaling or deformation, so all vertices share a transformation matrix.

[0223] In one example, the points (source) of the object components in the component 3D model generated by the local view and the points (target) in the registered 3D model are made to coincide as much as possible by minimizing the objective function, wherein the objective function is shown in Formula 1.

[0224] Formula 1: .

[0225] in, is the rotation matrix, is the translation vector, is a point in the 3D model of a component generated by a local view. is the registration of the 3D model with The corresponding points.

[0226] In one example, the Procrustes analysis method is used to perform rigid registration on each object component, thereby achieving the combination of component three-dimensional models. Procrustes analysis is a method for achieving optimal alignment between objects in two-dimensional or three-dimensional space. Specifically, the method minimizes the distance between one shape and another shape by calculating the optimal rotation, translation and scaling transformations. Schematically, the Procrustes distance between the component three-dimensional model of the object component generated by the local view and the model area corresponding to the object component in the registered three-dimensional model is obtained, and the position and size of the component three-dimensional model of the object component generated by the local view are adjusted according to the Procrustes distance so that it overlaps with the model area corresponding to the object component in the registered three-dimensional model, thereby achieving the combination process of each object component.

[0227] In the application of the embodiment of the present application, the Procrustes analysis method is used, combined with the fourth mapping relationship between the component three-dimensional model of the known object components and the registered three-dimensional model of the first object, it is possible to quickly perform rigid registration on each object component and the registered three-dimensional model, thereby achieving a combination of at least two object components.

[0228] When combining the 3D models of components of an object, rigid registration can ensure that the shape and size of the object indicated by the model data of the 3D model of the component remain unchanged during the combination process, and ensure that the actual geometric characteristics of the 3D model of the component will not be affected during the combination process, thereby improving the accuracy of the model combination process.

[0229] In some embodiments, before performing rigid registration, the model data corresponding to each object component is also registered through a second registration mode. Schematically, according to the positional relationship between at least two object components indicated by the first position information, the model data of the three-dimensional model of the component is registered with the second registration mode to obtain registration model data corresponding to at least two object components, and the registration model data is used to indicate the model data obtained after the deformation of the three-dimensional model of the component is completed through the second registration mode; according to the registration result, the registration model data corresponding to at least two object components are combined to obtain the object three-dimensional model of the first object.

[0230] Optionally, the second registration mode includes at least one of a non-rigid deformation processing mode, a deep learning-based deformable registration network mode, a diffusion model-based unsupervised deformable registration mode, a physical constraint-based deformable registration mode, and the like.

[0231] In one example, taking the implementation of the second registration mode as non-rigid registration as an example, schematically, based on the positional relationship between at least two object components indicated by the first position information, non-rigid registration is performed on the model data of the at least two object components to obtain registration model data corresponding to the at least two object components, wherein the registration model data is used to indicate the model data obtained after non-rigid deformation of the three-dimensional model of the component is completed by non-rigid registration; and based on the registration results, the registration model data corresponding to the at least two object components are combined to obtain the object three-dimensional model of the first object.

[0232] That is, in order to solve the problem of penetration and connection between at least two object components, it is necessary to perform non-rigid deformation processing on the components. Non-rigid deformation refers to modifying the shape of an object while keeping certain properties of the object unchanged. In the process of combining the three-dimensional models of the components of at least two object components, the adjustment of the model data through non-rigid deformation can make the connection between the at least two object components more natural and avoid the occurrence of holes and overlaps.

[0233] In an illustrative manner, after obtaining the registration result, when combining the model data corresponding to each object component according to the registration result, the model representation unit in the model data of each object component is combined to the three-dimensional space where the registered three-dimensional model is located according to the mapping relationship indicated by the registration result, thereby realizing the model data combination process. For example, taking the model data as mesh data, the mesh data includes meshes (including vertices, edges and faces) constituting the three-dimensional model of the component, and taking the registration result as an example of the mapping relationship between the vertices of the mesh, according to the mapping relationship indicated by the registration result, the first vertex in the three-dimensional model of the component is configured to the position of the second vertex in the registered three-dimensional model that has a mapping relationship with the first vertex, thereby realizing the combination process of the model data of each object component based on the registered three-dimensional model.

[0234] Optionally, non-rigid deformation can be achieved through a variety of methods, such as physics-based simulation, energy minimization-based optimization methods, etc.

[0235] For example, Fig.12 As shown, after the mapping relationship 1205 is obtained, the object three-dimensional model 1206 corresponding to the first object is finally obtained by combining the rigid registration 1220 and the non-rigid registration 1230 .

[0236] In summary, in the process of generating a three-dimensional model through a two-dimensional image, the global view of the first object is divided into local views corresponding to at least two object components, and the component three-dimensional models corresponding to each object component are generated through the local views, so that the object three-dimensional model of the first object is obtained by combining the component three-dimensional models of at least two object components. By decomposing the first object into at least two object components, the generation granularity of the object three-dimensional model is refined, and an independent model data is generated for each object component, which solves the problem of object component adhesion caused by all components belonging to the same model data in the object three-dimensional model of the first object finally obtained, so that the object three-dimensional model has higher editing flexibility in downstream applications. In the downstream application process, when the object three-dimensional model needs to be locally adjusted, it supports local editing by adjusting the model data corresponding to the component three-dimensional model, which reduces the workload of the object three-dimensional model in downstream applications.

[0237] In an embodiment of the present application, the global view is directly converted into a three-dimensional model of the first object, and the three-dimensional model is used as a registration three-dimensional model to assist in establishing the position information of the object components in the three-dimensional space, thereby improving the combination efficiency of the model data of the object components under the guidance of the registration three-dimensional model.

[0238] In an embodiment of the present application, a mapping chain of 2D component-3D component, 2D component-2D entity, and 2D entity-3D entity is established by aligning the three-dimensional model, so that the mapping relationship between the 3D component and the 3D entity is known through the mapping chain, and the combination of the object components is realized through the mapping relationship, thereby improving the combination efficiency when combining the three-dimensional models of the component parts of the object.

[0239] The three-dimensional model data processing method provided in the embodiment of the present application decomposes the first object into at least two object components so that the generation granularity of the object three-dimensional model is refined, and an independent model data is generated for each object component, thereby solving the problem of object component adhesion caused by all components in the object three-dimensional model of the first object finally obtained belonging to the same model data, so that the object three-dimensional model has higher editing flexibility in downstream applications. During the downstream application process, when the object three-dimensional model needs to be locally adjusted, it supports local editing by adjusting the model data corresponding to the object components, thereby reducing the workload of the object three-dimensional model in downstream applications.

[0240] For example, Fig.13 As shown, it shows a schematic diagram comparing the effects of directly generating a three-dimensional model through a CraftsMan model and generating a three-dimensional model through the method of an embodiment of the present application.

[0241] In the generated result 1310 directly obtained through the CraftsMan model, since the global view of the first object is directly input into the CraftsMan model, the mesh data of the first object is output through the CraftsMan model, and the three-dimensional model of the first object is indicated by the mesh data, in the generated result 1310, since the three-dimensional models of all object components in the first object are represented by the same mesh, there is an adhesion problem between the object components.

[0242] In the generation result 1320 of the first object generated by the method provided in the embodiment of the present application, since the model data corresponding to each object component is independently generated, the three-dimensional model of the first object is obtained by combining the model data of the independently generated three-dimensional models of the components of each object component. Therefore, the problem of adhesion between the object components is solved, which is more conducive to the needs of model driving of the three-dimensional model in the downstream application process, collision between object components and other physical simulation needs.

[0243] It should be noted that this application can display a prompt interface, pop-up window or output voice prompt information before collecting relevant data of the user and during the process of collecting relevant data of the user. The prompt interface, pop-up window or voice prompt information is used to prompt the user that its relevant data is currently being collected, so that this application only starts to execute the relevant steps of obtaining relevant data of the user after obtaining the confirmation operation issued by the user on the prompt interface or pop-up window, otherwise (that is, when the confirmation operation issued by the user on the prompt interface or pop-up window is not obtained), the relevant steps of obtaining relevant data of the user are terminated, that is, the relevant data of the user is not obtained. In other words, all user data collected by this application are collected with the consent and authorization of the user, and the collection, use and processing of relevant user data need to comply with the relevant laws, regulations and standards of relevant countries and regions.

[0244] Please refer to Fig.14 , which shows a structural block diagram of a three-dimensional model data processing device provided by an exemplary embodiment of the present application, and the device includes the following modules.

[0245] An acquisition module 1410 is configured to acquire a global view of a first object, wherein the global view includes a two-dimensional image showing the first object from a first viewing direction, wherein the first object includes at least two object components;

[0246] A segmentation module 1420, configured to segment from the global view to obtain local views corresponding to the at least two object components, the local views comprising two-dimensional images showing the corresponding at least two object components from the first viewing angle;

[0247] A generating module 1430, configured to convert the local views corresponding to the at least two object components into a component three-dimensional model;

[0248] The combining module 1440 is used to combine the component three-dimensional models corresponding to the at least two object components to obtain the object three-dimensional model of the first object.

[0249] In some optional embodiments, such as Fig.15 As shown, the segmentation module 1420 includes:

[0250] A first acquiring unit 1421 is configured to acquire position distribution information of the at least two object components in the global view, where the position distribution information is used to indicate distribution of the at least two object components in the global view;

[0251] The segmentation unit 1422 is configured to segment the first object in the global view based on the position distribution information to obtain local views corresponding to the at least two object components.

[0252] In some optional embodiments, the first acquisition unit 1421 is further configured to identify object components existing in the global view based on the image semantics of the global view, and obtain position distribution information corresponding to the at least two object components respectively;

[0253] The first acquisition unit 1421 is also used to receive marking operations corresponding to the at least two object components in the global view, and the marking operation is used to mark at least one marking point corresponding to the at least two object components; based on the marking point indicated by the marking operation, generate position distribution information corresponding to the at least two object components.

[0254] In some optional embodiments, the segmentation unit 1422 is further used to input the position distribution information corresponding to the global view and the at least two object components into a regional segmentation model to obtain local views corresponding to the at least two object components. The regional segmentation model is a pre-trained machine learning model for segmenting the area where each object component is located from the global view.

[0255] In some optional embodiments, the generation module 1430 is also used to input the local views corresponding to the at least two object components into the three-dimensional modeling model to obtain the three-dimensional models of the components corresponding to the at least two object components, and the three-dimensional modeling model is a pre-trained machine learning model for outputting the three-dimensional model of the component according to the local views.

[0256] In some optional embodiments, the device further comprises:

[0257] A completion module 1450 is used to input the local views corresponding to the at least two object components into an image generation model to obtain a third view after the occluded part in the local view is completed, wherein the image generation model is a machine learning model pre-trained to fill the image content of the occluded area of ​​the object component in the local view;

[0258] The generating module 1430 is further configured to input the third views respectively corresponding to the at least two object components into the three-dimensional modeling model to obtain the component three-dimensional models respectively corresponding to the at least two object components.

[0259] In some optional embodiments, the combining module 1440 includes:

[0260] A second acquisition unit 1441 is used to acquire first position information of the at least two object components in the three-dimensional space, where the first position information is used to indicate the positions of the at least two object components in the three-dimensional space when they constitute the first object.

[0261] The combining unit 1442 is configured to combine the component three-dimensional models corresponding to the at least two object components according to the first position information to obtain the object three-dimensional model of the first object.

[0262] In some optional embodiments, the generating module 1430 is further configured to convert the global view into a registered three-dimensional model corresponding to the first object;

[0263] The second acquisition unit 1441 is further configured to determine the first position information of the at least two object components according to a mapping relationship between the component three-dimensional models respectively corresponding to the at least two object components and the registered three-dimensional model.

[0264] In some optional embodiments, the combining module 1440 further includes:

[0265] A mapping unit 1443 is used to establish a first mapping relationship between the component three-dimensional models corresponding to the at least two object components and a first registration area in the global view, wherein the first registration area is used to indicate the areas where the at least two object components are located in the global view;

[0266] The mapping unit 1443 is further configured to establish a second mapping relationship between first registration regions corresponding to the at least two object components in the global view and a second registration region of the first object, wherein the second registration region is used to indicate a region where the first object is located in the global view;

[0267] The mapping unit 1443 is further configured to establish a third mapping relationship between the second registration region of the first object in the global view and the registered three-dimensional model;

[0268] The mapping unit 1443 is further configured to determine a fourth mapping relationship between the component three-dimensional models respectively corresponding to the at least two object components and the registered three-dimensional model according to a mapping chain formed by the first mapping relationship, the second mapping relationship and the third mapping relationship;

[0269] The second acquiring unit 1441 is further configured to determine the first position information of the at least two object components according to the fourth mapping relationship.

[0270] In some optional embodiments, the fourth mapping relationship includes at least two first point pair relationships corresponding to the at least two object components, respectively, and the first point pair relationship is used to indicate a mapping relationship between a mapping point pair formed by a first position point in the component three-dimensional model and a second position point in the registration three-dimensional model;

[0271] The second acquisition unit 1441 is further configured to filter the at least two first point pair relationships according to the mapping accuracy of the at least two first point pair relationships corresponding to the i-th object component to obtain a second point pair relationship, where i is a positive integer;

[0272] The second acquiring unit 1441 is further configured to determine the first position information according to second point pair relationships respectively corresponding to the at least two object components.

[0273] In some optional embodiments, the second acquisition unit 1441 is further configured to obtain at least one third point pair relationship by sampling from the at least two first point pair relationships corresponding to the i-th object component;

[0274] The second acquisition unit 1441 is further configured to use the at least one third point pair relationship as a reference transformation relationship to determine mapping accuracies corresponding to the at least two first point pair relationships respectively;

[0275] The second acquisition unit 1441 is further configured to filter the at least two first point pair relationships based on a threshold according to the mapping accuracies respectively corresponding to the at least two first point pair relationships, so as to obtain the second point pair relationship.

[0276] In some optional embodiments, the combining module 1440 further includes:

[0277] A registration unit 1444 is configured to register the at least two object components with the registration three-dimensional model in a first registration mode based on the first position information to obtain registration results corresponding to the at least two object components respectively;

[0278] The combining unit 1442 is further configured to combine the component three-dimensional models corresponding to the at least two object components respectively based on the registration result to obtain the object three-dimensional model of the first object.

[0279] In some optional embodiments, the registration unit 1444 is further used to register the model data of the component three-dimensional model in a second registration mode according to the positional relationship between the at least two object components indicated by the first position information, to obtain registration model data corresponding to the at least two object components, respectively, the registration model data being used to indicate the model data obtained after the deformation of the component three-dimensional model is completed by the second registration mode;

[0280] The combining unit 1442 is further configured to combine the registration model data corresponding to the at least two object components respectively according to the registration result to obtain the three-dimensional object model of the first object.

[0281] It should be noted that the three-dimensional model data processing device provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the three-dimensional model data processing device provided in the above embodiment and the three-dimensional model data processing method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0282] The present application also provides a computer device, which includes a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the three-dimensional model data processing method provided by each of the above method embodiments. It should be noted that the computer device can be as follows Fig.16 Computer equipment provided.

[0283] Please refer to Fig.16 , which shows a schematic diagram of the structure of a computer device provided by an exemplary embodiment of the present application. Specifically, the computer device 1600 includes a central processing unit (CPU) 1601, a system memory 1604 including a random access memory (RAM) 1602 and a read-only memory (ROM) 1603, and a system bus 1605 connecting the system memory 1604 and the central processing unit 1601. The computer device 1600 also includes a basic input / output system (I / O system) 1606 that helps transmit information between various devices in the computer, and a large-capacity storage device 1607 for storing an operating system 1613, an application program 1614, and other program modules 1615.

[0284] The basic input / output system 1606 includes a display 1608 for displaying information and an input device 1609 such as a mouse and a keyboard for user inputting information. The display 1608 and the input device 1609 are connected to the central processing unit 1601 through an input / output controller 1610 connected to the system bus 1605. The basic input / output system 1606 may also include an input / output controller 1610 for receiving and processing inputs from a plurality of other devices such as a keyboard, a mouse, or an electronic stylus. Similarly, the input / output controller 1610 also provides output to a display screen, a printer, or other types of output devices.

[0285] The mass storage device 1607 is connected to the central processing unit 1601 via a mass storage controller (not shown) connected to the system bus 1605. The mass storage device 1607 and its associated computer readable medium provide non-volatile storage for the computer device 1600. That is, the mass storage device 1607 may include a computer readable medium (not shown) such as a hard disk or a Compact Disc Read-Only Memory (CD-ROM) drive.

[0286] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), flash memory or other solid-state storage technology, CD-ROM, Digital Versatile Disc (DVD) or other optical storage, cassettes, tapes, disk storage or other magnetic storage devices. Of course, those skilled in the art will know that the computer storage medium is not limited to the above. The above-mentioned system memory 1604 and mass storage device 1607 can be collectively referred to as memory.

[0287] The memory stores one or more programs, and the one or more programs are configured to be executed by one or more central processing units 1601. The one or more programs include instructions for implementing the above-mentioned three-dimensional model data processing method or the translation method based on the machine translation model. The central processing unit 1601 executes the one or more programs to implement the three-dimensional model data processing method or the translation method based on the machine translation model provided by the above-mentioned various method embodiments.

[0288] According to various embodiments of the present invention, the computer device 1600 can also be connected to a remote computer on the network through a network such as the Internet. That is, the computer device 1600 can be connected to the network 1612 through the network interface unit 1611 connected to the system bus 1605, or the network interface unit 1611 can be used to connect to other types of networks or remote computer systems (not shown).

[0289] The memory also includes one or more programs, which are stored in the memory and include steps executed by a computer device in the three-dimensional model data processing method provided in an embodiment of the present invention.

[0290] An embodiment of the present application also provides a computer device, which includes a memory and a processor, wherein the memory stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded by the processor to implement the above-mentioned three-dimensional model data processing method.

[0291] An embodiment of the present application also provides a computer-readable storage medium, which stores at least one instruction, at least one program, a code set or an instruction set. The at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by the processor to implement the above-mentioned three-dimensional model data processing method.

[0292] The present application also provides a computer program product. When the computer program product is run on a computer, the computer executes the three-dimensional model data processing methods provided by the above-mentioned various method embodiments.

[0293] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium, which can be a computer-readable storage medium contained in the memory in the above embodiments; or it can be a computer-readable storage medium that exists alone and is not assembled in the terminal. The computer-readable storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by the processor to implement the above three-dimensional model data processing method or the translation method based on the machine translation model. Optionally, the computer-readable storage medium may include: a read-only memory (ROM, ReadOnly Memory), a random access memory (RAM, Random Access Memory), a solid state drive (SSD, SolidState Drives) or an optical disk, etc. Among them, the random access memory may include a resistance random access memory (ReRAM, Resistance Random Access Memory) and a dynamic random access memory (DRAM, Dynamic RandomAccess Memory). The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0294] Those skilled in the art can understand that all or part of the steps of the above embodiments can be completed by hardware, or by a program to instruct the relevant hardware to complete, and the program can be stored in a computer-readable storage medium, and the above-mentioned storage medium can be a read-only memory, a disk or an optical disk, etc. The above description is only a preferred embodiment of the present application, and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A three-dimensional model data processing method, characterized in that: The method comprises: Acquire a global view of a first object, the global view comprising a two-dimensional image showing the first object from a first viewing direction, the first object comprising at least two object components; Segmenting the at least two object components from the global view to obtain local views respectively corresponding to the at least two object components, the local views comprising two-dimensional images showing the corresponding object components from the first viewing direction, wherein the i-th local view comprises a two-dimensional image showing the i-th object component from the first viewing direction; Converting the local views corresponding to the at least two object components into component three-dimensional models; converting the global view into a registered three-dimensional model corresponding to the first object; Determine first position information of the at least two object components according to a mapping relationship between the component three-dimensional models respectively corresponding to the at least two object components and the registered three-dimensional model; The three-dimensional model of the components corresponding to the at least two object components is combined according to the first position information to obtain a three-dimensional object model of the first object.

2. The method according to claim 1, characterized in that The step of segmenting the global view to obtain the local views corresponding to the at least two object components respectively comprises: Acquire position distribution information of the at least two object components in the global view, where the position distribution information is used to indicate distribution of the at least two object components in the global view; The first object in the global view is segmented based on the position distribution information to obtain local views corresponding to the at least two object components.

3. The method according to claim 2, characterized in that The obtaining of position distribution information of the at least two object components in the global view includes: Based on the image semantics of the global view, identifying the object components existing in the global view, and obtaining the position distribution information corresponding to the at least two object components respectively; or Receive marking operations corresponding to the at least two object components in the global view, wherein the marking operations are used to mark at least one marking point corresponding to the at least two object components; and generate position distribution information corresponding to the at least two object components based on the marking points indicated by the marking operations.

4. The method according to claim 2 or 3, characterized in that: The step of segmenting the first object in the global view based on the position distribution information to obtain local views corresponding to the at least two object components respectively includes: Inputting the global view and the position distribution information respectively corresponding to the at least two object components into a region segmentation model to obtain the local views respectively corresponding to the at least two object components; The region segmentation model is a pre-trained machine learning model used to segment the regions where the components of each object are located from the global view.

5. The method according to any one of claims 1 to 3, characterized in that: The converting of the local views corresponding to the at least two object components into a component three-dimensional model comprises: The local views corresponding to the at least two object components are input into a three-dimensional modeling model to obtain the three-dimensional models of the components corresponding to the at least two object components. The three-dimensional modeling model is a pre-trained machine learning model for outputting the three-dimensional models of the components according to the local views.

6. The method according to claim 5, characterized in that The step of inputting the local views corresponding to the at least two object components into a three-dimensional modeling model to obtain the component three-dimensional models corresponding to the at least two object components comprises: Inputting the local views corresponding to the at least two object components into an image generation model to obtain a third view after the occluded portion in the local view is completed, wherein the image generation model is a machine learning model pre-trained to fill the image content of the occluded area of ​​the object component in the local view; The third views respectively corresponding to the at least two object components are input into the three-dimensional modeling model to obtain the component three-dimensional models respectively corresponding to the at least two object components.

7. The method according to any one of claims 1 to 3, characterized in that: The determining the first position information of the at least two object components according to the mapping relationship between the component three-dimensional models respectively corresponding to the at least two object components and the registered three-dimensional model comprises: Establishing a first mapping relationship between the component three-dimensional models corresponding to the at least two object components and a first registration area in the global view, wherein the first registration area is used to indicate the areas where the at least two object components are located in the global view; Establishing a second mapping relationship between first registration regions corresponding to the at least two object components in the global view and a second registration region of the first object, wherein the second registration region is used to indicate a region where the first object is located in the global view; establishing a third mapping relationship between the second registration region of the first object in the global view and the registered three-dimensional model; Determine, according to a mapping chain formed by the first mapping relationship, the second mapping relationship and the third mapping relationship, a fourth mapping relationship between the component three-dimensional models respectively corresponding to the at least two object components and the registered three-dimensional model; The first position information of the at least two object components is determined according to the fourth mapping relationship.

8. The method according to claim 7, characterized in that The fourth mapping relationship includes at least two first point pair relationships corresponding to the at least two object components, respectively, wherein the first point pair relationship is used to indicate a mapping relationship between a mapping point pair formed by a first position point in the component three-dimensional model and a second position point in the registration three-dimensional model; The determining the first position information of the at least two object components according to the fourth mapping relationship includes: According to the mapping accuracy of the at least two first point pair relationships corresponding to the i-th object component, the at least two first point pair relationships are screened to obtain a second point pair relationship, where i is a positive integer; The first position information is determined according to the second point pair relationships respectively corresponding to the at least two object components.

9. The method according to claim 8, characterized in that The step of selecting the at least two first point pair relationships according to the mapping accuracy of the at least two first point pair relationships corresponding to the i-th object component to obtain the second point pair relationship includes: Sampling at least one third point pair relationship from the at least two first point pair relationships corresponding to the i-th object component; Taking the at least one third point pair relationship as a reference transformation relationship, determining the mapping accuracies respectively corresponding to the at least two first point pair relationships; According to the mapping accuracies respectively corresponding to the at least two first point pair relationships, the at least two first point pair relationships are screened based on a threshold to obtain the second point pair relationship.

10. The method according to claim 6, characterized in that The step of combining the component three-dimensional models corresponding to the at least two object components respectively according to the first position information to obtain the object three-dimensional model of the first object includes: Based on the first position information, register the at least two object components with the registered three-dimensional model in a first registration mode to obtain registration results corresponding to the at least two object components respectively; The component three-dimensional models corresponding to the at least two object components are combined based on the registration result to obtain an object three-dimensional model of the first object.

11. The method according to claim 10, characterized in that Combining the component three-dimensional models corresponding to the at least two object components respectively based on the registration result to obtain the object three-dimensional model of the first object includes: According to the positional relationship between the at least two object components indicated by the first position information, register the model data of the component three-dimensional model in a second registration mode to obtain registration model data corresponding to the at least two object components, wherein the registration model data is used to indicate the model data obtained after the deformation of the component three-dimensional model is completed by the second registration mode; The registration model data corresponding to the at least two object components are combined according to the registration result to obtain a three-dimensional object model of the first object.

12. A three-dimensional model data processing device, characterized in that: The device comprises: an acquisition module, configured to acquire a global view of a first object, wherein the global view comprises a two-dimensional image showing the first object from a first viewing direction, wherein the first object comprises at least two object components; a segmentation module, configured to segment from the global view to obtain local views corresponding to the at least two object components, the local views comprising two-dimensional images showing the corresponding object components from the first viewing direction, wherein the i-th local view comprises a two-dimensional image showing the i-th object component from the first viewing direction; A generating module, used for converting the local views corresponding to the at least two object components into a component three-dimensional model; The generating module is further used to convert the global view into a registered three-dimensional model corresponding to the first object; A second acquisition unit, configured to determine first position information of the at least two object components according to a mapping relationship between the component three-dimensional models respectively corresponding to the at least two object components and the registered three-dimensional model; A combining unit is used to combine the component three-dimensional models corresponding to the at least two object components according to the first position information to obtain the object three-dimensional model of the first object.

13. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one program, and the at least one program is loaded and executed by the processor to implement the three-dimensional model data processing method as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one program code, and the program code is loaded and executed by a processor to implement the three-dimensional model data processing method according to any one of claims 1 to 11.

15. A computer program product, characterized in that It comprises a computer program or an instruction, which, when executed by a processor, implements the three-dimensional model data processing method as described in any one of claims 1 to 11.

Images