Digital asset generation method, computer readable storage medium and intelligent equipment

Abstract

The invention relates to the technical field of image processing, particularly provides a digital asset generation method, a computer readable storage medium and intelligent equipment, and aims to solve the problems that an existing digital asset generation method is not convenient enough and is poor in accuracy. In order to achieve the purpose, the digital asset generation method comprises the steps of obtaining a plurality of images of a target object under a plurality of view angles, and obtaining an initial three-dimensional color point cloud of the target object based on the plurality of images; when the point cloud sparse region exists in the initial three-dimensional color point cloud, projecting the initial three-dimensional color point cloud to a target two-dimensional plane to obtain a re-projected two-dimensional image; inputting the re-projected two-dimensional image into an image restoration model to obtain a restored two-dimensional image; and performing three-dimensional reconstruction of the target object based on the repaired two-dimensional image and the plurality of images to generate digital assets corresponding to the target object. According to the method, the two-dimensional image of the target object at more visual angles can be obtained, the accuracy of generating digital assets can be improved, and the process is more convenient.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and specifically provides a digital asset generation method, a computer-readable storage medium, and an intelligent device. Background Art

[0002] In order to generate digital assets, related technologies usually use image acquisition equipment, sensors, etc. to collect data from various perspectives of the target object offline, and perform three-dimensional reconstruction of the target object based on these data to generate digital assets corresponding to the target object. Alternatively, a generative AI (Artificial Intelligence) algorithm is used based on a small number of images of the target object to obtain a three-dimensional model of the target object to generate digital assets corresponding to the target object.

[0003] However, the above methods that use image acquisition equipment, sensors, etc. require high equipment costs, complex acquisition processes, and cannot be produced offline, which is not convenient enough. Although the use of generative AI algorithms can be based on a small number of pictures and avoid complex acquisition processes, the image information used by this method is relatively limited. For example, only the color information of the image is used, and the corresponding three-dimensional model has poor accuracy, which affects the accuracy of digital assets. Summary of the invention

[0004] The present application aims to solve the above technical problem, that is, to solve the problem that the existing digital asset generation method is not convenient enough and has poor accuracy.

[0005] In a first aspect, the present application provides a method for generating a digital asset, which comprises: Acquire multiple images of the target object at multiple viewing angles, and obtain an initial three-dimensional color point cloud of the target object based on the multiple images; When there is a sparse point cloud area in the initial three-dimensional color point cloud, projecting the initial three-dimensional color point cloud onto a target two-dimensional plane to obtain a reprojected two-dimensional image; Inputting the reprojected two-dimensional image into an image restoration model to obtain a restored two-dimensional image; the restored two-dimensional image includes image information restored in the sparse area of ​​the point cloud; A target three-dimensional color point cloud of the target object is obtained based on the repaired two-dimensional image and the multiple images, and three-dimensional reconstruction of the target object is performed based on the target three-dimensional color point cloud to generate the digital asset corresponding to the target object.

[0006] In some embodiments, inputting the reprojected two-dimensional image into an image restoration model to obtain a restored two-dimensional image includes: The reprojected two-dimensional image is input into an image diffusion model to obtain the restored two-dimensional image.

[0007] In some embodiments, obtaining an initial three-dimensional color point cloud of the target object based on the multiple images includes: Based on each of the multiple images, respectively acquire the depth information and the color information of the target object, and obtain a color point cloud based on the depth information and the color information of the target object; The color point clouds corresponding to the multiple images are stitched together to obtain an initial three-dimensional color point cloud of the target object.

[0008] In some embodiments, based on each of the multiple images, respectively acquiring the depth information and the color information of the target object includes: Based on each of the multiple images, acquiring depth information of the image, and performing semantic segmentation on the image to obtain a target mask area; The depth information and color information corresponding to the target mask area in the image are acquired to obtain the depth information and color information of the target object.

[0009] In some embodiments, before projecting the initial three-dimensional color point cloud onto a target two-dimensional plane to obtain a reprojected two-dimensional image, the method further includes: performing dilation processing on the initial three-dimensional color point cloud.

[0010] In some embodiments, obtaining a target three-dimensional color point cloud of the target object based on the restored two-dimensional image and the multiple images includes: Respectively acquiring the depth information and the color information of the restored two-dimensional image, and obtaining a restored color point cloud of the target object based on the depth information and the color information of the restored two-dimensional image; Based on the multiple images, respectively acquire color point clouds of the target object to obtain multiple color point clouds; The repaired color point cloud and the multiple color point clouds are spliced ​​to obtain a target three-dimensional color point cloud of the target object.

[0011] In some embodiments, performing three-dimensional reconstruction of the target object based on the target three-dimensional color point cloud to generate the digital asset corresponding to the target object includes: Reconstructing the surface of the target three-dimensional color point cloud to obtain a polygonal mesh model; Texture mapping is performed on the polygonal mesh model to obtain the digital asset corresponding to the target object.

[0012] In some embodiments, the method further comprises: Determine whether the initial three-dimensional color point cloud exists in the point cloud sparse area; and when the initial three-dimensional color point cloud exists in the point cloud sparse area, project the initial three-dimensional color point cloud onto a target two-dimensional plane.

[0013] In a second aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the digital asset generation method described in any one of the above items is implemented.

[0014] In a third aspect, the present application provides a smart device, comprising: at least one processor; and, a memory communicatively coupled to the at least one processor; Wherein, a computer program is stored in the memory, and when the computer program is executed by the at least one processor, any of the digital asset generation methods described above is implemented.

[0015] In the case of adopting the above technical solution, the present application can obtain multiple images of the target object at multiple perspectives, and obtain the initial three-dimensional color point cloud of the target object based on the multiple images; when there is a sparse point cloud area in the initial three-dimensional color point cloud, the initial three-dimensional color point cloud is projected to the target two-dimensional plane to obtain a reprojected two-dimensional image; the reprojected two-dimensional image is input into the image restoration model to obtain a restored two-dimensional image; the restored two-dimensional image includes the image information after restoration of the sparse point cloud area; the target three-dimensional color point cloud of the target object is obtained based on the restored two-dimensional image and multiple images, and the target object is reconstructed in three dimensions based on the target three-dimensional color point cloud to generate a digital asset corresponding to the target object. This method can be implemented based on multiple images, and the acquisition process is convenient. In addition, by combining the image restoration model, the image information corresponding to the sparse point cloud area can be effectively restored to obtain two-dimensional images of the target object from more perspectives, which is conducive to improving the accuracy of generating digital assets. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The preferred embodiments of the present application are described below in conjunction with the accompanying drawings, in which: Figure 1 It is a flowchart of a method for generating digital assets provided in an embodiment of the present application; Figure 2 It is a schematic diagram of the process of obtaining an initial three-dimensional color point cloud provided in an embodiment of the present application; Figure 3 It is a schematic diagram of the process of obtaining a target three-dimensional color point cloud provided by an embodiment of the present application; Figure 4 It is a schematic diagram of the structure of the smart device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0017] Some embodiments of the present application are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are only used to explain the technical principles of the present application and are not intended to limit the protection scope of the present application.

[0018] Digital assets in this application refer to valuable resources that exist in digital form, virtual items or data that can be created, managed, stored and traded, and can be represented and stored in a mesh format. As an example, digital assets can include three-dimensional models, which are used to represent the three-dimensional shape and structure of real or fictional objects. Digital assets can also include three-dimensional textures, colors, etc.

[0019] Based on the description in the background technology section, it can be seen that in order to generate digital assets corresponding to the target object in the related technology, it is necessary to collect data from various perspectives of the target object. The equipment cost is high, the collection process is complicated, and it is not convenient enough. Although the use of AI algorithm can be based on a small number of pictures to avoid the complicated collection process, the image information used by this method is relatively limited, and only the color information of the image is used. The accuracy of reconstructing the shape characteristics of the three-dimensional digital assets is low, and the rationality of the generated digital assets cannot be controlled.

[0020] In view of this, the present application provides a method for generating digital assets, which can obtain two-dimensional images of the target object from more perspectives based on multiple images of the target object at multiple perspectives combined with the restored two-dimensional images obtained based on the image restoration model, thereby facilitating the acquisition of more image information of the target object and improving the accuracy of generating digital assets of the target object. At the same time, it can reduce the equipment cost of collecting images, simplify the process, and improve the convenience of generating digital assets.

[0021] See also Figure 1 As shown, Figure 1 : is a flowchart of a method for generating digital assets provided in an embodiment of the present application, which may include: Step S11: acquiring multiple images of the target object at multiple viewing angles, and obtaining an initial three-dimensional color point cloud of the target object based on the multiple images; Step S12: when there is a sparse point cloud area in the initial three-dimensional color point cloud, projecting the initial three-dimensional color point cloud onto a target two-dimensional plane to obtain a reprojected two-dimensional image; Step S13: inputting the reprojected two-dimensional image into the image restoration model to obtain a restored two-dimensional image; the restored two-dimensional image includes image information restored in the sparse area of ​​the point cloud; Step S14: obtaining a target three-dimensional color point cloud of the target object based on the repaired two-dimensional image and the multiple images, and performing three-dimensional reconstruction of the target object based on the target three-dimensional color point cloud to generate a digital asset corresponding to the target object.

[0022] In some embodiments, obtaining multiple images of the target object at multiple viewing angles in step S11 may specifically include: obtaining multiple images of the target object at multiple viewing angles using an image acquisition device, and the multiple images may all be color images, including color information of each pixel in the image.

[0023] In some embodiments, see Figure 2 As shown, the initial three-dimensional color point cloud of the target object obtained based on multiple images in step S11 can be specifically: Step S21: based on each of the multiple images, respectively acquiring depth information and color information of the target object, and obtaining a color point cloud based on the depth information and color information of the target object; Step S22: stitching the color point clouds corresponding to the multiple images to obtain an initial three-dimensional color point cloud of the target object.

[0024] In some embodiments, in step S21, respectively acquiring the depth information and the color information of the target object based on each of the multiple images may include: Based on each of the multiple images, depth information of the image is obtained, and semantic segmentation is performed on the image to obtain a target mask area; The depth information and color information corresponding to the target mask area in the image are obtained to obtain the depth information and color information of the target object.

[0025] In some embodiments, a trained model may be used to perform depth estimation on an image to obtain depth information of the image. The trained model may be a Depth Anything model, etc.

[0026] In some embodiments, a semantic segmentation model may be used to perform semantic segmentation on an image to obtain the type corresponding to each region in the image, wherein the region whose type is the target object is used as the target mask region. The semantic segmentation model may be UNet, DeepLab, etc., and is not particularly limited here.

[0027] In some embodiments, obtaining a color point cloud based on the depth information and color information of the target object in step S21 can be specifically as follows: using the depth information of the target object to convert the pixel coordinates in the image into point coordinates in three-dimensional space to obtain a point cloud; extracting the color information corresponding to each point in the point cloud from the color information of the target object to obtain a color point cloud of the current image.

[0028] In some embodiments, step S22 may specifically be to use an ICP (Iterative Closest Point) algorithm to stitch the color point clouds corresponding to the multiple images to obtain an initial three-dimensional color point cloud of the target object.

[0029] In some embodiments, the target two-dimensional plane in step S12 can be pre-set according to actual needs. Based on the setting of the target two-dimensional plane, a reprojected two-dimensional image with a different perspective from the multiple images acquired in step S11 can be obtained after projecting the initial three-dimensional color point cloud to the target two-dimensional plane, thereby facilitating the subsequent acquisition of the repaired two-dimensional image of the target object at more perspectives and enriching the image information of the target object.

[0030] In some embodiments, the image restoration model may adopt an image diffusion model, wherein the image diffusion model is a generation model based on a diffusion process, which is mainly used for image generation tasks, and achieves high-quality image generation by gradually adding noise to the image and learning how to restore the image from the noise. The image diffusion model may include StableDiffusion, ControlNet Inpaint, etc.

[0031] Step S13 may specifically include: inputting the reprojected two-dimensional image into the image diffusion model to obtain a repaired two-dimensional image. The image diffusion model may repair the point cloud sparse area in the reprojected two-dimensional image according to the surrounding pixels of the point cloud sparse area in the reprojected two-dimensional image to obtain the repaired two-dimensional image.

[0032] In some embodiments, the restored two-dimensional image may include image information after restoration of a sparse region of a point cloud, and the image information may include pixel values, color information, semantic information, and the like.

[0033] In some embodiments, see Figure 3 As shown, obtaining a target three-dimensional color point cloud of the target object based on the restored two-dimensional image and the multiple images in step S14 may include: Step S31: respectively acquiring the depth information and color information of the restored two-dimensional image, and obtaining a restored color point cloud of the target object based on the depth information and color information of the restored two-dimensional image; Step S32: acquiring color point clouds of the target object based on the multiple images respectively, to obtain multiple color point clouds; Step S33: splicing the repaired color point cloud and multiple color point clouds to obtain a target three-dimensional color point cloud of the target object.

[0034] In some embodiments, step S31 can be implemented by adaptively replacing the multiple images in step S21 with the repaired two-dimensional image, and then using the same method as step S21. For the sake of brevity, it will not be repeated here. For details, please refer to the description above.

[0035] In some embodiments, step S32 may specifically be to obtain the depth information and color information of the target object based on each of the multiple images, and obtain a color point cloud based on the depth information and color information of the target object. This step may also be implemented using the same method as step S21.

[0036] In some embodiments, step S33 may specifically be to use an ICP algorithm to stitch the repaired color point cloud and multiple color point clouds to obtain a target three-dimensional color point cloud of the target object.

[0037] In some embodiments, performing three-dimensional reconstruction of the target object based on the target three-dimensional color point cloud in step S14 to generate a digital asset corresponding to the target object includes: Perform surface reconstruction on the target three-dimensional color point cloud to obtain a polygonal mesh model; Texture mapping is performed on the polygonal mesh model to obtain the digital assets corresponding to the target object.

[0038] In some embodiments, a triangulation algorithm may be used to reconstruct the surface of the target three-dimensional color point cloud. The triangulation algorithm may transform the target three-dimensional color point cloud into a polyhedron, i.e., a polygonal mesh model, by connecting the points in the target three-dimensional color point cloud into a series of triangular meshes. In other embodiments, other surface reconstruction algorithms in the art may also be used.

[0039] In some embodiments, texture information can also be obtained from multiple two-dimensional images and restored two-dimensional images of the target object, and the texture information can be mapped to a polygonal mesh model to obtain a three-dimensional model that better matches the actual surface appearance of the target object, that is, a digital asset corresponding to the target object.

[0040] The above is a method for generating digital assets provided by an embodiment of the present application, which obtains multiple images of the target object at multiple perspectives, and obtains the initial three-dimensional color point cloud of the target object based on the multiple images; when there is a sparse point cloud area in the initial three-dimensional color point cloud, the initial three-dimensional color point cloud is projected to the target two-dimensional plane to obtain a reprojected two-dimensional image; the reprojected two-dimensional image is input into the image restoration model to obtain a restored two-dimensional image; the restored two-dimensional image includes the image information after restoration of the sparse point cloud area; the target three-dimensional color point cloud of the target object is obtained based on the restored two-dimensional image and multiple images, and the target object is reconstructed in three dimensions based on the target three-dimensional color point cloud to generate a digital asset corresponding to the target object. This method can be implemented based on multiple images, and the acquisition process is convenient. In combination with the image restoration model, the image information corresponding to the sparse point cloud area can be effectively restored to obtain two-dimensional images of the target object from more perspectives, which is conducive to improving the accuracy of generating digital assets.

[0041] In other embodiments, in order to improve the reprojection effect, the initial three-dimensional color point cloud may be expanded before being projected onto the target two-dimensional plane to obtain the reprojected two-dimensional image.

[0042] The expansion processing of the initial three-dimensional color point cloud may specifically be to increase the diameter of the initial three-dimensional color point cloud.

[0043] In other embodiments, the method provided by the embodiments of the present application may also include: determining whether there is a point cloud sparse area in the initial three-dimensional color point cloud, so as to project the initial three-dimensional color point cloud to the target two-dimensional plane when there is a point cloud sparse area in the initial three-dimensional color point cloud.

[0044] In some embodiments, the number of points in each area of ​​the initial three-dimensional color point cloud can be obtained to determine whether the number of points in each area has reached the corresponding preset number. If not, the current area is determined to be a point cloud sparse area.

[0045] It should be noted that in the embodiment of the present application, it is also possible to determine whether there is a point cloud sparse area in the three-dimensional color point cloud obtained based on the repaired two-dimensional image and multiple images. If not, the three-dimensional color point cloud is used as the target three-dimensional color point cloud; if yes, steps S12 to S13 can be repeatedly executed based on the three-dimensional color point cloud until the target three-dimensional color point cloud is obtained.

[0046] It is understood by those skilled in the art that all or part of the processes in the method for implementing the above-mentioned embodiments of the present application can also be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may include: any entity or device, medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory, random access memory, electric carrier signal, telecommunication signal and software distribution medium, etc. that can carry the computer program code.

[0047] Another aspect of the present application further provides a computer-readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, the digital asset generation method in any of the above embodiments can be implemented. The computer-readable storage medium can be a storage device formed by various electronic devices. Optionally, the computer-readable storage medium in the embodiment of the present application is a non-transitory computer-readable storage medium.

[0048] In another aspect of the present application, a smart device is provided.

[0049] In an embodiment of an intelligent device according to the present application, the intelligent device may include at least one processor; and a memory connected to the at least one processor in communication; wherein a computer program is stored in the memory, and when the computer program is executed by the at least one processor, the method described in any of the above embodiments is implemented. The intelligent device described in the present application may include a driving device, a smart car, a robot, and the like. Figure 4 , Figure 4 4 shows exemplarily that the memory 41 and the processor 42 are communicatively connected via a bus.

[0050] In some embodiments of the present application, the smart device may further include at least one sensor for sensing information. The sensor is communicatively connected to any type of processor mentioned in the present application. Optionally, the smart device may further include an autonomous driving system for guiding the smart device to drive itself or assist in driving. The processor communicates with the sensor and / or the autonomous driving system to complete the method described in any of the above embodiments.

[0051] The relevant user personal information that may be involved in the various embodiments of this application is strictly in accordance with the requirements of laws and regulations, following the principles of legality, legitimacy and necessity, based on the reasonable purposes of business scenarios, to process the personal information that users actively provide during the use of products / services or generated due to the use of products / services, as well as the personal information obtained with the user's authorization.

[0052] The user personal information processed by this application will vary depending on the specific product / service scenario, and shall be based on the specific scenario in which the user uses the product / service, and may involve the user's account information, device information, driving information, vehicle information or other related information. This application will treat the user's personal information and its processing with a high degree of diligence.

[0053] This application attaches great importance to the security of user personal information and has taken reasonable and feasible security protection measures that meet industry standards to protect user information and prevent personal information from being accessed, disclosed, used, modified, damaged or lost without authorization.

[0054] So far, the technical solutions of the present application have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present application is obviously not limited to these specific embodiments. Without departing from the principles of the present application, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present application.

Claims

1. A method for generating digital assets, characterized in that: include: Acquire multiple images of the target object at multiple viewing angles, and obtain an initial three-dimensional color point cloud of the target object based on the multiple images; When there is a sparse point cloud area in the initial three-dimensional color point cloud, projecting the initial three-dimensional color point cloud onto a target two-dimensional plane to obtain a reprojected two-dimensional image; Inputting the reprojected two-dimensional image into an image restoration model to obtain a restored two-dimensional image; the restored two-dimensional image includes image information restored in the sparse area of ​​the point cloud; A target three-dimensional color point cloud of the target object is obtained based on the repaired two-dimensional image and the multiple images, and three-dimensional reconstruction of the target object is performed based on the target three-dimensional color point cloud to generate the digital asset corresponding to the target object.

2. The method according to claim 1, characterized in that The step of inputting the reprojected two-dimensional image into an image restoration model to obtain a restored two-dimensional image comprises: The reprojected two-dimensional image is input into an image diffusion model to obtain the restored two-dimensional image.

3. The method according to claim 1 or 2, characterized in that: Obtaining an initial three-dimensional color point cloud of the target object based on the multiple images includes: Based on each of the multiple images, respectively acquire the depth information and the color information of the target object, and obtain a color point cloud based on the depth information and the color information of the target object; The color point clouds corresponding to the multiple images are stitched together to obtain an initial three-dimensional color point cloud of the target object.

4. The method according to claim 3, characterized in that Based on each of the multiple images, respectively acquiring the depth information and the color information of the target object includes: Based on each of the multiple images, acquiring depth information of the image, and performing semantic segmentation on the image to obtain a target mask area; The depth information and color information corresponding to the target mask area in the image are acquired to obtain the depth information and color information of the target object.

5. The method according to claim 1 or 2, characterized in that: Before projecting the initial three-dimensional color point cloud onto a target two-dimensional plane to obtain a reprojected two-dimensional image, the method further includes: performing expansion processing on the initial three-dimensional color point cloud.

6. The method according to claim 1 or 2, characterized in that: Obtaining a target three-dimensional color point cloud of the target object based on the restored two-dimensional image and the multiple images includes: Respectively acquiring the depth information and the color information of the restored two-dimensional image, and obtaining a restored color point cloud of the target object based on the depth information and the color information of the restored two-dimensional image; Based on the multiple images, respectively acquire color point clouds of the target object to obtain multiple color point clouds; The repaired color point cloud and the multiple color point clouds are spliced ​​to obtain a target three-dimensional color point cloud of the target object.

7. The method according to claim 1 or 2, characterized in that: Performing three-dimensional reconstruction of the target object based on the target three-dimensional color point cloud to generate the digital asset corresponding to the target object includes: Reconstructing the surface of the target three-dimensional color point cloud to obtain a polygonal mesh model; Texture mapping is performed on the polygonal mesh model to obtain the digital asset corresponding to the target object.

8. The method according to claim 1 or 2, characterized in that: The method further comprises: It is determined whether the initial three-dimensional color point cloud has the point cloud sparse area, so that when the initial three-dimensional color point cloud has the point cloud sparse area, the initial three-dimensional color point cloud is projected onto a target two-dimensional plane.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the digital asset generation method described in any one of claims 1 to 8 is implemented.

10. A smart device, characterized in that: include: at least one processor; and, a memory communicatively coupled to the at least one processor; Wherein, a computer program is stored in the memory, and when the computer program is executed by the at least one processor, the digital asset generation method according to any one of claims 1 to 8 is implemented.

Images