A model reconstruction method and apparatus
By generating object point clouds and adjusting the positions of detailed data points, the problem of lack of detailed information in existing model reconstruction techniques is solved, achieving a more realistic model reconstruction effect.
Patent Information
- Application Number
- CN202211730661.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-12-30
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2042-12-30
AI Technical Summary
Existing model reconstruction techniques produce models that are highly similar to real-world objects, but lack detailed information, resulting in low model realism.
By obtaining multiple object images and their relative positions with the acquisition devices, an object point cloud is generated. The position of detail data points in the object point cloud is then adjusted using the image gradient of detail pixels, and the object model is finally reconstructed.
It improves the accuracy and realism of model reconstruction, enhances the model's ability to express details, and has a low cost.
Smart Images

Figure CN115830245B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to the fields of digital human, metaverse, augmented reality, virtual reality, mixed reality, extended reality, etc., and especially to a model reconstruction method and apparatus. Background Technology
[0002] Model reconstruction technology can be applied to various scenarios, including metaverse, digital humans, AR (Augmented Reality), and VR (Virtual Reality). For example, in the digital human scenario, model reconstruction technology can be used to construct virtual characters such as virtual customer service representatives and virtual anchors.
[0003] In the technical solutions related to reconstructing models of real-world objects, images of the real-world object collected from multiple angles can be obtained, along with the relative positional relationships of the image acquisition devices when collecting these multiple images. Based on the obtained multiple images and their relative positional relationships, model reconstruction is performed with the aim of maximizing the overall similarity between the constructed model and the actual object. Summary of the Invention
[0004] This disclosure provides a model reconstruction method and apparatus.
[0005] According to one aspect of this disclosure, a model reconstruction method is provided, comprising:
[0006] The process involves obtaining multiple images of a real-world object, as well as the relative positions of the image acquisition devices when acquiring each object image, with each object image having a different acquisition angle.
[0007] Based on the images of each object and their relative positional relationships, an object point cloud of the real-world object is generated;
[0008] Obtain the image gradient of detail pixels, wherein the detail pixels are the pixels in the object image that represent the detail information of the real object;
[0009] Based on the image gradient of the detail pixel, the position of the detail data point in the object point cloud is adjusted, wherein the detail data point is: the data point corresponding to the detail pixel;
[0010] Based on the adjusted object point cloud, reconstruct the object model of the real-world object.
[0011] According to another aspect of this disclosure, a model reconstruction apparatus is provided, comprising:
[0012] The information acquisition module is used to acquire multiple object images of the real object, as well as the relative positional relationship of the image acquisition devices when acquiring each object image, wherein the acquisition angle corresponding to each object image is different;
[0013] The point cloud generation module is used to generate object point clouds of the real objects based on the images of each object and the relative positional relationships.
[0014] The gradient acquisition module is used to obtain the image gradient of detail pixels, wherein the detail pixels are pixels in the object image that represent the detail information of the real object;
[0015] The position adjustment module is used to adjust the position of the detail data points in the object point cloud according to the image gradient of the detail pixel points, wherein the detail data points are data points corresponding to the detail pixel points;
[0016] The model reconstruction module is used to reconstruct the object model of the real-world object based on the adjusted object point cloud.
[0017] According to another aspect of this disclosure, an electronic device is provided, comprising:
[0018] At least one processor; and
[0019] A memory communicatively connected to the at least one processor; wherein,
[0020] The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the model reconstruction method described above.
[0021] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform the above-described model reconstruction method.
[0022] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the above-described model reconstruction method.
[0023] As can be seen from the above, when constructing a model using the scheme provided in the embodiments of this disclosure, an object point cloud is generated based on each object image and the aforementioned relative positional relationship, and the image gradient of detail pixels in the object image is obtained. The position of detail data points in the object point cloud is adjusted based on the image gradient of the detail pixels, which makes the detailed information of the object reflected by the data points in the adjusted object point cloud more accurate. Thus, model reconstruction based on the adjusted object point cloud makes the reconstructed object model more realistic. Therefore, the model reconstruction scheme provided in the embodiments of this disclosure can improve the accuracy of model reconstruction and enhance the realism and detail expression ability of the constructed model.
[0024] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description
[0025] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:
[0026] Figure 1 A flowchart illustrating the first model reconstruction method provided in this embodiment of the disclosure;
[0027] Figure 2 A flowchart illustrating the second model reconstruction method provided in this embodiment of the disclosure;
[0028] Figure 3 This is a schematic diagram of the structure of the first model reconstruction device provided in the embodiments of this disclosure;
[0029] Figure 4 This is a schematic diagram of the structure of the second model reconstruction device provided in the embodiments of this disclosure;
[0030] Figure 5 This is a block diagram of an electronic device used to implement the model reconstruction method of the embodiments of this disclosure. Detailed Implementation
[0031] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.
[0032] Because the model reconstruction schemes in related technologies aim to achieve the highest overall similarity between the constructed model and the real object, the constructed model often lacks the object details of the real object. Thus, although the model reconstruction aims to achieve the highest similarity, the similarity between the reconstructed model and the real object is still low, and the model's realism is low.
[0033] To address the aforementioned problems, this disclosure provides a model reconstruction method and apparatus. Detailed descriptions are provided below through specific embodiments.
[0034] See Figure 1 , Figure 1 This is a flowchart illustrating the first model reconstruction method provided in this embodiment. In this embodiment, the method includes the following steps S101-S105.
[0035] Step S101: Obtain multiple object images of the real object, and the relative positional relationship of the image acquisition devices when acquiring each object image.
[0036] The acquisition angles for each object image are different.
[0037] The aforementioned real-world objects can be people, animals, buildings, etc., and can also be human faces, animal bodies, etc.
[0038] Specifically, the image of the above-mentioned object and its relative positional relationship can be obtained through any of the following methods.
[0039] In the first implementation, multiple acquisition positions of the image acquisition device can be preset, the relative positional relationship between these multiple acquisition positions can be obtained, and the images of the real objects acquired by the image acquisition device at each acquisition position can be obtained.
[0040] In the second implementation, when the image acquisition device acquires an image of a real object, the acquisition position of the image acquisition device can be recorded, and the image acquired by the image acquisition device can be obtained. Based on the recorded acquisition position, the relative positional relationship of the image acquisition device can be determined.
[0041] In the third implementation, the device parameters of the image acquisition device can be pre-calibrated. For example, the image acquisition device can be a camera, and the device parameters can be camera intrinsic parameters. After obtaining multiple images of the real object, image key points that are representative of the image content in each object image are extracted. The image key points of each object image are matched to obtain the matched image key points that represent the same image content. Based on the positions of the matched image key points and the aforementioned device parameters, the relative positional relationship of the image acquisition device when acquiring each object image can be calculated using existing technology.
[0042] Step S102: Generate object point clouds of real objects based on the images of each object and their relative positional relationships.
[0043] Specifically, based on the images of each object and their relative positions, object point clouds of real-world objects can be generated using existing point cloud generation techniques, which will not be elaborated here.
[0044] Step S103: Obtain the image gradient of the detail pixels.
[0045] Among them, detail pixels are pixels in the object image that represent detailed information about the real object.
[0046] Real-world objects possess object details. In an object image, the detail pixels that represent the detailed information of the real-world object are the pixels corresponding to the object details of the real-world object. For example, a real-world object can be a human face. Facial details include wrinkles, pores, acne scars, and so on. The aforementioned detail pixels are the pixels corresponding to facial wrinkles, pores, acne scars, skin texture, and so on.
[0047] Specifically, the image gradient of detailed pixels can be obtained through either of the following two methods.
[0048] In the first implementation, a high-pass filter can be applied to the object image to obtain the detail pixels and their image gradients.
[0049] Because the pixel values of detail pixels in an object image differ significantly from those of surrounding pixels—for example, the pixel value of a wrinkle on a face in an object image varies considerably compared to the pixel value of the skin surrounding the wrinkle—high-pass filtering can filter out high-frequency information in the image, namely the edges of the object. The edges of an object in an image are precisely the pixels in the region where the pixel value changes significantly. Therefore, high-pass filtering of an object image can accurately filter out detail pixels in the object image.
[0050] High-pass filtering of an object image can be achieved using existing filtering techniques, which will not be detailed here. Furthermore, since high-pass filtering requires calculating the image gradient of each pixel in the image, the image gradient of the detail pixels can be determined from the calculated image gradients of each pixel after obtaining the detail pixels in the object image.
[0051] In this implementation, high-pass filtering is applied to the object image to accurately obtain the detail pixels and their image gradients. By utilizing the more accurate image gradients of the detail pixels, the accuracy of subsequent processing can be improved.
[0052] In the second implementation, a detail detection model can be pre-trained to detect detail pixels and their image gradients in an image. By inputting the object image into this detail detection model, the position of the detail pixels and the image gradient output by the model can be obtained.
[0053] In addition, after obtaining the position of the detail pixels in the model output, the detail pixels in the object image and the neighboring pixels around the detail pixels can be determined. Based on the pixel values of the determined detail pixels and neighboring pixels, the image gradient representing the degree of pixel value change can be calculated.
[0054] Step S104: Adjust the position of detail data points in the object point cloud according to the image gradient of detail pixels.
[0055] Among them, detail data points are the data points corresponding to detail pixels.
[0056] Specifically, since the object point cloud is generated based on the object images, there is a correspondence between each data point in the object point cloud and each pixel in the object image. When obtaining the image gradient of the aforementioned detail pixels, the detail pixels in the object image can be determined. Thus, based on the correspondence between the data points in the object point cloud and the pixels in the object image, the detail data points corresponding to the detail pixels can be determined in the object point cloud. Therefore, based on the image gradient of the detail pixels, the positions of the detail data points corresponding to the detail pixels can be adjusted. For example, the depth of the detail data points can be adjusted.
[0057] In one embodiment of this disclosure, when adjusting the position of detail data points according to the image gradient of detail pixels, the adjustment range for adjusting the detail data points can be determined according to the image gradient of detail pixels, and the position of the detail data points can be adjusted in a preset direction of the object point cloud according to the determined adjustment range.
[0058] The aforementioned preset direction can be the depth direction, height direction, etc. of the object point cloud.
[0059] The aforementioned preset direction can be set manually.
[0060] In addition, since the image gradient of a pixel is a vector with direction and magnitude, the magnitude of the image gradient of a pixel represents the degree of change of the pixel value at the location of the pixel, and the direction of the image gradient of a pixel represents the change of the pixel value at the location of the pixel from high to low or from low to high. Therefore, when adjusting the position of detail data points according to the image gradient of a pixel, the above-mentioned preset direction can be determined according to the direction of the image gradient of the detail pixel.
[0061] The aforementioned adjustment range can be represented by the distance between the positions of the data points before and after the adjustment; the aforementioned adjustment range can also be represented by the adjustment angle formed by the first straight line and the second straight line, where the first straight line is the straight line between the data point before adjustment and the origin of the object point cloud, and the second straight line is the straight line between the data point after adjustment and the origin.
[0062] Specifically, when determining the aforementioned adjustment range, the image gradient of the detail pixels and the adjustment range of the detail data points can be positively correlated. The larger the image gradient of the detail pixels, the larger the adjustment range of the detail data points can be; conversely, the smaller the image gradient of the detail pixels, the smaller the adjustment range of the detail data points can be. Therefore, a proportionality coefficient can be preset, and this coefficient can be multiplied by the magnitude of the image gradient to obtain the multiplied result, which can then be used as the aforementioned adjustment range. Furthermore, a correspondence between the image gradient range and the adjustment range can be established. This way, after obtaining the image gradient of the detail pixels, the range of the obtained image gradient can be determined, and the adjustment range corresponding to that range can be determined based on the aforementioned correspondence. After determining the aforementioned adjustment range, the position of the detail data points can be adjusted in the preset direction according to the determined adjustment range.
[0063] In this solution, since there is a significant difference between the object details and other parts of the real object, this is reflected in the object image as a large image gradient of the detail pixels. Therefore, based on the image gradient of the detail pixels, the adjustment range of the detail data points can be accurately determined, thereby accurately adjusting the position of the detail data points in the aforementioned preset direction.
[0064] Furthermore, after determining the aforementioned detail data points, other data points surrounding them can be identified and also treated as detail data points. This allows for the adjustment of the positions of these other data points simultaneously when adjusting the positions of the detail data points based on the image gradient. The specific implementation method for adjusting the positions of these other data points can be found in subsequent embodiments and will not be detailed here.
[0065] Step S105: Reconstruct the object model of the real object based on the adjusted object point cloud.
[0066] Specifically, based on the adjusted object point cloud, the object model of the real object can be reconstructed using existing model reconstruction techniques, which will not be elaborated here.
[0067] For example, the existing Persson reconstruction algorithm can be used to reconstruct the object model of a real-world object based on the adjusted object point cloud.
[0068] As can be seen from the above, when constructing a model using the scheme provided in the embodiments of this disclosure, an object point cloud is generated based on each object image and the aforementioned relative positional relationship, and the image gradient of detail pixels in the object image is obtained. The position of detail data points in the object point cloud is adjusted based on the image gradient of the detail pixels, which makes the detailed information of the object reflected by the data points in the adjusted object point cloud more accurate. Thus, model reconstruction based on the adjusted object point cloud makes the reconstructed object model more realistic. Therefore, the model reconstruction scheme provided in the embodiments of this disclosure can improve the accuracy of model reconstruction and enhance the realism and detail expression ability of the constructed model.
[0069] In addition, the model reconstruction scheme provided in this disclosure has low requirements for image acquisition equipment. Therefore, by applying the model reconstruction scheme provided in this disclosure, high-precision model reconstruction can be achieved at a lower cost.
[0070] In one embodiment of this disclosure, after adjusting the positions of detail data points in the object point cloud and before reconstructing the object model, each detail data point can be smoothed, and then the object model can be reconstructed based on the smoothed object point cloud, which can further improve the realism of the reconstructed object model.
[0071] In one embodiment of this disclosure, the real object is a face, and the object image is a face image.
[0072] In this scenario, object point clouds are generated based on each face image, and the positions of detail data points within these point clouds are adjusted. Based on the adjusted object point cloud, a face model can be reconstructed. Because the object point cloud used for reconstruction has been adjusted, the data points in the point cloud reflect more accurate details of the face, resulting in a more realistic face model.
[0073] The following explains the specific implementation method for adjusting the position of detailed data points and other surrounding data points.
[0074] In one embodiment of this disclosure, when adjusting the position of detail data points, a target data point corresponding to a detail pixel can be determined in the object point cloud, and a data point whose distance from the target data point is less than a preset distance threshold can be determined in the object point cloud. Both of these determined data points are used as detail data points, and then the position of the detail data points is adjusted according to the image gradient of the detail pixel and the target distance.
[0075] The target distance is the distance between the detailed data point and the target data point.
[0076] The aforementioned distance threshold can be set manually.
[0077] For details on how to determine the target data points corresponding to the detail pixels, please refer to step S104 above.
[0078] Specifically, after determining the target data points, a spherical range with the target data point's location as the center and the aforementioned distance threshold as the radius can be defined. Data points within this spherical range are then identified, thus defining all data points within the range as detail data points. After determining the detail data points, for each detail data point, if it is the target data point, the target distance is 0. In this case, the target adjustment magnitude can be determined based on the image gradient of the detail pixel. If the detail data point is not the target data point, the adjustment magnitude can be determined based on the target distance between the detail data point and the target data point, as well as the aforementioned target adjustment magnitude. The larger the target distance, the smaller the adjustment magnitude; that is, the farther the detail data point is from the target data point, the smaller the adjustment magnitude. After determining the adjustment magnitude for each detail data point, the position of each detail data point can be adjusted according to its respective adjustment magnitude.
[0079] As can be seen from the above, in the solution provided by the embodiments of this disclosure, the target data points corresponding to the detail pixels and the data points whose distance from the target data points is less than a preset distance threshold can all be regarded as detail data points, thereby adjusting the position of multiple detail data points. This can further make the detailed information of the object reflected by the adjusted object point cloud more accurate, and thus the object model can be reconstructed based on the adjusted object point cloud, making the reconstructed object model more realistic.
[0080] To further improve the accuracy of model reconstruction, in one embodiment of this disclosure, see [link to relevant documentation]. Figure 2 The flowchart of the second model reconstruction method is provided. In this embodiment, the above method includes the following steps S201-S207.
[0081] Step S201: Obtain multiple object images of the real object, and the relative positional relationship of the image acquisition devices when acquiring each object image.
[0082] Step S202: Generate object point clouds of real objects based on the images of each object and their relative positional relationships.
[0083] Step S203: Obtain the image gradient of the detail pixels.
[0084] Step S204: Adjust the position of detail data points in the object point cloud according to the image gradient of detail pixels.
[0085] The steps S201-S204 are the same as the steps S101-S104 above, and will not be repeated here.
[0086] Step S205: Based on the adjusted positions and relative positions of the detail data points, determine the mapped pixel values of the pixels obtained by mapping the detail data points back to the object image.
[0087] Specifically, based on the adjusted positions of the detail data points and the aforementioned relative positional relationships, existing mapping techniques can be used to determine the mapped pixel values of the pixels obtained by mapping the detail data points back to the object image, which will not be elaborated here.
[0088] Step S206: Based on the pixel values of the detail pixels and the mapped pixel values, readjust the position of the detail data points, and return to the step of determining the mapped pixel values of the pixels obtained by mapping the detail data points back to the object image, until the adjustment termination condition is met.
[0089] There can be multiple conditions for ending the adjustment, and the execution flow of this step will be different depending on the conditions for ending the adjustment.
[0090] In one embodiment of this disclosure, the adjustment termination condition may be that the number of adjustments to the detailed data points reaches a preset number.
[0091] In this case, the number of times the detail data points are adjusted can be recorded. After each adjustment of the position of the detail data points based on the pixel value of the detail pixel and the mapped pixel value, the recorded number of adjustments is incremented by one, and it is determined whether the number of adjustments after incrementing by one has reached the preset number. If it has not reached the preset number, the process returns to the step of determining the mapped pixel value of the pixel obtained by mapping the detail data points back to the object image. If it has reached the preset number, the subsequent step S207 is executed.
[0092] In another embodiment of this disclosure, the above-mentioned adjustment termination condition may also be that the difference between the pixel value of the detail pixel and the redetermined mapped pixel value is less than a preset pixel value threshold.
[0093] In this case, after adjusting the position of the detail data points and remapping to obtain the mapped pixel value, the difference between the pixel value of the detail pixel point and the newly determined mapped pixel value can be calculated. If the difference is greater than or equal to the preset pixel value threshold, the position of the detail data points is adjusted again based on the pixel value of the detail pixel point and the newly determined mapped pixel value, and the process returns to the step of determining the mapped pixel value. If the difference is less than the preset pixel value threshold, the subsequent step S207 is executed.
[0094] Other conditions may also be used to end the above adjustment, which will not be detailed here.
[0095] Furthermore, the specific implementation method for adjusting the position of detailed data points can be found in subsequent embodiments, which will not be detailed here.
[0096] Step S207: Reconstruct the object model of the real object based on the adjusted object point cloud.
[0097] This step is the same as step S105 above, and will not be repeated here.
[0098] As can be seen from the above, when using the scheme provided in this embodiment to reconstruct the model, after adjusting the position of the detail data points according to the image gradient, the position of the detail data points can be adjusted again according to the pixel value of the detail pixel and the above-mentioned mapped pixel value until the adjustment end condition is met. This enables the detail data points after the adjustment to more accurately represent the detail information of the real object, thereby reconstructing the object model based on the adjusted object point cloud, which can improve the accuracy of model reconstruction and the realism of the reconstructed object model.
[0099] The following explains the specific implementation method for adjusting the position of detailed data points.
[0100] In one embodiment of this disclosure, the position of the detail data points can be adjusted using either of the following two implementation methods.
[0101] In the first implementation, the pixel value difference between the pixel value of the detail pixel and the mapped pixel value can be calculated, and the difference can be multiplied by a pre-set adjustment ratio coefficient to obtain the adjustment parameter used to adjust the position of the data point, thereby adjusting the position of the detail data point according to the adjustment parameter.
[0102] For example, the aforementioned adjustment ratio coefficient can represent that for every 1 pixel value difference between the pixel value of the detail pixel and the mapped pixel value, the detail data point will be adjusted by m millimeters in the preset direction. If the difference in the pixel value is n, then the aforementioned adjustment parameter can be calculated as m*n millimeters. In this case, the detail data point can be adjusted by m*n millimeters in the preset direction.
[0103] In the second implementation, the pixel value loss between the pixel value of the detail pixel and the mapped pixel value can also be calculated, and the position of the detail data point can be adjusted according to the pixel value loss.
[0104] The pixel value loss mentioned above can be calculated using existing loss functions and models, which will not be detailed here.
[0105] In this implementation, the pixel value loss between the pixel value of the detail pixel and the mapped pixel value reflects the difference between the detail information of the real object represented by the detail pixel and the detail information of the real object represented by the detail data point. Adjusting the position of the detail data point according to the above pixel value loss value can make the detail information represented by the adjusted detail data point more consistent with the detail information represented by the detail pixel. This enables the adjusted object point cloud to have rich and accurate object details. Therefore, reconstructing the object model based on the adjusted object point cloud can improve the accuracy of model reconstruction and enhance the realism of the constructed model.
[0106] In one embodiment of this disclosure, when adjusting the position of detail data points using the second implementation method described above, the adjustment termination condition may also be that the pixel value loss is less than a preset loss value threshold.
[0107] The aforementioned preset loss threshold can be set manually.
[0108] At this point, after determining the above-mentioned mapped pixel value, the above-mentioned pixel value loss value can be calculated first, and it can be determined whether the pixel value loss value is less than the preset loss value threshold. If not, the position of the detail data point is adjusted according to the pixel value loss value, and the step of determining the mapped pixel value of the pixel point obtained by mapping the detail data point back to the object image is returned. If yes, step S207 is executed.
[0109] In this scheme, since the pixel value loss value reflects the difference between the detailed information of the real object represented by the detailed pixels and the detailed information of the real object represented by the detailed data points, if the pixel value loss value is less than the preset loss value threshold, it means that the difference is not significant. At this time, the object point cloud already has rich object details. Therefore, the object model can be reconstructed based on the adjusted object point cloud, which can improve the accuracy of model reconstruction and the realism of the constructed model.
[0110] Corresponding to the above-described model reconstruction method, this disclosure also provides a model reconstruction apparatus.
[0111] In one embodiment of this disclosure, see [link to embodiment]. Figure 3 A schematic diagram of the structure of the first model reconstruction device is provided. In this embodiment, the device includes:
[0112] The information acquisition module 301 is used to acquire multiple object images of the real object, as well as the relative positional relationship of the image acquisition devices when acquiring each object image, wherein the acquisition angle corresponding to each object image is different;
[0113] The point cloud generation module 302 is used to generate an object point cloud of the real object based on the images of each object and the relative positional relationship.
[0114] The gradient acquisition module 303 is used to acquire the image gradient of detail pixels, wherein the detail pixels are pixels in the object image that represent the detail information of the real object;
[0115] The position adjustment module 304 is used to adjust the position of the detail data points in the object point cloud according to the image gradient of the detail pixel points, wherein the detail data points are data points corresponding to the detail pixel points;
[0116] The model reconstruction module 305 is used to reconstruct the object model of the real object based on the adjusted object point cloud.
[0117] As can be seen from the above, when constructing a model using the scheme provided in the embodiments of this disclosure, an object point cloud is generated based on each object image and the aforementioned relative positional relationship, and the image gradient of detail pixels in the object image is obtained. The position of detail data points in the object point cloud is adjusted based on the image gradient of the detail pixels, which makes the detailed information of the object reflected by the data points in the adjusted object point cloud more accurate. Thus, model reconstruction based on the adjusted object point cloud makes the reconstructed object model more realistic. Therefore, the model reconstruction scheme provided in the embodiments of this disclosure can improve the accuracy of model reconstruction and enhance the realism and detail expression ability of the constructed model.
[0118] In addition, the model reconstruction scheme provided in this disclosure has low requirements for image acquisition equipment. Therefore, by applying the model reconstruction scheme provided in this disclosure, high-precision model reconstruction can be achieved at a lower cost.
[0119] In one embodiment of this disclosure, see [link to embodiment]. Figure 4 A schematic diagram of a second model reconstruction device is provided. In this embodiment, the device includes:
[0120] The information acquisition module 401 is used to acquire multiple object images of the real object, as well as the relative positional relationship of the image acquisition devices when acquiring each object image, wherein the acquisition angle corresponding to each object image is different;
[0121] The point cloud generation module 402 is used to generate an object point cloud of the real object based on the images of each object and the relative positional relationship.
[0122] The gradient acquisition module 403 is used to acquire the image gradient of detail pixels, wherein the detail pixels are pixels in the object image that represent the detail information of the real object;
[0123] The position adjustment module 404 is used to adjust the position of the detail data points in the object point cloud according to the image gradient of the detail pixel points, wherein the detail data points are data points corresponding to the detail pixel points;
[0124] The pixel mapping module 405 is used to determine the mapped pixel value of the pixel obtained by mapping the detail data points back to the object image based on the adjusted position of the detail data points and the relative position relationship after adjusting the position of the detail data points according to the image gradient of the detail pixel points.
[0125] The optimization and adjustment module 406 is used to adjust the position of the detail data point again based on the pixel value of the detail pixel point and the mapped pixel value, and return to the step of determining the mapped pixel value of the pixel point obtained by mapping the detail data point back to the object image, until the adjustment end condition is met.
[0126] The model reconstruction module 407 is used to reconstruct the object model of the real object based on the adjusted object point cloud.
[0127] As can be seen from the above, when using the scheme provided in this embodiment to reconstruct the model, after adjusting the position of the detail data points according to the image gradient, the position of the detail data points can be adjusted again according to the pixel value of the detail pixel and the above-mentioned mapped pixel value until the adjustment end condition is met. This enables the detail data points after the adjustment to more accurately represent the detail information of the real object, thereby reconstructing the object model based on the adjusted object point cloud, which can improve the accuracy of model reconstruction and the realism of the reconstructed object model.
[0128] In one embodiment of this disclosure, the optimization adjustment module 406 is specifically used for:
[0129] Calculate the pixel value loss between the pixel value of the detail pixel and the mapped pixel value. Based on the pixel value loss, adjust the position of the detail data point and return to the step of determining the mapped pixel value of the pixel obtained by mapping the detail data point back to the object image, until the adjustment termination condition is met.
[0130] In this scheme, the pixel value loss between the pixel value of the detail pixel and the mapped pixel value can reflect the difference between the detail information of the real object represented by the detail pixel and the detail information of the real object represented by the detail data point. Adjusting the position of the detail data point according to the above pixel value loss value can make the detail information represented by the adjusted detail data point more consistent with the detail information represented by the detail pixel. This can make the adjusted object point cloud have rich and accurate object details. Therefore, the object model can be reconstructed based on the adjusted object point cloud, which can improve the accuracy of model reconstruction and improve the realism of the constructed model.
[0131] In one embodiment of this disclosure, the adjustment termination condition is that the pixel value loss is less than a preset loss value threshold.
[0132] In this scheme, since the pixel value loss value reflects the difference between the detailed information of the real object represented by the detailed pixels and the detailed information of the real object represented by the detailed data points, if the pixel value loss value is less than the preset loss value threshold, it means that the difference is not significant. At this time, the object point cloud already has rich object details. Therefore, the object model can be reconstructed based on the adjusted object point cloud, which can improve the accuracy of model reconstruction and the realism of the constructed model.
[0133] In one embodiment of this disclosure, the position adjustment module 304 is specifically used for:
[0134] Based on the image gradient of the detailed pixels, determine the adjustment range for adjusting the detailed data points in the object point cloud;
[0135] The positions of the detail data points are adjusted in a predetermined direction along the object point cloud according to a determined adjustment range.
[0136] In this solution, since there is a significant difference between the object details and other parts of the real object, this is reflected in the object image as a large image gradient of the detail pixels. Therefore, based on the image gradient of the detail pixels, the adjustment range of the detail data points can be accurately determined, thereby accurately adjusting the position of the detail data points in the aforementioned preset direction.
[0137] In one embodiment of this disclosure, the position adjustment module 304 is specifically used for:
[0138] In the object point cloud, the target data point corresponding to the detail pixel is determined;
[0139] The target data point and data points whose distance to the target data point is less than a preset distance threshold are identified as the detailed data points.
[0140] The position of the detail data point is adjusted based on the image gradient of the detail pixel and the target distance, wherein the target distance is the distance between the detail data point and the target data point.
[0141] As can be seen from the above, in the solution provided by the embodiments of this disclosure, the target data points corresponding to the detail pixels and the data points whose distance from the target data points is less than a preset distance threshold can all be regarded as detail data points, thereby adjusting the position of multiple detail data points. This can further make the detailed information of the object reflected by the adjusted object point cloud more accurate, and thus the object model can be reconstructed based on the adjusted object point cloud, making the reconstructed object model more realistic.
[0142] In one embodiment of this disclosure, the gradient acquisition module 303 is specifically used for:
[0143] The object image is subjected to high-pass filtering to obtain the detail pixels in the object image and the image gradient of the detail pixels.
[0144] In this scheme, high-pass filtering is applied to the object image to accurately obtain the detail pixels and their image gradients. By utilizing the more accurate image gradients of the detail pixels, the accuracy of subsequent processing can be improved.
[0145] In one embodiment of this disclosure, the real object is a face, and the object image is a face image.
[0146] In this scheme, object point clouds are generated based on each face image, and the positions of detail data points in the object point cloud are adjusted. Based on the adjusted object point cloud of the face, a face model can be reconstructed. Because the object point cloud used for reconstruction has been adjusted, the data points in the point cloud reflect the detailed information of the face more accurately, thus the reconstructed face model is more realistic.
[0147] It should be noted that the face model in this embodiment is not a face model for a specific user and cannot reflect the personal information of a specific user.
[0148] It should be noted that the object images in this embodiment are from a publicly available dataset.
[0149] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.
[0150] In one embodiment of this disclosure, an electronic device is provided, comprising:
[0151] At least one processor; and
[0152] A memory communicatively connected to the at least one processor; wherein,
[0153] The memory stores instructions that can be executed by the at least one processor, which enables the at least one processor to perform any of the model reconstruction methods in the foregoing method embodiments.
[0154] In one embodiment of this disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to perform any of the model reconstruction methods in the foregoing method embodiments.
[0155] In one embodiment of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements any of the model reconstruction methods in the foregoing method embodiments.
[0156] Figure 5 A schematic block diagram of an example electronic device 500 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.
[0157] like Figure 5 As shown, device 500 includes a computing unit 501, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 502 or a computer program loaded from storage unit 508 into random access memory (RAM) 503. RAM 503 may also store various programs and data required for the operation of device 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via bus 504. Input / output (I / O) interface 505 is also connected to bus 504.
[0158] Multiple components in device 500 are connected to I / O interface 505, including: input unit 506, such as keyboard, mouse, etc.; output unit 507, such as various types of monitors, speakers, etc.; storage unit 508, such as disk, optical disk, etc.; and communication unit 509, such as network card, modem, wireless transceiver, etc. Communication unit 509 allows device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.
[0159] The computing unit 501 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as the model reconstruction method. For example, in some embodiments, the model reconstruction method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of the model reconstruction method described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform the model reconstruction method by any other suitable means (e.g., by means of firmware).
[0160] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.
[0161] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.
[0162] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0163] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).
[0164] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.
[0165] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.
[0166] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.
[0167] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.
Claims
1. A model reconstruction method, comprising: The process involves obtaining multiple images of a real-world object, as well as the relative positions of the image acquisition devices when acquiring each object image, with each object image having a different acquisition angle. Based on the images of each object and their relative positional relationships, an object point cloud of the real-world object is generated; The object image is subjected to high-pass filtering to obtain the detail pixels in the object image and the image gradient of the detail pixels, wherein the detail pixels are the pixels in the object image that represent the detail information of the real object; Based on the image gradient of the detail pixel, the position of the detail data point in the object point cloud is adjusted, wherein the detail data point is: the data point corresponding to the detail pixel; Based on the adjusted positions of the detailed data points and the relative positional relationships, determine the mapped pixel values of the pixels obtained by mapping the detailed data points back to the object image; Based on the pixel value of the detail pixel and the mapped pixel value, the position of the detail data point is adjusted again, and the step of determining the mapped pixel value of the pixel obtained by mapping the detail data point back to the object image is returned until the adjustment end condition is met. Based on the adjusted object point cloud, reconstruct the object model of the real-world object.
2. The method according to claim 1, wherein, The step of readjusting the position of the detail data point based on the pixel value of the detail pixel and the mapped pixel value includes: Calculate the pixel value loss between the pixel value of the detailed pixel and the mapped pixel value; The position of the detail data points is adjusted based on the pixel value loss.
3. The method according to claim 2, wherein, The adjustment ends when the pixel value loss is less than a preset loss threshold.
4. The method according to any one of claims 1-3, wherein, The step of adjusting the position of detail data points in the object point cloud based on the image gradient of the detail pixels includes: Based on the image gradient of the detailed pixels, determine the adjustment range for adjusting the detailed data points in the object point cloud; The positions of the detail data points are adjusted in a predetermined direction along the object point cloud according to a determined adjustment range.
5. The method according to any one of claims 1-3, wherein, The step of adjusting the position of detail data points in the object point cloud based on the image gradient of the detail pixels includes: In the object point cloud, the target data point corresponding to the detail pixel is determined; The target data point and data points whose distance to the target data point is less than a preset distance threshold are identified as the detailed data points. The position of the detail data point is adjusted based on the image gradient of the detail pixel and the target distance, wherein the target distance is the distance between the detail data point and the target data point.
6. A model reconstruction apparatus, comprising: The information acquisition module is used to acquire multiple object images of the real object, as well as the relative positional relationship of the image acquisition devices when acquiring each object image, wherein the acquisition angle corresponding to each object image is different; The point cloud generation module is used to generate object point clouds of the real objects based on the images of each object and the relative positional relationships. The gradient acquisition module is used to perform high-pass filtering on the object image to obtain the detail pixels in the object image and the image gradient of the detail pixels, wherein the detail pixels are the pixels in the object image that represent the detail information of the real object. The position adjustment module is used to adjust the position of the detail data points in the object point cloud according to the image gradient of the detail pixel points, wherein the detail data points are data points corresponding to the detail pixel points; The pixel mapping module is used to determine the mapped pixel value of the pixel obtained by mapping the detail data points back to the object image based on the adjusted position of the detail data points and the relative positional relationship. The optimization and adjustment module is used to adjust the position of the detail data point again based on the pixel value of the detail pixel and the mapped pixel value, and return to the step of determining the mapped pixel value of the pixel obtained by mapping the detail data point back to the object image, until the adjustment end condition is met; The model reconstruction module is used to reconstruct the object model of the real-world object based on the adjusted object point cloud.
7. The apparatus according to claim 6, wherein, The optimization and adjustment module is specifically used for: Calculate the pixel value loss between the pixel value of the detail pixel and the mapped pixel value. Based on the pixel value loss, adjust the position of the detail data point and return to the step of determining the mapped pixel value of the pixel obtained by mapping the detail data point back to the object image, until the adjustment termination condition is met.
8. The apparatus according to claim 7, wherein, The adjustment ends when the pixel value loss is less than a preset loss threshold.
9. The apparatus according to any one of claims 6-8, wherein, The position adjustment module is specifically used for: Based on the image gradient of the detailed pixels, determine the adjustment range for adjusting the detailed data points in the object point cloud; The positions of the detail data points are adjusted in a predetermined direction along the object point cloud according to a determined adjustment range.
10. The apparatus according to any one of claims 6-8, wherein, The position adjustment module is specifically used for: In the object point cloud, the target data point corresponding to the detail pixel is determined; The target data point and data points whose distance to the target data point is less than a preset distance threshold are identified as the detailed data points. The position of the detail data point is adjusted based on the image gradient of the detail pixel and the target distance, wherein the target distance is the distance between the detail data point and the target data point.
11. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-5.
12. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-5.
13. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-5.
Citation Information
Patent Citations
Three-dimensional reconstruction method and device and terminal equipment
CN111640180A
Image processing method, image processing apparatus, and meta-cosmic three-dimensional reconstruction method
CN115131507A
