VR-based garment rendering method and system

Abstract

The invention provides a VR-based garment rendering method and system, and the method comprises the steps: obtaining a plurality of images of a garment shot from different perspectives, carrying out the feature matching and geometric information correction, selecting one image, and carrying out the triangulation, and obtaining a preliminarily recovered sparse point cloud structure diagram; performing pose estimation and light beam optimization on the point cloud structure diagram, sequentially adding images corresponding to other visual angles into the point cloud structure diagram after light beam optimization, and performing triangulation and light beam optimization to obtain a point cloud clothing diagram of the clothing and corresponding pose data; generating a three-dimensional Gaussian by taking each point of the point cloud clothing graph as a center, projecting the generated three-dimensional Gaussian to an image plane by utilizing a shooting pose, and calculating the light intensity from each point to the projection center based on a volume rendering algorithm to obtain a preliminary rendered image; and performing aliasing artifact elimination on the preliminary rendering image to obtain a target garment rendering image. The VR-based garment rendering method provided by the invention is high in efficiency and good in rendering quality.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and particularly to a VR-based clothing rendering method and system. Background Art

[0002] As a cutting-edge human-computer interaction technology, virtual reality (VR) technology has shown great potential in fields such as entertainment, education, and healthcare. VR technology utilizes data from real life, generates electronic signals through computer technology, and combines them with various output devices to transform them into phenomena that people can perceive.

[0003] When designing clothing, in order to provide high-fidelity virtual fitting services and support the display of clothing effects from multiple angles and in multiple scenarios, VR-based virtual simulation can achieve seamless integration of product design, build an immersive virtual fitting experience, and enhance the immersion and interactivity of the user experience; the construction and rendering of VR scenes are the key to achieving an immersive experience. Currently, traditional clothing rendering methods mainly rely on lidar to scan the surface of an object or scene and measure the time and intensity of reflected light to obtain three-dimensional data, and then use rendering software to render the obtained three-dimensional data. The modeling and rendering processes are separated, resulting in slow rendering speed and low image quality. Summary of the Invention

[0004] Based on this, the purpose of the present invention is to provide a VR-based clothing rendering method and system to solve the technical problems existing in the prior art.

[0005] The present invention provides a VR-based clothing rendering method, including: Obtaining a plurality of images of clothing taken from different perspectives, performing feature matching and geometric information correction on the plurality of images, selecting one image for triangulation, and obtaining a preliminarily restored sparse point cloud structure diagram; Performing pose estimation and beam optimization on the preliminarily restored sparse point cloud structure diagram, sequentially adding images from other perspectives to the point cloud structure diagram after beam optimization, and performing triangulation and beam optimization to obtain a point cloud clothing diagram of the clothing and corresponding pose data; Generating a three-dimensional Gaussian with the points of the point cloud clothing diagram as the center, projecting the generated three-dimensional Gaussian onto the image plane using the pose data, and calculating the light intensity of each point from the projection center based on the volume rendering algorithm to obtain a preliminary rendered image; Eliminating aliasing artifacts from the preliminary rendered image to obtain a target clothing rendered image.

[0006] Optionally, the step of sequentially adding images from other perspectives to the point cloud structure diagram after beam optimization, and performing triangulation and beam optimization to obtain a point cloud clothing diagram of the clothing and corresponding pose data includes: Map a number of images into a multi-modal semantic space through CLIP to obtain the encoded vector corresponding to each image, and calculate the similarity between the images corresponding to other perspectives and the image for triangulation initially according to the encoded vector; Select the image with the highest similarity among the images corresponding to other shooting perspectives as the conditional input for triangulation. Under the guidance of the initially restored sparse point cloud structure diagram, perform triangulation through an end-to-end generator model to obtain the current point cloud structure diagram from a new perspective and perform pose estimation; Iterate the current point cloud structure diagram as the input of the fractional distillation sampling algorithm, and batch select some light ray clusters in the current point cloud structure diagram for beam optimization based on the frequency domain consistency error; Perform triangulation and beam optimization on the remaining images among the images corresponding to other perspectives in sequence to obtain the point cloud clothing diagram of the clothing and the corresponding pose data.

[0007] Optionally, the sampling loss expression of the fractional distillation sampling algorithm is:

[0008] In the formula, represents the multi-modal semantic space, represents the input current point cloud structure diagram, represents the model parameters, represents the light ray cluster corresponding to the current point cloud structure diagram, () represents the expected solution, represents the time step weight, represents the conditional vector for iteration, represents the image implicit vector corresponding to the current point cloud structure diagram mapped from the pixel space to the implicit vector space, represents the noise estimation guided by the fractional distillation sampling algorithm including the conditional vector at time t, represents the noise estimation guided by the fractional distillation sampling algorithm not including the conditional vector at time t.

[0009] Optionally, the steps of projecting the generated three-dimensional Gaussian onto the image plane based on the pose data and calculating the light intensity of each point to the projection center based on the volume rendering algorithm include: Determine the influence range when projecting the voxel corresponding to each point in the point cloud clothing diagram based on the footprint function, and further determine the overall contribution degree of each point in the point cloud clothing diagram to the image; Construct projection rays corresponding to each point in the point cloud clothing diagram to the image plane, and calculate the light intensity of the projection rays corresponding to each point to the projection center based on the sputtering diffusion equation of the volume rendering algorithm, so as to determine the color and brightness of each pixel projection; Overlay the colors and brightness of all pixel projections to obtain a preliminary rendered image.

[0010] Optionally, the expression of the sputtering diffusion equation is:

[0011] In the formula, represents the light wavelength, represents the projection ray passing through the projection center point and the plane point of the projection ray, represents the distance from the projection center to the ray corresponding to the k point in the point cloud clothing diagram, represents the luminous coefficient of the k point, is the extinction coefficient of the k point, represents the reconstruction kernel of the k point, which is used to reflect the position and shape of individual particles, represents the Taylor operation, is the extinction coefficient of the j point in the point cloud clothing diagram, represents the reconstruction kernel of the j point.

[0012] The present invention also provides a VR-based clothing rendering system, including: A measurement module that acquires a number of images of the clothing taken from different perspectives, performs feature matching and geometric information correction on the number of images, selects one image for triangulation, and obtains a preliminary restored sparse point cloud structure diagram; An optimization module for performing pose estimation and beam optimization on the preliminary restored sparse point cloud structure diagram, sequentially adding images from other perspectives to the point cloud structure diagram after beam optimization and performing triangulation and beam optimization to obtain the point cloud clothing diagram of the clothing and the corresponding pose data; A calculation module for generating a three-dimensional Gaussian centered on the points of the point cloud clothing diagram, projecting the generated three-dimensional Gaussian onto the image plane using the pose data, and calculating the light intensity from each point to the projection center based on the volume rendering algorithm to obtain a preliminary rendered image; An elimination module for eliminating aliasing artifacts in the preliminary rendered image to obtain a target clothing rendering image.

[0013] Optionally, the optimization module includes: An encoding unit for mapping a number of images through CLIP into a multi-modal semantic space to obtain an encoding vector corresponding to each image, and calculating the similarity between the images corresponding to other perspectives and the image initially used for triangulation according to the encoding vector; An estimation unit is configured to select the image with the highest similarity among the images corresponding to other shooting perspectives as the conditional input for triangulation. Under the guidance of the initially recovered sparse point cloud structure diagram, triangulation is performed through an end-to-end generator model to obtain the current point cloud structure diagram from a new perspective and perform pose estimation; An optimization unit is configured to iterate by using the current point cloud structure diagram as the input of the fractional distillation sampling algorithm, and batch select partial light clusters in the current point cloud structure diagram based on the frequency domain consistency error for beam optimization; An iteration unit is configured to sequentially perform triangulation and beam optimization on the remaining images corresponding to other perspectives to obtain the point cloud clothing diagram of the clothing and the corresponding pose data.

[0014] Optionally, the sampling loss expression of the fractional distillation sampling algorithm is:

[0015] In the formula, represents the multi-modal semantic space, represents the input current point cloud structure diagram, represents the model parameters, represents the light cluster corresponding to the current point cloud structure diagram, () represents the expected solution, represents the time step weight, represents the conditional vector for iteration, represents the image implicit vector corresponding to the mapping of the current point cloud structure diagram from the pixel space to the implicit vector space, represents the noise estimation guided by the fractional distillation sampling algorithm including the conditional vector at time t, represents the noise estimation guided by the fractional distillation sampling algorithm not including the conditional vector at time t.

[0016] Optionally, the calculation module includes: A determination unit is configured to determine the influence range when each point corresponding to the voxel in the point cloud clothing diagram is projected based on the footprint function, and further determine the overall contribution degree of each point in the point cloud clothing diagram to the image; A construction unit is configured to construct projection rays corresponding to each point in the point cloud clothing diagram to the image plane, and calculate the light intensity from each point corresponding to the projection ray to the projection center based on the sputtering diffusion equation of the volume rendering algorithm, so as to determine the color and brightness of each pixel projection; An overlay unit is configured to overlay the colors and brightnesses of all pixel projections to obtain a preliminary rendered image.

[0017] Optionally, the expression of the sputtering diffusion equation is:

[0018] In the formula, represents the light wavelength, represents the projection ray passing through the projection center point and the plane point of the projection ray, represents the distance from the projection center to the ray corresponding to point k in the point cloud clothing diagram, represents the luminous coefficient of point k, is the extinction coefficient of point k, represents the reconstruction kernel of point k, which is used to reflect the position and shape of individual particles, represents the Taylor operation, is the extinction coefficient of point j in the point cloud clothing diagram, represents the reconstruction kernel of point j.

[0019] The beneficial effects of the present invention compared with the prior art are as follows: The VR-based clothing rendering method provided by this application acquires a number of images of the clothing taken from different perspectives, performs feature matching and geometric information correction on the number of images, selects one image for triangulation to obtain a preliminarily restored sparse point cloud structure diagram; performs pose estimation and beam optimization on the preliminarily restored sparse point cloud structure diagram, and sequentially adds images corresponding to other perspectives to the point cloud structure diagram after beam optimization for triangulation and beam optimization to obtain the point cloud clothing diagram of the clothing and the corresponding pose data; first constructs the point cloud clothing diagram of the clothing in a new perspective according to the images from different perspectives; generates a three-dimensional Gaussian centered on each point of the point cloud clothing diagram, projects the generated three-dimensional Gaussian onto the image plane using the shooting pose, calculates the light intensity from each point to the projection center based on the volume rendering algorithm to obtain a preliminary rendered image; generates a three-dimensional Gaussian for each point in the point cloud clothing diagram of the clothing based on the neural radiance field, projects according to the shooting pose, and then calculates the light intensity from each point to the projection center based on the volume rendering algorithm to reduce the rendering calculation amount, and finally eliminates the aliasing artifacts in the preliminary rendered image to improve the quality of the rendered image to obtain the target clothing rendered image; the VR-based clothing rendering method provided by this application renders according to the point cloud clothing diagram, with less calculation amount, high rendering efficiency and high rendering quality.

[0020] Additional aspects and advantages of the present invention will be given in part in the following description, become apparent in part from the following description, or be understood through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is the flowchart of the VR-based clothing rendering method in Embodiment 1 of the present invention; Figure 2 is the structural block diagram of the computer in Embodiment 4 of the present invention.

[0022] The following specific embodiments will further illustrate the present invention in conjunction with the above-mentioned drawings. Detailed implementation manners

[0023] To facilitate the understanding of the present invention, the present invention will be described more comprehensively below with reference to the relevant drawings. Several embodiments of the present invention are shown in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, these embodiments are provided to make the disclosure of the present invention more thorough and comprehensive.

[0024] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present invention belongs. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0025] Embodiment 1 Please refer to Figure 1 , which shows the VR-based clothing rendering method in the first embodiment of the present invention. The VR-based clothing rendering method specifically includes steps S10 to S40: S10, obtain several images of the clothing taken from different perspectives, perform feature matching and geometric information correction on the several images, select one image for triangulation, and obtain a preliminarily restored sparse point cloud structure diagram; In specific implementation, first obtain the clothing images taken from different perspectives. The three-dimensional information of the clothing can be captured through multi-perspective shooting and image matching. Then, a deep neural network is used to simulate the optical principle for modeling, and the multi-layer perception radiation mechanism in the deep neural network is used to effectively represent the appearance of geometric figures related to the perspective, and learn the color and density information of each point in the scene, so as to highly realistically reconstruct the three-dimensional scene; a preliminarily restored sparse point cloud structure diagram is obtained through triangulation of multi-perspective images.

[0026] S20, perform pose estimation and beam optimization on the preliminarily restored sparse point cloud structure diagram, sequentially add images from other perspectives to the point cloud structure diagram after beam optimization, and perform triangulation and beam optimization to obtain a point cloud clothing diagram of the clothing and corresponding pose data; The step of sequentially adding images from other perspectives to the point cloud structure diagram after beam optimization, and performing triangulation and beam optimization to obtain a point cloud clothing diagram of the clothing and corresponding pose data includes: Map several images into a multi-modal semantic space through CLIP to obtain an encoded vector corresponding to each image, and calculate the similarity between the images corresponding to other perspectives and the image initially used for triangulation according to the encoded vector; Select the image with the highest similarity among the images corresponding to other shooting perspectives as the conditional input for triangulation. Under the guidance of the initially restored sparse point cloud structure diagram, perform triangulation through an end-to-end generator model to obtain the current point cloud structure diagram in the new perspective and perform pose estimation; Iterate by using the current point cloud structure diagram as the input of the fractional distillation sampling algorithm, and batch select some light clusters in the current point cloud structure diagram for beam optimization based on the frequency domain consistency error; Perform triangulation and beam optimization on the remaining images corresponding to other perspectives in sequence to obtain the point cloud clothing diagram of the clothing and the corresponding pose data.

[0027] The sampling loss expression of the fractional distillation sampling algorithm is:

[0028] In the formula, represents the multi-modal semantic space, represents the input current point cloud structure diagram, represents the model parameters, represents the light cluster corresponding to the current point cloud structure diagram, () represents the expected solution, represents the time step weight, represents the conditional vector of the iteration, represents the image implicit vector corresponding to the mapping of the current point cloud structure diagram from the pixel space to the implicit vector space, represents the noise estimation guided by the fractional distillation sampling algorithm including the conditional vector at time t, represents the noise estimation guided by the fractional distillation sampling algorithm without the conditional vector at time t.

[0029] In this embodiment, after mapping images from different perspectives into a multi-modal semantic space, similarity calculation is performed. According to the similarity levels, the images from different perspectives are sequentially used as conditional inputs for triangulation. Under the guidance of the initially recovered sparse point cloud structure diagram, triangulation is carried out through an end-to-end generator model to obtain the current point cloud structure diagram from the combined new perspective and perform pose estimation; then batch beam optimization is performed on the current point cloud structure diagram; schematically, assuming that the set of images from several perspectives is: A = (A1, A2, A3...), first obtain the point cloud structure diagram corresponding to A1, then use A2 as the conditional input for the second triangulation. Under the guidance of the point cloud structure diagram corresponding to A1, generate the point cloud structure diagram B1. The perspective of B1 is different from both A1 and A2 and is the point cloud structure diagram from the new perspective. The pose of B1 can be estimated through the software COLMAP; then select A3 as the input. Under the guidance of the point cloud structure diagram corresponding to B1, generate the point cloud structure diagram B2, and so on, to obtain the point cloud clothing diagram from the finally synthesized new perspective of the clothing. S30. Generate a three-dimensional Gaussian centered on the points of the point cloud clothing diagram, project the generated three-dimensional Gaussian onto the image plane using the pose data, and calculate the light intensity of each point to the projection center based on the volume rendering algorithm to obtain a preliminary rendered image; Starting from the obtained point cloud structure diagram, perform the reconstruction and rendering of the clothing based on 3D Gaussian sputtering rendering. The three-dimensional Gaussian can describe and represent data in the three-dimensional space, improving the richness of data description; through the parameters of the three-dimensional Gaussian, the accurate position of the point and attribute information such as the shape and transparency of the point can be obtained; for example, the coordinates of the three-dimensional Gaussian can describe the position of the point, the covariance matrix can describe the shape and direction, and the transparency can be used for rendering.

[0030] The steps of projecting the generated three-dimensional Gaussian onto the image plane using the pose data and calculating the light intensity of each point to the projection center based on the volume rendering algorithm include: Based on the footprint function, determine the influence range when projecting the voxels corresponding to each point in the point cloud clothing diagram, and then determine the overall contribution of each point in the point cloud clothing diagram to the image; Construct projection rays corresponding to each point in the point cloud clothing diagram to the image plane, and calculate the light intensity of the projection rays corresponding to each point to the projection center based on the sputtering diffusion equation of the volume rendering algorithm, so as to determine the color and brightness of each pixel projection; Overlay the colors and brightness of all pixel projections to obtain a preliminary rendered image.

[0031] The expression of the sputtering diffusion equation is:

[0032] In the formula, represents the light wavelength, represents passing through the projection center point and the planar point of the projection ray, represents the distance on the ray corresponding to point k in the point cloud clothing diagram from the projection center, represents the luminous coefficient of point k, is the extinction coefficient of point k, represents the reconstruction kernel of point k, which is used to reflect the position and shape of individual particles, represents the Taylor operation, is the extinction coefficient of point j in the point cloud clothing diagram, represents the reconstruction kernel of point j.

[0033] In this embodiment, the region of the image plane covers a three-dimensional Gaussian footprint of the projection. Based on volume reconstruction, a linear volume integration is performed on the projection in the 2D plane. By projecting the three-dimensional Gaussian corresponding to each point in the point cloud clothing diagram onto the 2D image, during volume integration rendering, only two-dimensional convolution calculation is required for each sampling point along the projection ray. Compared with the conventional object rendering that requires three-dimensional convolution integration, the rendering efficiency is greatly improved.

[0034] S40. Eliminate the aliasing artifacts in the preliminary rendered image to obtain the target clothing rendered image.

[0035] In specific implementation, when the rendered image or a part of the image is sampled onto a discrete raster grid, due to the difference between the low sampling frequency and the high signal frequency, aliasing will occur during the rendering process, resulting in visual artifacts and affecting the quality of the rendered image. Therefore, it is necessary to eliminate the aliasing artifacts. In specific implementation, the artifacts can be eliminated based on a low-pass filter. The specific method is as follows: evenly distribute the light intensity of each pixel point from its surrounding area to the pixel point itself to perform an accurate approximation of a single pixel. Compared with the conventional low-pass filter, performing an accurate approximation of a single pixel helps to reduce the computational complexity, while removing the high-frequency details in the rendered image, effectively reducing the aliasing effect and improving the quality of the rendered image.

[0036] In summary, the VR-based clothing rendering method provided by this application obtains several images of the clothing taken from different perspectives, performs feature matching and geometric information correction on the several images, selects one image for triangulation to obtain a preliminarily restored sparse point cloud structure diagram; performs pose estimation and beam optimization on the preliminarily restored sparse point cloud structure diagram, sequentially adds images corresponding to other perspectives to the point cloud structure diagram after beam optimization and performs triangulation and beam optimization to obtain the point cloud clothing diagram of the clothing and the corresponding pose data; first constructs the point cloud clothing diagram of the clothing in a new perspective according to the images from different perspectives; generates a three-dimensional Gaussian centered on each point of the point cloud clothing diagram, projects the generated three-dimensional Gaussian onto the image plane using the shooting pose, calculates the light intensity of each point to the projection center based on the volume rendering algorithm to obtain a preliminary rendered image; generates a three-dimensional Gaussian for each point in the point cloud clothing diagram of the clothing based on the neural radiance field, projects it according to the shooting pose, and then calculates the light intensity of each point to the projection center based on the volume rendering algorithm to reduce the rendering calculation amount, and finally eliminates the aliasing artifacts of the preliminary rendered image to improve the quality of the rendered image and obtain the target clothing rendered image; the VR-based clothing rendering method provided by this application performs rendering according to the point cloud clothing diagram, with less calculation amount, high rendering efficiency and high rendering quality.

[0037] Embodiment 2 This embodiment provides a VR-based clothing rendering system, including: A measurement module that obtains several images of the clothing taken from different perspectives, performs feature matching and geometric information correction on the several images, selects one image for triangulation, and obtains a preliminarily restored sparse point cloud structure diagram; An optimization module for performing pose estimation and beam optimization on the preliminarily restored sparse point cloud structure diagram, sequentially adding images corresponding to other perspectives to the point cloud structure diagram after beam optimization and performing triangulation and beam optimization to obtain the point cloud clothing diagram of the clothing and the corresponding pose data; A calculation module for generating a three-dimensional Gaussian centered on the points of the point cloud clothing diagram, projecting the generated three-dimensional Gaussian onto the image plane using the pose data, and calculating the light intensity of each point to the projection center based on the volume rendering algorithm to obtain a preliminary rendered image; An elimination module for eliminating the aliasing artifacts of the preliminary rendered image to obtain the target clothing rendered image.

[0038] Optionally, the optimization module includes: An encoding unit for mapping several images through CLIP into a multi-modal semantic space to obtain an encoding vector corresponding to each image, and calculating the similarity between the images corresponding to other perspectives and the image initially used for triangulation according to the encoding vector; An estimation unit is configured to select the image with the highest similarity among the images corresponding to other shooting perspectives as the conditional input for triangulation. Under the guidance of the initially recovered sparse point cloud structure diagram, triangulation is performed through an end-to-end generator model to obtain the current point cloud structure diagram from a new perspective and perform pose estimation; An optimization unit is configured to perform iteration by using the current point cloud structure diagram as the input of the fractional distillation sampling algorithm, and batch-select some light clusters in the current point cloud structure diagram based on the frequency domain consistency error for beam optimization; An iteration unit is configured to sequentially perform triangulation and beam optimization on the remaining images corresponding to other perspectives to obtain the point cloud clothing diagram of the clothing and the corresponding pose data.

[0039] Optionally, the sampling loss expression of the fractional distillation sampling algorithm is:

[0040] In the formula, represents the multi-modal semantic space, represents the input current point cloud structure diagram, represents the model parameters, represents the light cluster corresponding to the current point cloud structure diagram, () represents the expected solution, represents the time step weight, represents the conditional vector for iteration, represents the image implicit vector corresponding to the mapping of the current point cloud structure diagram from the pixel space to the implicit vector space, represents the noise estimation guided by the fractional distillation sampling algorithm including the conditional vector at time t, represents the noise estimation guided by the fractional distillation sampling algorithm not including the conditional vector at time t.

[0041] Optionally, the calculation module includes: A determination unit is configured to determine the influence range when each point in the point cloud clothing diagram is projected onto the corresponding voxel based on the footprint function, and further determine the overall contribution degree of each point in the point cloud clothing diagram to the image; A construction unit is configured to construct a projection ray corresponding to each point in the point cloud clothing diagram to the image plane, and calculate the light intensity of each point corresponding to the projection ray to the projection center based on the sputtering diffusion equation of the volume rendering algorithm, so as to determine the color and brightness of each pixel projection; A superposition unit is configured to superpose the colors and brightnesses of all pixel projections to obtain a preliminary rendered image.

[0042] Optionally, the expression of the sputtering diffusion equation is:

[0043] In the formula, represents the light wavelength, represents the projection ray passing through the projection center point and the plane point ; represents the distance from the projection center to the ray corresponding to point k in the point cloud clothing diagram, represents the luminous coefficient of point k, is the extinction coefficient of point k, represents the reconstruction kernel of point k, which is used to reflect the position and shape of individual particles, represents the Taylor operation, is the extinction coefficient of point j in the point cloud clothing diagram, represents the reconstruction kernel of point j.

[0044] Embodiment III This embodiment provides a storage medium on which a computer program is stored. When the program is executed by a processor, it implements the VR-based clothing rendering method as described above.

[0045] Embodiment IV The present invention also provides a computer. Please refer to Figure 2 . As shown in the computer in the embodiment of the present invention, it includes a memory 10, a processor 20, and a computer program 30 stored on the memory 10 and executable on the processor 20. When the processor 20 executes the computer program 30, it implements the VR-based clothing rendering method as described above.

[0046] Among them, the memory 10 includes at least one type of storage medium. The storage medium includes flash memory, hard disk, multimedia card, card-type memory (such as SD or DX memory, etc.), magnetic memory, magnetic disk, optical disc, etc. The memory 10 can be an internal storage unit of the computer in some embodiments, such as the hard disk of the computer. The memory 10 can also be an external storage device in other embodiments, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. Further, the memory 10 can also include both the internal storage unit of the computer and the external storage device. The memory 10 can be used not only to store application software installed on the computer and various types of data, but also to temporarily store data that has been output or will be output.

[0047] Among them, in some embodiments, the processor 20 may be an Electronic Control Unit (ECU, also known as an on-board computer), a Central Processing Unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips, which are used to run the program code stored in the memory 10 or process data, such as executing an access restriction program and the like.

[0048] It should be noted that Figure 2 the structure shown does not constitute a limitation on the computer. In other embodiments, the computer may include fewer or more components than shown in the figure, or combine certain components, or have a different component arrangement.

[0049] Those skilled in the art can understand that the logic and / or steps represented in the flowchart or described in other ways herein, for example, can be considered as a predefined sequence list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device), or in combination with these instruction execution systems, apparatus, or devices. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device.

[0050] More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection portion (electronic device) having one or more wirings, a portable computer disk cartridge (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpretation, or other suitable processing as necessary, and then storing it in a computer memory.

[0051] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application specific integrated circuits having appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.

[0052] The technical features of the above-described embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the various technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as falling within the scope described in this specification.

[0053] The above-described embodiments merely represent several implementation manners of the present application. The description is relatively specific and detailed, but it should not be construed as a limitation on the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the appended claims.

Claims

1. A VR-based clothing rendering method, characterized in that a number of images of the clothing taken from different perspectives are obtained, feature matching and geometric information correction are performed on the number of images, and one image is selected for triangulation to obtain a preliminarily restored sparse point cloud structure diagram; pose estimation and beam optimization are performed on the preliminarily restored sparse point cloud structure diagram, and images from other perspectives are successively added to the point cloud structure diagram after beam optimization for triangulation and beam optimization to obtain a point cloud clothing diagram of the clothing and corresponding pose data; a three-dimensional Gaussian is generated with the points of the point cloud clothing diagram as the center, the generated three-dimensional Gaussian is projected onto the image plane using the pose data, and the light intensity from each point to the projection center is calculated based on the volume rendering algorithm to obtain a preliminary rendered image; aliasing artifacts are removed from the preliminary rendered image to obtain a target clothing rendered image.

2. The VR-based clothing rendering method according to claim 1, wherein The step of successively adding images from other perspectives to the point cloud structure diagram after beam optimization for triangulation and beam optimization to obtain a point cloud clothing diagram of the clothing and corresponding pose data includes: mapping a number of images through CLIP into a multi-modal semantic space to obtain an encoded vector corresponding to each image, and calculating the similarity between the image corresponding to other perspectives and the image initially used for triangulation based on the encoded vector; selecting the image with the highest similarity among the images corresponding to other shooting perspectives as the conditional input for triangulation, and performing triangulation through an end-to-end generator model under the guidance of the preliminarily restored sparse point cloud structure diagram to obtain the current point cloud structure diagram from a new perspective and performing pose estimation; iterating the current point cloud structure diagram as the input of the fractional distillation sampling algorithm, and batch-selecting some light clusters in the current point cloud structure diagram for beam optimization based on the frequency domain consistency error; successively performing triangulation and beam optimization on the remaining images corresponding to other perspectives to obtain a point cloud clothing diagram of the clothing and corresponding pose data.

3. The VR-based clothing rendering method according to claim 2, characterized in that the sampling loss expression of the fractional distillation sampling algorithm is: In the formula, represents the multi-modal semantic space, represents the current point cloud structure diagram of the input, represents the model parameters, represents the ray cluster corresponding to the current point cloud structure diagram, () represents the expected solution, represents the time step weight, represents the conditional vector of the iteration, represents the image implicit vector corresponding to the mapping of the current point cloud structure diagram from the pixel space to the implicit vector space, represents the noise estimation guided by the conditional vector fractional distillation sampling algorithm at time t, represents the noise estimation without the guidance of the conditional vector fractional distillation sampling algorithm at time t.

4. The VR-based clothing rendering method according to claim 1, wherein The step of projecting the generated three-dimensional Gaussian onto the image plane using the pose data and calculating the light intensity from each point to the projection center based on the volume rendering algorithm includes: determining the influence range when projecting the voxel corresponding to each point in the point cloud clothing diagram based on the footprint function, and further determining the overall contribution degree of each point in the point cloud clothing diagram to the image; constructing a projection ray corresponding to each point in the point cloud clothing diagram to the image plane, and calculating the light intensity from the projection ray corresponding to each point to the projection center based on the sputtering diffusion equation of the volume rendering algorithm, so as to determine the color and brightness of each pixel projection; superimposing the colors and brightness of all pixel projections to obtain a preliminary rendered image.

5. The VR-based clothing rendering method according to claim 4, wherein The expression of the sputtering diffusion equation is: In the formula, represents the light wavelength, represents the projection ray passing through the projection center point and the plane point ; represents the distance from the projection center to the ray corresponding to point k in the point cloud clothing diagram, represents the luminous coefficient of point k, is the extinction coefficient of point k, represents the reconstruction kernel of point k, which is used to reflect the position and shape of individual particles, represents the Taylor operation, is the extinction coefficient of point j in the point cloud clothing diagram, represents the reconstruction kernel of point j.

6. A VR-based clothing rendering system, characterized in that, including: a measurement module, which obtains a number of images of the clothing taken from different perspectives, performs feature matching and geometric information correction on the number of images, selects one image for triangulation, and obtains a preliminarily restored sparse point cloud structure diagram; An optimization module, configured to perform pose estimation and beam optimization on the preliminarily restored sparse point cloud structure diagram, sequentially add images from other perspectives to the point cloud structure diagram after beam optimization, and perform triangulation and beam optimization to obtain the point cloud clothing diagram of the clothing and the corresponding pose data; A calculation module, configured to generate a three-dimensional Gaussian centered on the points of the point cloud clothing diagram, project the generated three-dimensional Gaussian onto the image plane using the pose data, and calculate the light intensity of each point to the projection center based on the volume rendering algorithm to obtain a preliminary rendered image; An elimination module, configured to eliminate aliasing artifacts from the preliminary rendered image to obtain a target clothing rendered image.

7. The VR-based clothing rendering system according to claim 6, wherein, The optimization module includes: An encoding unit, configured to map a plurality of images into a multi-modal semantic space through CLIP to obtain an encoding vector corresponding to each image, and calculate the similarity between the images corresponding to other perspectives and the images initially used for triangulation based on the encoding vector; An estimation unit, configured to select the image with the highest similarity among the images corresponding to other shooting perspectives as the conditional input for triangulation, and perform triangulation through an end-to-end generator model under the guidance of the preliminarily restored sparse point cloud structure diagram to obtain the current point cloud structure diagram in a new perspective and perform pose estimation; An optimization unit, configured to iterate the current point cloud structure diagram as the input of the fractional distillation sampling algorithm, and batch-select partial light ray clusters in the current point cloud structure diagram for beam optimization based on the frequency domain consistency error; An iteration unit, configured to sequentially perform triangulation and beam optimization on the remaining images corresponding to other perspectives to obtain the point cloud clothing diagram of the clothing and the corresponding pose data.

8. The VR-based clothing rendering system according to claim 7, wherein The sampling loss expression of the fractional distillation sampling algorithm is: In the formula, represents the multi-modal semantic space, represents the current point cloud structure diagram of the input, represents the model parameters, represents the light ray cluster corresponding to the current point cloud structure diagram, () represents the expected solution, represents the time step weight, represents the conditional vector of the iteration, represents the image implicit vector corresponding to the mapping of the current point cloud structure diagram from the pixel space to the implicit vector space, represents the noise estimation guided by the conditional vector fractional distillation sampling algorithm at time t, represents the noise estimation without the guidance of the conditional vector fractional distillation sampling algorithm at time t.

9. The VR-based clothing rendering system according to claim 6, wherein The calculation module includes: A determination unit, configured to determine the influence range when the voxels corresponding to each point in the point cloud clothing diagram are projected based on the footprint function, and further determine the overall contribution degree of each point in the point cloud clothing diagram to the image; A construction unit, configured to construct projection rays corresponding to each point in the point cloud clothing diagram onto the image plane, and calculate the light intensity of the projection rays corresponding to each point to the projection center based on the sputtering diffusion equation of the volume rendering algorithm, so as to determine the color and brightness of each pixel projection; An overlay unit, configured to overlay the colors and brightness of all pixel projections to obtain a preliminary rendered image.

10. The VR-based clothing rendering system according to claim 9, wherein The expression of the sputtering diffusion equation is: In the formula, represents the light wavelength, represents the projection ray passing through the projection center point and the plane point of the projection ray, represents the distance from the projection center to the ray corresponding to the k-th point in the point cloud clothing diagram, represents the light emission coefficient of the k-th point, is the extinction coefficient of the k-th point, represents the reconstruction kernel of the k-th point, which is used to reflect the position and shape of individual particles, represents the Taylor operation, is the extinction coefficient of the j-th point in the point cloud clothing diagram, represents the reconstruction kernel of the j-th point.

Images