Method, device, equipment and medium for generating three-dimensional model based on polarization information

Abstract

The embodiments of the present disclosure disclose a method, apparatus, device, and medium for generating a three-dimensional model based on polarization information. A specific implementation of the method includes: receiving information of each polarization image; generating information of each camera position; generating information of each sampling light direction; generating each sampling point; determining the sampling points that meet the preset surface point conditions among the sampling points as surface points to obtain each surface point; generating information of each light reflection prediction; generating information of each pixel color prediction; generating information of each light reflection; generating a camera position group for each surface point; generating each perspective inconsistency point; generating each reflection confidence information; generating a three-dimensional model network; and storing the three-dimensional model network in a data storage server. This implementation improves the effect of expressing the complex surface features of an object and improves the applicability to objects with complex media.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and more particularly to a method, apparatus, device, and medium for generating a three-dimensional model based on polarization information. Background Art

[0002] The 3D model generation method for objects with special materials is a method for reconstructing models of special materials (for example, transparent materials or materials with highlights on the surface). The commonly used method currently is the 3D reconstruction method based on neural radiation fields.

[0003] However, when using the above method to generate a 3D model of an object made of a special material, the following technical problems often occur:

[0004] In neural radiation field-based 3D reconstruction technology, light is primarily transmitted in straight lines, making it less effective at depicting complex surface features (such as transparency). Furthermore, because pixel color prediction is performed through volume rendering using direct light, the predicted pixel colors for objects in complex media do not match the actual colors, resulting in poor applicability.

[0005] The above information disclosed in this Background section is only for enhancement of understanding of the background of the inventive concept and therefore it may contain information that does not form the prior art that is already known to a person of ordinary skill in the art. Summary of the Invention

[0006] The content of this disclosure is used to briefly introduce concepts that will be described in detail in the detailed description section below. The content of this disclosure is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0007] Some embodiments of the present disclosure propose a method, device, electronic device, and computer-readable medium for generating a three-dimensional model based on polarization information to solve one or more of the technical problems mentioned in the above background technology section.

[0008] In a first aspect, some embodiments of the present disclosure provide a method for generating a three-dimensional model based on polarization information, the method comprising: receiving various polarization picture information, wherein each polarization picture information in the above-mentioned various polarization picture information comprises a polarization intensity picture, a polarization angle picture and a color picture; generating various camera pose information based on the above-mentioned various polarization picture information, wherein each camera pose information in the above-mentioned various camera pose information comprises camera intrinsic parameter information, camera extrinsic parameter information and camera origin information; generating various sampling light direction information based on the above-mentioned various camera pose information; generating various sampling points based on the above-mentioned various sampling light direction information and a pre-trained signed distance network; determining the sampling points in the above-mentioned various sampling points that meet the preset surface point conditions as surface points to obtain various surface points; generating various light reflection prediction information based on the above-mentioned various surface points and a pre-trained light decomposition network, wherein each light reflection prediction information in the above-mentioned various light reflection prediction information comprises mirror reflection Based on the surface points and the pre-trained color network, each pixel color prediction information is generated; based on the light reflection prediction information, the pre-stored coordinate system transformation matrix information and the pre-stored polarization matrix information, each light reflection information is generated, wherein each light reflection information in the light reflection information includes specular reflection information, diffuse reflection component information and specular vector information; based on the surface points, the sampling light direction information and the camera pose information, each surface point camera pose group is generated; based on the surface point camera pose group, each perspective inconsistency point is generated; based on the surface points, each reflection confidence information is generated; based on the pixel color prediction information, the light reflection information, the perspective inconsistency point, the reflection confidence information and the polarization picture information, a three-dimensional model network is generated; the three-dimensional model network is stored in a data storage server.

[0009] In a second aspect, some embodiments of the present disclosure provide a three-dimensional model generation device based on polarization information, the device comprising: a receiving unit configured to receive various polarization picture information, wherein each polarization picture information in the above-mentioned various polarization picture information includes a polarization intensity picture, a polarization angle picture and a color picture; a first generating unit configured to generate various camera pose information based on the above-mentioned various polarization picture information, wherein each camera pose information in the above-mentioned various camera pose information includes camera intrinsic parameter information, camera extrinsic parameter information and camera origin information; a second generating unit configured to generate various sampling light direction information based on the above-mentioned various camera pose information; a third generating unit configured to generate various sampling points based on the above-mentioned various sampling light direction information and a pre-trained signed distance network; a determining unit configured to determine the sampling points in the above-mentioned various sampling points that meet the preset surface point conditions as surface points to obtain various surface points; a fourth generating unit configured to generate various light reflection prediction information based on the above-mentioned various surface points and a pre-trained light decomposition network, wherein each light reflection prediction information in the above-mentioned various light reflection prediction information includes specular reflection prediction information and diffuse reflection prediction information. reflection component prediction information; a fifth generation unit, configured to generate each pixel color prediction information based on the above-mentioned each surface point and a pre-trained color network; a sixth generation unit, configured to generate each light reflection information based on the above-mentioned each light reflection prediction information, pre-stored coordinate system transformation matrix information and pre-stored polarization matrix information, wherein each light reflection information in the above-mentioned each light reflection information includes specular reflection information, diffuse reflection component information and specular vector information; a seventh generation unit, configured to generate each surface point camera pose group based on the above-mentioned each surface point, the above-mentioned each sampling light direction information and the above-mentioned each camera pose information; an eighth generation unit, configured to generate each perspective inconsistency point based on the above-mentioned each surface point camera pose group; a ninth generation unit, configured to generate each reflection confidence information based on the above-mentioned each surface point; a tenth generation unit, configured to generate a three-dimensional model network based on the above-mentioned each pixel color prediction information, the above-mentioned each light reflection information, the above-mentioned each perspective inconsistency point, the above-mentioned each reflection confidence information and the above-mentioned each polarization picture information; a storage unit, configured to store the above-mentioned three-dimensional model network in a data storage server.

[0010] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by one or more processors, the one or more processors implement the method described in any implementation of the first aspect above.

[0011] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation of the first aspect is implemented.

[0012] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: through the three-dimensional model generation method based on polarization information of some embodiments of the present disclosure, the way of expressing other complex surface features of an object and the applicability to objects of complex media are improved. Specifically, the reason for the poor way of solving other complex surface features of an object is that the light in the three-dimensional reconstruction technology based on the neural radiation field is mainly transmitted in a straight line, and the effect of expressing other complex surface features of the object (such as transparency) is poor. And because the pixel color is predicted by volume rendering of direct light, when predicting the pixel color of an object of a complex medium, a predicted color that does not match the actual color is generated, resulting in poor applicability to objects of complex media. Based on this, the three-dimensional model generation method based on polarization information of some embodiments of the present disclosure first receives each polarization picture information, wherein each polarization picture information in the above-mentioned each polarization picture information includes a polarization intensity picture, a polarization angle picture and a color picture. In this way, a picture with polarization information can be obtained. Then, based on the polarization image information, camera pose information is generated. Each of the camera pose information includes camera intrinsic parameter information, camera extrinsic parameter information, and camera origin information. Thus, the pose information of the camera corresponding to each polarization image in the world coordinate system can be obtained. Then, based on the camera pose information, sampled light direction information is generated. Thus, the direction of each sampled light can be obtained. Next, based on the sampled light direction information and a pre-trained signed distance network, sampling points are generated. Thus, sampling points associated with the three-dimensional model are obtained. Then, among the sampling points, those that meet preset surface point conditions are determined as surface points, thereby obtaining surface points. Thus, surface points on the three-dimensional model surface are obtained. Then, based on the surface points and a pre-trained light decomposition network, light reflection prediction information is generated. Each of the light reflection prediction information includes specular reflection prediction information and diffuse reflection component prediction information. Thus, light reflection prediction information related to each surface point is obtained. Next, based on the surface points and a pre-trained color network, color prediction information for each pixel is generated. Thus, pixel color prediction information for each surface point can be obtained. Then, based on the aforementioned light reflection prediction information, pre-stored coordinate system transformation matrix information, and pre-stored polarization matrix information, each light reflection information is generated, wherein each of the aforementioned light reflection information includes specular reflection information, diffuse reflection component information, and specular vector information. Thus, light reflection-related information for each surface point can be obtained. Then, based on the aforementioned surface points, the aforementioned sampled light direction information, and the aforementioned camera pose information, a camera pose group for each surface point is generated. Thus, each surface point camera pose group under different camera poses can be obtained.Next, based on the camera pose groups for each surface point, each perspective inconsistency point is generated. This yields perspective inconsistency points. Then, based on each surface point, each reflection confidence information is generated. This yields reflection confidence information. Next, based on the pixel color prediction information, the light reflection information, the perspective inconsistency points, the reflection confidence information, and the polarization image information, a 3D model network is generated. This yields a network model for 3D model generation. Finally, the 3D model network is stored in a data storage server. This allows the 3D model network to be stored and used for 3D model generation. Because the 3D model network is generated by calculating the light reflection information, a more physically accurate reconstruction result is achieved, improving the model's ability to accurately represent complex surface features of an object. Because the light reflection information is generated using polarization matrix information, the propagation mode of light can be determined, enabling better pixel color prediction and generating predicted colors that match actual colors, thereby improving its applicability to objects with complex media. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The above and other features, advantages, and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0014] Figure 1 is a flow chart of some embodiments of a method for generating a three-dimensional model based on polarization information according to the present disclosure;

[0015] Figure 2 is a flowchart of generating a three-dimensional model network according to some embodiments of the present disclosure;

[0016] Figure 3 is a schematic structural diagram of some embodiments of a three-dimensional model generation device based on polarization information according to the present disclosure;

[0017] Figure 4 It is a structural diagram of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0018] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0019] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other.

[0020] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0021] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0022] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0023] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0024] Figure 1 A process 100 of some embodiments of the method for generating a three-dimensional model based on polarization information according to the present disclosure is shown. The method for generating a three-dimensional model based on polarization information includes the following steps:

[0025] Step 101: Receive information of each polarization image.

[0026] In some embodiments, the execution entity (e.g., a computing device) of the three-dimensional model generation method based on polarization information may receive various polarization picture information. The various polarization picture information may be pictures taken at various angles for representing the target object that requires three-dimensional modeling. Each polarization picture information in the various polarization picture information may include a polarization intensity picture, a polarization angle picture, and a color picture. The polarization intensity picture may be a picture for representing the polarization intensity of the image. The polarization intensity picture may be a DoLP (Degree of Linear Polarization) picture. The polarization angle picture may be a picture for representing the polarization angle of the image. The polarization angle picture may be an AoLP (Angle of Linear Polarization) picture. The color picture may be a picture for representing the color of the image. The color picture may be an RGB picture.

[0027] Step 102: Generate camera pose information based on each polarization image information.

[0028] In some embodiments, the above-mentioned execution entity may generate each camera pose information based on the above-mentioned polarization picture information. Particularly, each camera pose information in the above-mentioned camera pose information may be information related to the pose of the camera in the world coordinate system when shooting the polarization picture information. Particularly, each camera pose information in the above-mentioned camera pose information may include camera intrinsic parameter information, camera extrinsic parameter information and camera origin information. Particularly, the above-mentioned camera intrinsic parameter information may be used to characterize the intrinsic parameters (Intrinsics) of the camera. Particularly, the above-mentioned camera extrinsic parameter information may be used to characterize the extrinsics (Extrinsics) of the camera. In practice, for each polarization picture in the above-mentioned polarization pictures, the above-mentioned execution entity may input the above-mentioned polarization picture information into computer vision software to obtain camera pose information. Particularly, the above-mentioned computer vision software may be Colmap software.

[0029] Step 103: Generate information on the direction of each sampling light based on the position information of each camera.

[0030] In some embodiments, the execution entity may generate each sampling light direction information based on the camera pose information, wherein each sampling light direction information may be used to represent the direction of the sampling light.

[0031] In some optional implementations of some embodiments, the execution entity may generate each sampling light direction information based on each camera pose information through the following steps:

[0032] The first step is to perform the following steps for each of the above camera pose information:

[0033] In the first sub-step, the polarization image information corresponding to the camera posture information is determined as the target polarization image information.

[0034] In the second sub-step, for each pixel in the color image included in the target polarization image information, the following steps are performed:

[0035] Sub-step 1: Generate world coordinate information based on the camera extrinsic information and the pixel points included in the camera pose information. The world coordinate information can be the coordinates of the pixel points in the world coordinate system. In practice, the execution subject can input the camera extrinsic information and the pixel points into a first preset formula to obtain the world coordinate information. The first preset formula can be world coords =pose×pixel points .world coords is the world coordinate information. pose is the camera external parameter information. pixel points are the coordinates of the pixel in the color image above. × is matrix multiplication.

[0036] Sub-step 2: Based on the world coordinate information and the camera origin information included in the camera pose information, generate initial light direction information. The initial light direction information can be used to characterize the direction of the sampling light. In practice, the execution entity can input the world coordinate information and the camera origin information into a second preset formula to obtain the initial light direction information. The second preset formula can be v=world coords -cam loc v is the initial light direction information. world coords It is the world coordinate information. loc The camera origin information.

[0037] Sub-step three: normalize the initial light direction information to obtain sampled light direction information. In practice, the execution entity may use a normalization function to normalize the initial light direction information to obtain sampled light direction information. The normalization function may be a normalize() function.

[0038] Step 104 : generating each sampling point based on the direction information of each sampling light and a pre-trained signed distance network.

[0039] In some embodiments, the execution entity may generate sampling points based on the sampling light direction information and a pre-trained signed distance network. Each of the sampling points may be information representing a point on the sampling light direction information in a world coordinate system. The sampling point may include sampling point coordinates, distance information, feature information, and surface point information. The sampling point coordinates may represent the coordinates of the sampling point in the world coordinate system. The distance information may represent the distance from the sampling point to the surface of an object. The feature information may represent the feature corresponding to the sampling point. The surface point information may be text information indicating whether the sampling point is a surface point. The signed distance network may be a multi-layer perceptron network. The signed distance network may include eight linear layers for extracting feature information, with each layer being a 256-dimensional linear layer. The signed distance network takes as input the coordinate x of the sampling point in space, and outputs the distance d from the sampling point x to the surface of the object, as well as the feature z corresponding to each sampling point.

[0040] In the process of adopting technical solutions to solve the above technical problems, the following problems often arise:

[0041] After obtaining the sampling light direction, only using the points obtained by the light direction to perform three-dimensional modeling of the object results in some points that are not helpful for modeling being used as modeling points, thus making the established three-dimensional model less effective.

[0042] Faced with the above technical problems, we decided to adopt the following solutions:

[0043] In some optional implementations of some embodiments, the execution entity may generate each sampling point based on the above-mentioned sampling light direction information and a pre-trained signed distance network through the following steps:

[0044] In the first step, for each piece of sampled light direction information, perform the following steps:

[0045] In the first sub-step, the camera pose information corresponding to the sampled light direction information is determined as the target camera pose information.

[0046] The second sub-step involves generating sampling point coordinates based on the camera origin information and the sampling ray direction information included in the target camera pose information. In practice, the execution entity may first determine the ray passing through the sampling ray direction information of the camera origin information as the target ray. The execution entity may then randomly sample a point on the target ray as the sampling point coordinates.

[0047] The third sub-step is to execute the following sampling point loop steps based on the sampling point coordinates:

[0048] Sub-step 1: Input the sampling point coordinates into the signed distance network to obtain distance information and feature information. The distance information may represent the distance between the sampling point and the surface of the object. The feature information may represent the feature corresponding to the sampling point. In practice, the execution entity may input the sampling point coordinates into the signed distance network to obtain the distance information and feature information.

[0049] Sub-step 2: Generate sampling point coordinates based on the distance information, the sampling light direction information, and the sampling point coordinates to update the sampling point coordinates. In practice, the execution entity may input the distance information, the sampling light direction information, and the sampling point coordinates into a third preset formula to obtain updated sampling point coordinates. The third preset formula may be x′ = x + d*v, where x′ represents the updated sampling point coordinates, x represents the sampling point coordinates, d represents the distance information, and v represents the sampling light direction information.

[0050] Sub-step three: In response to determining that the number of cycles in the sampling point loop step does not meet a preset sampling point loop condition, updating the distance information and feature information, and re-performing the sampling point loop step based on the updated sampling point coordinates. The preset sampling point loop condition may be that the number of cycles in the sampling point loop step is greater than a preset loop threshold. The preset loop threshold may be a pre-set value, which is not limited herein.

[0051] Sub-step 4: In response to determining that the number of iterations of the sampling point iteration step satisfies the preset sampling point iteration condition, the updated sampling point coordinates, the updated distance information, and the updated feature information are integrated to obtain a sampling point. In practice, the execution entity may combine the updated sampling point coordinates, the updated distance information, and the updated feature information into a sampling point.

[0052] In a fourth sub-step, in response to determining that the distance information included in the sampling point satisfies a preset distance threshold condition, the following steps are performed:

[0053] Sub-step 1: Determine the first preset surface point information as the surface point information. The preset distance threshold condition may be that the distance information satisfies a preset distance threshold. The preset distance threshold may be a pre-set value, which is not limited herein. The first preset surface point information may be text information indicating that the sampling point is a surface point. For example, the first preset surface point information may be "Y."

[0054] Sub-step 2: adding the surface point information to the sampling point to update the sampling point.

[0055] In a fifth sub-step, in response to determining that the distance information included in the sampling point does not meet the preset distance threshold condition, executing the following steps:

[0056] Sub-step 1: Determine the second preset surface point information as the surface point information. The second preset surface point information may be text information indicating that the sampling point is not a surface point. For example, the second preset surface point information may be "N."

[0057] Sub-step 2: adding the surface point information to the sampling point to update the sampling point.

[0058] The above technical solution and its related contents, as an inventive feature of an embodiment of the present disclosure, address the problem that, after obtaining a sampling light direction, only points derived from that direction are used to model an object, resulting in some points that are not helpful for modeling being used as modeling points, thus resulting in poor quality of the constructed 3D model. Factors that often contribute to poor 3D model quality are as follows: after obtaining a sampling light direction, only points derived from that direction are used to model an object, resulting in some points that are not helpful for modeling being used as modeling points. If this factor is addressed, the quality of the 3D model can be improved. To achieve this, the present disclosure first performs the following steps for each of the aforementioned sampling light direction information: determining the camera pose information corresponding to the sampling light direction information as the target camera pose information. Then, based on the camera origin information included in the target camera pose information and the sampling light direction information, sampling point coordinates are generated. Thus, randomly generated sampling point coordinates along the sampling light direction are obtained. Then, based on the sampling point coordinates, the following sampling point loop step is performed: inputting the sampling point coordinates into the aforementioned signed distance network to obtain distance information and feature information. Next, based on the distance information, the sampling light direction information, and the sampling point coordinates, sampling point coordinates are generated to update the sampling point coordinates. This allows the sampling point coordinates to be updated to obtain more optimal sampling point coordinates. Then, in response to determining that the number of iterations of the sampling point loop step does not meet a preset sampling point loop condition, the distance information and feature information are updated, and the sampling point loop step is performed again based on the updated sampling point coordinates. This allows the sampling point coordinates that do not meet the loop condition to be looped again. Then, in response to determining that the number of iterations of the sampling point loop step meets the preset sampling point loop condition, the updated sampling point coordinates, the updated distance information, and the updated feature information are integrated to obtain a sampling point. This allows the optimal sampling point to be obtained. Next, in response to determining that the distance information included in the sampling point meets a preset distance threshold condition, the following steps are performed: first preset surface point information is determined as the surface point information. Then, the surface point information is added to the sampling point to update the sampling point. Then, in response to determining that the distance information included in the sampling point does not meet the preset distance threshold condition, the following steps are performed: second preset surface point information is determined as the surface point information. Finally, the surface point information is added to the sampling point to update the sampling point. This allows the surface point information to be added to the sampling point. Because the sampling point coordinates are iterated through a loop, the obtained sampling point coordinates are more accurate than the true coordinates of the three-dimensional object. Furthermore, because the distance information included in the sampling point is evaluated against the preset distance threshold condition, the obtained surface point information sampling point, and thus the quality of the constructed three-dimensional model, is improved.

[0059] Step 105 : Determine the sampling points that meet the preset surface point conditions among the sampling points as surface points, and obtain the surface points.

[0060] In some embodiments, the execution entity may determine, among the sampling points, those that satisfy a preset surface point condition as surface points, thereby obtaining the surface points. The preset surface point condition may be that the surface point information included in the sampling point is "Y." Each of the surface points may be used to characterize a sampling point that satisfies the preset surface point condition.

[0061] Step 106 : Generate respective light reflection prediction information based on respective surface points and the pre-trained light decomposition network.

[0062] In some embodiments, the execution entity may generate light reflection prediction information based on the surface points and a pre-trained light decomposition network. Each of the light reflection prediction information may be a Stokes vector representing the surface point. The Stokes vector is a four-dimensional vector. Since the circular polarization represented by the fourth dimension is relatively weak, only the first three dimensions of the Stokes vector are considered. Each of the light reflection prediction information may include specular reflection prediction information and diffuse reflection component prediction information. The Stokes vector may be divided into a diffuse reflection component and a specular reflection component. The specular reflection prediction information may be the specular reflection component included in the Stokes vector. The diffuse reflection component prediction information may be the diffuse reflection component included in the Stokes vector. The light decomposition network may be a neural network comprising three multi-layer perceptron networks. Each of the multi-layer perceptron networks may comprise four linear layers, each of which is a 256-dimensional linear layer. The first multi-layer perceptron network may be used to predict the first dimension of the Stokes vector. The first dimension of the Stokes vector comprises the first dimension of the specular reflection component and the first dimension of the diffuse reflection component. The second multilayer perceptron network can be used to predict the second and third dimensions of the specular reflection component included in the output Stokes vector. The third multilayer perceptron network can be used to predict the second and third dimensions of the diffuse reflection component included in the output Stokes vector.

[0063] In some optional implementations of some embodiments, the execution entity may generate the respective light reflection prediction information based on the respective surface points and a pre-trained light decomposition network through the following steps:

[0064] In the first step, for each of the above surface points, perform the following steps:

[0065] In the first sub-step, the surface point is input into the ray decomposition network to obtain specular reflection prediction information and diffuse reflection component prediction information. In practice, the execution entity may input the surface point into the ray decomposition network to obtain specular reflection prediction information and diffuse reflection component prediction information.

[0066] The second sub-step is to integrate the specular reflection prediction information and the diffuse reflection component prediction information to obtain light reflection prediction information. In practice, the execution entity may combine the specular reflection prediction information and the diffuse reflection component prediction information into light reflection prediction information.

[0067] Step 107 : Generate color prediction information for each pixel based on each surface point and a pre-trained color network.

[0068] In some embodiments, the execution entity may generate color prediction information for each pixel based on the surface points and a pre-trained color network. The pixel color prediction information may be prediction information representing the pixel color corresponding to the surface point. The color network may be a multi-layer perceptron network. The color network may include eight linear layers, each with 256 dimensions. The input of the color network may be the coordinate x of the sampling point in space, and the output may be the predicted RGB color.

[0069] In some optional implementations of some embodiments, the execution entity may generate color prediction information for each pixel based on the surface points and a pre-trained color network through the following steps:

[0070] In the first step, for each of the surface points, the surface point is input into the color network to obtain pixel color prediction information. In practice, the execution entity may input the surface point into the color network to obtain pixel color prediction information.

[0071] Step 108 : generating each light reflection information based on each light reflection prediction information, pre-stored coordinate system transformation matrix information, and pre-stored polarization matrix information.

[0072] In some embodiments, the execution entity may generate each light reflection information based on the light reflection prediction information, pre-stored coordinate system transformation matrix information, and pre-stored polarization matrix information. The coordinate system transformation matrix information may be a rotation matrix used to characterize the coordinate system transformation. The polarization matrix information may be a Mueller matrix. The polarization matrix information may include a specular reflection polarization matrix and a diffuse reflection polarization matrix. The specular reflection polarization matrix may be a Mueller matrix representing specular reflection. The diffuse reflection polarization matrix may be a Mueller matrix representing diffuse reflection. Each of the light reflection information may be light reflection prediction information after processing. Each of the light reflection information may include specular reflection information, diffuse reflection component information, and specular vector information. The specular reflection information may be processed specular reflection prediction information. The diffuse reflection component information may be processed diffuse reflection component prediction information. The specular vector information may be a Stokes vector.

[0073] In some optional implementations of some embodiments, the execution entity may generate each light reflection information based on the above-mentioned each light reflection prediction information, pre-stored coordinate system transformation matrix information, and pre-stored polarization matrix information through the following steps:

[0074] In the first step, for each piece of light reflection prediction information, perform the following steps:

[0075] The first sub-step is to generate mirror reflection information based on the mirror reflection prediction information included in the light reflection prediction information, the pre-stored coordinate system change matrix information and the mirror reflection polarization matrix included in the polarization matrix information. The mirror reflection information can be a vector used to characterize the information related to mirror reflection in the camera coordinate system. The camera coordinate system can be the coordinate system where the camera is located, which is not limited here. The coordinate system change matrix information can be a coordinate transformation matrix used to characterize the changes between the various coordinate systems. The various coordinate systems can be the world coordinate system, the Mueller coordinate system or the camera coordinate system. The Mueller coordinate system can be the coordinate system where the Mueller matrix is ​​located. The camera coordinate system can be the coordinate system where the camera is located. In practice, the execution entity can input the mirror reflection prediction information, the coordinate system change matrix information and the mirror reflection polarization matrix into the fourth preset formula to obtain the mirror reflection information. The fourth preset formula can be Specular reflection information. is the specular reflection prediction information. i is the coordinate transformation matrix from the world coordinate system to the Muller coordinate system. ris the mirror reflection polarization matrix. R o is the coordinate transformation matrix from the Mueller coordinate system to the world coordinate system. c It is the coordinate transformation matrix from the world coordinate system to the camera coordinate system.

[0076] The second sub-step is to generate diffuse reflection component information based on the diffuse reflection component prediction information included in the light reflection prediction information, the coordinate system change matrix information and the diffuse reflection polarization matrix included in the polarization matrix information. The diffuse reflection component information can be a vector used to characterize diffuse reflection-related information in the camera coordinate system. In practice, the execution entity can input the diffuse reflection component prediction information, the coordinate system change matrix information and the diffuse reflection polarization matrix into a fifth preset formula to obtain the diffuse reflection component information. The fifth preset formula can be It is the diffuse reflection component information. Prediction information for the diffuse component. R i is the coordinate transformation matrix from the world coordinate system to the Muller coordinate system. d is the diffuse reflection polarization matrix. R o is the coordinate transformation matrix from the Mueller coordinate system to the world coordinate system. c It is the coordinate transformation matrix from the world coordinate system to the camera coordinate system.

[0077] The third sub-step is to generate mirror vector information based on the above-mentioned mirror reflection information and the above-mentioned diffuse reflection component information. In practice, the above-mentioned execution subject can input the above-mentioned mirror reflection information and the above-mentioned diffuse reflection component information into a sixth preset formula to obtain the mirror vector information. Among them, the above-mentioned sixth preset formula can be Among them, Stokes out is the mirror vector information. Prediction information for specular reflection. It is the diffuse reflection component information.

[0078] The fourth sub-step is to integrate the specular reflection information, the diffuse reflection component information, and the specular vector information to obtain light reflection information. In practice, the execution entity may combine the specular reflection information, the diffuse reflection component information, and the specular vector information into light reflection information.

[0079] Step 109 : generating a camera pose group for each surface point based on each surface point, each sampled light direction information, and each camera pose information.

[0080] In some embodiments, the execution entity may generate a camera pose group for each surface point based on the surface points, the sampled light direction information, and the camera pose information. Each surface point camera pose group in the surface point camera pose group may include camera pose information corresponding to each surface point.

[0081] In some optional implementations of some embodiments, the execution entity may generate a camera pose group for each surface point based on the surface points, the sampling light direction information, and the camera pose information by performing the following steps:

[0082] In the first step, for each of the above surface points, perform the following steps:

[0083] In the first sub-step, the sampling light direction information corresponding to the surface point is determined as the reference sampling light direction information.

[0084] In the second sub-step, the camera pose information corresponding to the reference sampling light direction information is determined as the reference camera pose information.

[0085] In a third sub-step, based on the reference camera pose information, each piece of camera pose information that satisfies a preset camera condition is determined as a surface point camera pose group. The preset camera condition may be that the surface point can be observed using polarization image information corresponding to camera pose information other than the reference camera pose information. In practice, the execution entity may determine each piece of camera pose information that satisfies the preset camera condition as a surface point camera pose group.

[0086] Step 110 : generating each perspective inconsistency point based on each surface point camera pose group.

[0087] In some embodiments, the execution entity may generate perspective inconsistency points based on the surface point camera pose groups. Each of the perspective inconsistency points may be a surface point where pixel color prediction information representing the surface point corresponding to the surface point camera pose group changes dramatically.

[0088] In some optional implementations of some embodiments, the execution entity may generate each perspective inconsistency point based on each surface point camera pose group through the following steps:

[0089] In the first step, for each of the surface point camera pose groups mentioned above, perform the following steps:

[0090] In the first sub-step, the surface point corresponding to the surface point camera pose group is determined as the target surface point.

[0091] In the second sub-step, for each camera pose information in the surface point camera pose group, perform the following steps:

[0092] Sub-step 1: Generate a camera surface point based on the target surface point and the camera extrinsic parameter information included in the camera pose information. The camera surface point can be a coordinate in the camera coordinate system. In practice, the execution subject can input the sampling point coordinates included in the target surface point and the camera extrinsic parameter information into the seventh preset formula to obtain the camera surface point. The seventh preset formula can be Y=pose -1 *X.Y is the camera surface point. pose is the camera external parameter information. pose -1 is the inverse matrix of the camera's extrinsic information. X is the coordinate of the sampling point.

[0093] Sub-step two, based on the camera surface point and the camera intrinsic parameter information included in the camera pose information, generates pixel coordinates. The pixel coordinates may be coordinates used to characterize the target surface point in the pixel coordinate system in the color image. The pixel coordinate system may be a coordinate system used to characterize the color image, which is not limited here. In practice, the execution entity may input the camera surface point and the camera intrinsic parameter information into an eighth preset formula to obtain pixel coordinates. The eighth preset formula may be (u, v) = intrinsic parameter * Y. (u, v) is the pixel coordinate. is the camera intrinsic parameter information. Y is the camera surface point.

[0094] Sub-step three: determining the polarization image information corresponding to the above-mentioned camera posture information as the reference polarization image information.

[0095] Sub-step 4: Generate target surface point color information based on the pixel coordinates and the color image included in the reference polarization image information. The target surface point color information may be the numerical value of the color corresponding to the pixel point corresponding to the pixel coordinates in the color image. In practice, the execution entity may determine the color corresponding to the pixel point corresponding to the pixel coordinates in the color image as the target surface point color information.

[0096] In the third sub-step, the obtained variance of the color information of each target surface point is determined as the target surface point variance.

[0097] In a fourth sub-step, in response to determining that the variance of the target surface point satisfies a preset color variance condition, the target surface point is determined to be a view-inconsistent point. The preset color variance condition may be that the variance of the target surface point is greater than a preset variance threshold. The preset variance threshold may be a pre-set value and is not limited herein.

[0098] Step 111 : Generate reflection confidence information based on each surface point.

[0099] In some embodiments, the execution entity may generate various reflection confidence information based on the various surface points, wherein each reflection confidence information may be used to represent the confidence level of the intensity of the reflected light.

[0100] In some optional implementations of some embodiments, the execution entity may generate each reflection confidence information based on each surface point through the following steps:

[0101] In the first step, for each of the above surface points, perform the following steps:

[0102] In the first sub-step, the sampling light direction information corresponding to the surface point is determined as the target sampling light direction information.

[0103] In the second sub-step, the polarization image information corresponding to the surface point is determined as the target polarization image information.

[0104] The third sub-step involves generating a surface point normal vector based on the surface point and the signed distance network. The surface point normal vector can be used to represent the normal vector of the surface point on the surface of the photographed object. In practice, the execution entity can use the signed distance network to derive the coordinates of the sampling points included in the surface point to obtain the surface point normal vector.

[0105] The fourth sub-step is to generate incident light intensity information based on the target polarization image information. The incident light intensity information may represent the intensity of light received by the camera. In practice, the execution entity may convert the color image included in the target polarization image information into a grayscale image to obtain a grayscale color image. The execution entity may then normalize the grayscale color image to obtain the incident light intensity information.

[0106] The fifth sub-step is to generate target incident light direction information based on the target sampling light direction information. The target incident light direction information may be the opposite direction of the sampling light direction. In practice, the execution entity may determine the opposite direction of the target sampling light direction information as the target incident light direction information.

[0107] The sixth sub-step is to generate target refracted light direction information based on the target incident light direction information. The target refracted light direction information may be the refraction direction of the target incident light direction information. In practice, the execution entity may apply the law of refraction to the target incident light direction information to obtain the target refracted light direction information.

[0108] The seventh sub-step is to generate a light coefficient based on the pre-stored refractive index information, the surface point normal vector, the target incident light direction information and the target refracted light direction information. The refractive index information may be used to characterize the refractive index in the target medium. The refractive index information may include the external medium refractive index and the object refractive index. The external medium refractive index may be the refractive index of air. The object refractive index may be the refractive index of the photographed object. The light coefficient may be a Fresnel coefficient. In practice, the execution entity may input the refractive index information, the target incident light direction information and the target refracted light direction information into a ninth preset formula to obtain the light coefficient. The ninth preset formula may be F is the light coefficient. η i is the refractive index of the external medium. V i is the incident light direction information of the target. n is the surface point normal vector. η t is the refractive index of the object. t Refracts light direction information for the target.

[0109] In the eighth sub-step, based on the light coefficient and the incident light intensity information, the reflected light intensity information is generated. The reflected light intensity information can be used to characterize the intensity of the light reflected by the object. In practice, the execution subject can input the light coefficient and the incident light intensity information into a tenth preset formula to obtain the reflected light intensity information. The tenth preset formula can be I r =F·I i I r is the reflected light intensity information. F is the light coefficient. I i is the incident light intensity information.

[0110] The ninth sub-step is to generate refracted light intensity information based on the light coefficient and the incident light intensity information. The refracted light intensity information may be used to characterize the intensity of the light refracted by the object. In practice, the execution entity may input the light coefficient and the incident light intensity information into an eleventh preset formula to obtain the refracted light intensity information. The eleventh preset formula may be: t =(1-F)·I i I t is the refracted light intensity information. F is the light coefficient. I i is the incident light intensity information.

[0111] The tenth sub-step is to generate reflection confidence information based on the above-mentioned reflected light intensity information and the above-mentioned refracted light intensity information. In practice, the above-mentioned execution subject can input the above-mentioned reflected light intensity information and the above-mentioned refracted light intensity information into the twelfth preset formula to obtain the reflection confidence information. Among them, the above-mentioned twelfth preset formula can be W is the reflection confidence information. r Is the reflected light intensity information. t It is the intensity information of refracted light.

[0112] Step 112 : generating a three-dimensional model network based on each pixel color prediction information, each light reflection information, each viewing angle inconsistency point, each reflection confidence information and each polarization image information.

[0113] In some embodiments, the execution subject may generate a three-dimensional model network based on the above-mentioned individual pixel color prediction information, the above-mentioned individual light reflection information, the above-mentioned individual perspective inconsistency points, the above-mentioned individual reflection confidence information and the above-mentioned individual polarization picture information. The above-mentioned three-dimensional model network may be a neural network that provides data information for subsequent three-dimensional model generation. In practice, the execution subject may use the above-mentioned individual pixel color prediction information, the above-mentioned individual light reflection information, the above-mentioned individual perspective inconsistency points, the above-mentioned individual reflection confidence information and the above-mentioned individual polarization picture information as parameters for neural network training. The neural network is optimized and trained using the Adam optimizer. The difference between the color prediction information of each pixel and the true color is used as the loss of the neural network. Through the back-propagation algorithm, as the number of iterations increases, the neural network eventually converges and the training is completed to obtain a three-dimensional model network. The specific process of generating the three-dimensional model network can be referred to. Figure 2 .

[0114] Figure 2 This is a flowchart of generating a three-dimensional model network in some embodiments of the present disclosure. Among them, "calibration of camera pose" can be steps 101-102. "Calculation of the direction of light emitted by the camera" can be step 103. "Random sampling of sampling points" can be step 104. "Sending the sampling points to the signed distance network to obtain the distance d and feature f" can be step 104. "Using the light decomposition network to obtain the specular reflection component and the diffuse reflection component" can be steps 106 and 108. "Using the color network to predict the pixel color" can be step 107. "Determining the visibility of surface points" can be step 105. "Calculating the color change variance var" can be step 110. "Calculating the loss and back-propagating with the stokes component and RGB value collected by the camera" can be steps 109 and steps 111-112.

[0115] Step 113: store the three-dimensional model network in a data storage server.

[0116] In some embodiments, the execution entity may store the three-dimensional model network in a data storage server, wherein the data storage server may be a server for managing and storing models and model-related information.

[0117] The above-described embodiments of the present disclosure have the following beneficial effects: Through the polarization information-based 3D model generation method of some embodiments of the present disclosure, the method for resolving other complex surface features of objects and the applicability for objects made of complex media are improved. Specifically, the reason for the poor resolution of other complex surface features of objects is that the light in the 3D reconstruction technology based on neural radiation fields is mainly transmitted in straight lines, which is less effective in resolving other complex surface features of objects (such as transparency). Moreover, because pixel color is predicted through volume rendering of direct light, the applicability for objects made of complex media is poor. Based on this, the polarization information-based 3D model generation method of some embodiments of the present disclosure first receives polarization image information, wherein each of the polarization image information includes a polarization intensity image, a polarization angle image, and a color image. Thus, an image containing polarization information is obtained. Then, based on the polarization image information, each camera pose information is generated, wherein each of the camera pose information includes camera intrinsic parameter information, camera extrinsic parameter information, and camera origin information. Thus, the pose information of the camera corresponding to each polarization image in the world coordinate system is obtained. Then, based on the aforementioned camera pose information, information about the direction of each sampled light ray is generated. Thus, the direction of each sampled light ray can be obtained. Next, based on the aforementioned sampled light ray direction information and a pre-trained signed distance network, each sampling point is generated. Thus, sampling points related to the three-dimensional model can be obtained. Then, among the aforementioned sampling points, those that meet preset surface point conditions are determined as surface points, thereby obtaining each surface point. Thus, surface points on the surface of the three-dimensional model can be obtained. Next, based on the aforementioned surface points and a pre-trained light decomposition network, information about light reflection predictions is generated, wherein each piece of light reflection prediction information includes specular reflection prediction information and diffuse reflection component prediction information. Thus, light reflection-related prediction information for each surface point can be obtained. Next, based on the aforementioned surface points and a pre-trained color network, information about pixel color predictions is generated. Thus, pixel color-related prediction information for each surface point can be obtained. Then, based on the aforementioned light reflection prediction information, pre-stored coordinate system transformation matrix information, and pre-stored polarization matrix information, each light reflection information is generated, wherein each light reflection information includes specular reflection information, diffuse reflection component information, and specular vector information. Thus, information related to light reflection at each surface point can be obtained. Then, based on the aforementioned surface points, the aforementioned sampled light direction information, and the aforementioned camera pose information, a camera pose group for each surface point is generated. Thus, each surface point camera pose group under different camera poses can be obtained. Next, based on the aforementioned surface point camera pose groups, each viewpoint inconsistency point is generated.Thus, the perspective inconsistency point can be obtained. Then, based on the above-mentioned surface points, each reflection confidence information is generated. Thus, each reflection confidence information can be obtained. Then, based on the above-mentioned pixel color prediction information, the above-mentioned light reflection information, the above-mentioned perspective inconsistency point, the above-mentioned reflection confidence information and the above-mentioned polarization image information, a three-dimensional model network is generated. Thus, a network model for three-dimensional model generation can be obtained. Finally, the above-mentioned three-dimensional model network is stored in a data storage server. Thus, the three-dimensional model network can be stored as a network used for three-dimensional model generation. Because the three-dimensional model network is generated by calculating each light reflection information, a more accurate reconstruction result that conforms to physical reality is achieved, so that the model is more effective in solving other complex surface features of the object. Because the light reflection information is generated using polarization matrix information, the propagation mode of light can be obtained, so that the pixel color can be better predicted, thereby improving the applicability to objects with complex media.

[0118] Further references Figure 3 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a three-dimensional model generation device based on polarization information. These device embodiments are similar to Figure 1 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices.

[0119] like Figure 2As shown, in some embodiments, the three-dimensional model generation device 300 based on polarization information includes: a receiving unit 301, a first generating unit 302, a second generating unit 303, a third generating unit 304, a determining unit 305, a fourth generating unit 306, a fifth generating unit 307, a sixth generating unit 308, a seventh generating unit 309, an eighth generating unit 310, a ninth generating unit 311, a tenth generating unit 312 and a storage unit 313. The receiving unit 301 is configured to receive polarization image information, wherein each polarization image information in the polarization image information includes a polarization intensity image, a polarization angle image and a color image; the first generating unit 302 is configured to generate camera pose information based on the polarization image information, wherein each camera pose information in the camera pose information includes camera intrinsic parameter information, camera extrinsic parameter information and camera origin information; the second generating unit 303 is configured to generate sampling light direction information based on the camera pose information; the third generating unit 304 is configured to generate sampling light direction information based on the camera pose information; 304 is configured to generate each sampling point based on the above-mentioned each sampling light direction information and the pre-trained signed distance network; the determination unit 305 is configured to determine the sampling points that meet the preset surface point conditions among the above-mentioned each sampling point as surface points, thereby obtaining each surface point; the fourth generation unit 306 is configured to generate each light reflection prediction information based on the above-mentioned each surface point and the pre-trained light decomposition network, wherein each light reflection prediction information in the above-mentioned each light reflection prediction information includes specular reflection prediction information and diffuse reflection component prediction information; the fifth generation unit 307 is configured to generate each light reflection prediction information based on the above-mentioned each surface point and the pre-trained light decomposition network. The system is configured to generate pixel color prediction information based on the surface points and a pre-trained color network. The sixth generation unit 308 is configured to generate light reflection information based on the light reflection prediction information, pre-stored coordinate system transformation matrix information, and pre-stored polarization matrix information, wherein each light reflection information in the light reflection information includes specular reflection information, diffuse reflection component information, and specular vector information. The seventh generation unit 309 is configured to generate a camera pose group for each surface point based on the surface points, the sampled light direction information, and the camera pose information. The eighth generation unit 310 is configured to generate perspective inconsistency points based on the surface point camera pose groups. The ninth generation unit 311 is configured to generate reflection confidence information based on the surface points. The tenth generation unit 312 is configured to generate a three-dimensional model network based on the pixel color prediction information, the light reflection information, the perspective inconsistency points, the reflection confidence information, and the polarization image information. The storage unit 313 is configured to store the three-dimensional model network in a data storage server.

[0120] It is understood that the units described in the device 300 are similar to those in the reference Figure 1 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the device 300 and the units included therein, and will not be repeated here.

[0121] Reference below Figure 4 , which shows a structural diagram of an electronic device 400 suitable for implementing some embodiments of the present disclosure. Figure 4 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0122] like Figure 4 As shown, the electronic device 400 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 408 into a random access memory (RAM) 403. Various programs and data required for the operation of the electronic device 400 are also stored in the RAM 403. The processing device 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0123] Typically, the following devices may be connected to the I / O interface 405: an input device 406 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 407 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 408 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 409. The communication device 409 may allow the electronic device 400 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 4 The electronic device 400 is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead. Figure 4 Each block shown in the figure may represent one device, or may represent multiple devices as needed.

[0124] In particular, according to some embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from a network via the communication device 409, or installed from the storage device 408, or installed from the ROM 402. When the computer program is executed by the processing device 401, the above-mentioned functions defined in the method of some embodiments of the present disclosure are performed.

[0125] It should be noted that the computer-readable medium described in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or device. In some embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0126] In some embodiments, the client and server can communicate using any currently known or future developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0127] The computer-readable medium may be included in the electronic device; or it may exist independently without being assembled into the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: receives various polarization picture information, wherein each polarization picture information in the various polarization picture information includes a polarization intensity picture, a polarization angle picture, and a color picture; generates various camera pose information based on the various polarization picture information, wherein each camera pose information in the various camera pose information includes camera intrinsic parameter information, camera extrinsic parameter information, and camera origin information; generates various sampling light direction information based on the various camera pose information; generates various sampling points based on the various sampling light direction information and a pre-trained signed distance network; determines the sampling points in the various sampling points that meet the preset surface point conditions as surface points to obtain various surface points; generates various light reflection prediction information based on the various surface points and a pre-trained light decomposition network, wherein each light reflection prediction information in the various light reflection prediction information Including mirror reflection prediction information and diffuse reflection component prediction information; based on the above-mentioned surface points and a pre-trained color network, generating each pixel color prediction information; based on the above-mentioned light reflection prediction information, pre-stored coordinate system transformation matrix information and pre-stored polarization matrix information, generating each light reflection information, wherein each light reflection information in the above-mentioned light reflection information includes mirror reflection information, diffuse reflection component information and mirror vector information; based on the above-mentioned surface points, the above-mentioned sampled light direction information and the above-mentioned camera pose information, generating each surface point camera pose group; based on the above-mentioned surface point camera pose group, generating each perspective inconsistency point; based on the above-mentioned surface points, generating each reflection confidence information; based on the above-mentioned pixel color prediction information, the above-mentioned light reflection information, the above-mentioned each perspective inconsistency point, the above-mentioned each reflection confidence information and the above-mentioned each polarization picture information, generating a three-dimensional model network; storing the above-mentioned three-dimensional model network to a data storage server.

[0128] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0129] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0130] The units described in some embodiments of the present disclosure may be implemented by software or by hardware. The described units may also be provided in a processor. For example, they may be described as follows: a processor includes a receiving unit, a first generating unit, a second generating unit, a third generating unit, a determining unit, a fourth generating unit, a fifth generating unit, a sixth generating unit, a seventh generating unit, an eighth generating unit, a ninth generating unit, a tenth generating unit, and a storage unit. The names of these units do not, in some cases, constitute a limitation on the units themselves. For example, the receiving unit may also be described as a "unit for receiving information of each polarization image."

[0131] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0132] The above description is only an illustration of some preferred embodiments of the present disclosure and the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A method for generating a three-dimensional model based on polarization information, comprising: Receive individual polarization picture information, wherein each polarization picture information in the individual polarization picture information includes a polarization intensity picture, a polarization angle picture, and a color picture; Based on the polarization image information, each camera pose information is generated, wherein each camera pose information in the camera pose information includes camera intrinsic parameter information, camera extrinsic parameter information and camera origin information; Based on the camera pose information, generate the direction information of each sampling light; Generating each sampling point based on the direction information of each sampling light and a pre-trained signed distance network; Determining sampling points that meet preset surface point conditions among the sampling points as surface points to obtain surface points; generating, based on the surface points and a pre-trained light decomposition network, respective light reflection prediction information, wherein each of the respective light reflection prediction information comprises specular reflection prediction information and diffuse reflection component prediction information; Generate color prediction information for each pixel based on each surface point and a pre-trained color network; generating respective light reflection information based on the respective light reflection prediction information, pre-stored coordinate system transformation matrix information, and pre-stored polarization matrix information, wherein each light reflection information in the respective light reflection information includes specular reflection information, diffuse reflection component information, and specular vector information; Generate a camera pose group for each surface point based on each surface point, each sampling light direction information and each camera pose information; generating each perspective inconsistency point based on each surface point camera pose group; generating respective reflection confidence information based on the respective surface points; generating a three-dimensional model network based on the respective pixel color prediction information, the respective light reflection information, the respective perspective inconsistency points, the respective reflection confidence information, and the respective polarization image information; The three-dimensional model network is stored in a data storage server.

2. The method according to claim 1, wherein The generating of each sampling light direction information based on each camera pose information includes: For each piece of camera pose information, perform the following steps: Determining the polarization image information corresponding to the camera pose information as the target polarization image information; For each pixel in the color image included in the target polarization image information, perform the following steps: Generate world coordinate information based on the camera extrinsic parameter information included in the camera pose information and the pixel point; Generate initial light direction information based on the world coordinate information and the camera origin information included in the camera pose information; Normalization is performed on the initial light direction information to obtain sampled light direction information.

3. The method according to claim 1, wherein The generating of each light reflection prediction information based on each surface point and a pre-trained light decomposition network includes: For each of the surface points, perform the following steps: Inputting the surface points into the ray decomposition network to obtain specular reflection prediction information and diffuse reflection component prediction information; The specular reflection prediction information and the diffuse reflection component prediction information are integrated to obtain light reflection prediction information.

4. The method according to claim 1, wherein The generating of color prediction information of each pixel based on each surface point and a pre-trained color network includes: For each of the surface points, the surface point is input into the color network to obtain pixel color prediction information.

5. The method according to claim 1, wherein The polarization matrix information includes a specular reflection polarization matrix and a diffuse reflection polarization matrix; and the generating of each light reflection information based on the respective light reflection prediction information, pre-stored coordinate system transformation matrix information, and pre-stored polarization matrix information includes: For each piece of light reflection prediction information, the following steps are performed: Generate specular reflection information based on specular reflection prediction information included in the light reflection prediction information, pre-stored coordinate system change matrix information, and the specular reflection polarization matrix included in the polarization matrix information; Generate diffuse reflection component information based on the diffuse reflection component prediction information included in the light reflection prediction information, the coordinate system change matrix information, and the diffuse reflection polarization matrix included in the polarization matrix information; generating specular vector information based on the specular reflection information and the diffuse reflection component information; The mirror reflection information, the diffuse reflection component information and the mirror vector information are integrated to obtain light reflection information.

6. The method according to claim 1, wherein Generating a camera pose group for each surface point based on each surface point, each sampling light direction information, and each camera pose information includes: For each of the surface points, perform the following steps: Determining the sampling light direction information corresponding to the surface point as reference sampling light direction information; Determining the camera pose information corresponding to the reference sampling light direction information as the reference camera pose information; Based on the reference camera pose information, each piece of camera pose information that meets a preset camera condition among the pieces of camera pose information is determined as a surface point camera pose group.

7. The method according to claim 1, wherein Generating each perspective inconsistency point based on each surface point camera pose group includes: For each surface point camera pose group in the surface point camera pose groups, perform the following steps: Determine the surface point corresponding to the surface point camera pose group as the target surface point; For each camera pose information in the surface point camera pose group, perform the following steps: Generate a camera surface point based on the target surface point and camera extrinsic parameter information included in the camera pose information; Generate pixel coordinates based on the camera surface point and camera intrinsic parameter information included in the camera pose information; Determining the polarization image information corresponding to the camera pose information as reference polarization image information; generating target surface point color information based on the pixel coordinates and the color image included in the reference polarization image information; The obtained variance of the color information of each target surface point is determined as the target surface point variance; In response to determining that the target surface point variance satisfies a preset color variance condition, the target surface point is determined as a perspective-inconsistent point.

8. The method according to claim 1, wherein Generating respective reflection confidence information based on the respective surface points includes: For each of the surface points, perform the following steps: Determining the sampling light direction information corresponding to the surface point as the target sampling light direction information; Determining the polarization image information corresponding to the surface point as target polarization image information; generating a surface point normal vector based on the surface point and the signed distance network; generating incident light intensity information based on the target polarization image information; Generating target incident light direction information based on the target sampling light direction information; Generating target refracted light direction information based on the target incident light direction information; Generate a light coefficient based on pre-stored refractive index information, the surface point normal vector, the target incident light direction information, and the target refracted light direction information; generating reflected light intensity information based on the light coefficient and the incident light intensity information; generating refracted light intensity information based on the light coefficient and the incident light intensity information; Reflection confidence information is generated based on the reflected light intensity information and the refracted light intensity information.

9. A three-dimensional model generation device based on polarization information, comprising: a receiving unit configured to receive individual polarization picture information, wherein each polarization picture information in the individual polarization picture information includes a polarization intensity picture, a polarization angle picture, and a color picture; A first generating unit is configured to generate each camera pose information based on each polarization image information, wherein each camera pose information in the each camera pose information includes camera intrinsic parameter information, camera extrinsic parameter information and camera origin information; A second generating unit is configured to generate each sampling light direction information based on each camera pose information; A third generating unit is configured to generate each sampling point based on the direction information of each sampling light and a pre-trained signed distance network; a determining unit configured to determine, among the sampling points, those that meet a preset surface point condition as surface points, to obtain surface points; a fourth generating unit configured to generate respective light reflection prediction information based on the respective surface points and a pre-trained light decomposition network, wherein each of the respective light reflection prediction information includes specular reflection prediction information and diffuse reflection component prediction information; a fifth generating unit, configured to generate color prediction information of each pixel based on each surface point and a pre-trained color network; a sixth generating unit, configured to generate each light reflection information based on the each light reflection prediction information, pre-stored coordinate system transformation matrix information, and pre-stored polarization matrix information, wherein each light reflection information in the each light reflection information includes specular reflection information, diffuse reflection component information, and specular vector information; a seventh generating unit, configured to generate a camera pose group for each surface point based on each surface point, each sampling light direction information and each camera pose information; An eighth generating unit is configured to generate each perspective inconsistency point based on each surface point camera pose group; a ninth generating unit configured to generate respective reflection confidence information based on the respective surface points; a tenth generating unit configured to generate a three-dimensional model network based on the respective pixel color prediction information, the respective light reflection information, the respective perspective inconsistency points, the respective reflection confidence information, and the respective polarization image information; The storage unit is configured to store the three-dimensional model network in a data storage server.

10. An electronic device comprising: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 8.

11. A computer-readable medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images