Three-dimensional model generation method and device based on polarization information, equipment and medium

Abstract

The embodiment of the invention discloses a three-dimensional model generation method and device based on polarization information, equipment and a medium. A specific embodiment of the method comprises the following steps: receiving information of each polarization picture; generating pose information of each camera; generating direction information of each sampling light; generating each sampling point; determining the sampling points meeting a preset surface point condition in all the sampling points as surface points, and obtaining all the surface points; generating reflection prediction information of each light ray; generating color prediction information of each pixel; generating reflection information of each light ray; generating a camera pose group of each surface point; generating various view angle inconsistent points; generating each piece of reflection confidence information; generating a three-dimensional model network; and storing the three-dimensional model network to a data storage server. According to the embodiment, the effect of expressing the complex surface features of the object is improved, and the applicability to the object of a complex medium is improved.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to a method, device, equipment, and medium for generating a three-dimensional model based on polarization information. Background Art

[0002] The method for generating a 3D model of an object of a special material is a method for 3D reconstruction of a model of a special material (for example, a transparent material, or a material with highlights on its surface). The commonly used method is a 3D reconstruction method based on a neural radiation field.

[0003] However, when using the above method to generate a 3D model of an object of special material, the following technical problems often occur:

[0004] In the 3D reconstruction technology based on neural radiation field, the light is mainly transmitted in straight lines, and the effect of expressing other complex surface features of objects (such as transparency) is poor. And because the pixel color is predicted by volume rendering of direct light, when predicting the pixel color of objects in complex media, the predicted color is inconsistent with the actual color, resulting in poor applicability for objects in complex media.

[0005] The above information disclosed in this Background section is only for enhancement of understanding of the background of the inventive concept and therefore it may contain information that does not form the prior art that is already known to a person of ordinary skill in the art. Summary of the invention

[0006] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0007] Some embodiments of the present disclosure propose a method, device, electronic device, and computer-readable medium for generating a three-dimensional model based on polarization information to solve one or more of the technical problems mentioned in the above background technology section.

[0008] In a first aspect, some embodiments of the present disclosure provide a method for generating a three-dimensional model based on polarization information, the method comprising: receiving various polarization picture information, wherein each of the above-mentioned polarization picture information comprises a polarization intensity picture, a polarization angle picture and a color picture; based on the above-mentioned various polarization picture information, generating various camera pose information, wherein each of the above-mentioned various camera pose information comprises camera intrinsic parameter information, camera extrinsic parameter information and camera origin information; based on the above-mentioned various camera pose information, generating various sampling light direction information; based on the above-mentioned various sampling light direction information and a pre-trained signed distance network, generating various sampling points; determining the sampling points among the above-mentioned various sampling points that meet the preset surface point conditions as surface points to obtain various surface points; based on the above-mentioned various surface points and a pre-trained light decomposition network, generating various light reflection prediction information, wherein each of the above-mentioned various light reflection prediction information comprises mirror reflection Generate each pixel color prediction information based on the above-mentioned surface points and the pre-trained color network; generate each light reflection information based on the above-mentioned light reflection prediction information, the pre-stored coordinate system transformation matrix information and the pre-stored polarization matrix information, wherein each light reflection information in the above-mentioned light reflection information includes specular reflection information, diffuse reflection component information and specular vector information; generate each surface point camera pose group based on the above-mentioned surface points, the above-mentioned each sampling light direction information and the above-mentioned each camera pose information; generate each viewpoint inconsistency point based on the above-mentioned each surface point camera pose group; generate each reflection confidence information based on the above-mentioned surface points; generate a three-dimensional model network based on the above-mentioned each pixel color prediction information, the above-mentioned each light reflection information, the above-mentioned each viewpoint inconsistency point, the above-mentioned each reflection confidence information and the above-mentioned each polarization picture information; store the above-mentioned three-dimensional model network in a data storage server.

[0009] In a second aspect, some embodiments of the present disclosure provide a three-dimensional model generation device based on polarization information, the device comprising: a receiving unit, configured to receive various polarization picture information, wherein each of the above-mentioned polarization picture information includes a polarization intensity picture, a polarization angle picture and a color picture; a first generating unit, configured to generate various camera pose information based on the above-mentioned various polarization picture information, wherein each of the above-mentioned camera pose information includes camera intrinsic parameter information, camera extrinsic parameter information and camera origin information; a second generating unit, configured to generate various sampling light direction information based on the above-mentioned various camera pose information; a third generating unit, configured to generate various sampling points based on the above-mentioned various sampling light direction information and a pre-trained signed distance network; a determining unit, configured to determine the sampling points among the above-mentioned various sampling points that meet the preset surface point conditions as surface points, to obtain various surface points; a fourth generating unit, configured to generate various light reflection prediction information based on the above-mentioned various surface points and a pre-trained light decomposition network, wherein each of the above-mentioned various light reflection prediction information includes mirror reflection prediction information and diffuse reflection prediction information. reflection component prediction information; a fifth generation unit, configured to generate each pixel color prediction information based on the above-mentioned surface points and a pre-trained color network; a sixth generation unit, configured to generate each light reflection information based on the above-mentioned light reflection prediction information, pre-stored coordinate system transformation matrix information and pre-stored polarization matrix information, wherein each light reflection information in the above-mentioned light reflection information includes mirror reflection information, diffuse reflection component information and mirror vector information; a seventh generation unit, configured to generate each surface point camera pose group based on the above-mentioned surface points, the above-mentioned each sampling light direction information and the above-mentioned each camera pose information; an eighth generation unit, configured to generate each viewpoint inconsistency point based on the above-mentioned surface point camera pose group; a ninth generation unit, configured to generate each reflection confidence information based on the above-mentioned surface points; a tenth generation unit, configured to generate a three-dimensional model network based on the above-mentioned each pixel color prediction information, the above-mentioned each light reflection information, the above-mentioned each viewpoint inconsistency point, the above-mentioned each reflection confidence information and the above-mentioned each polarization picture information; a storage unit, configured to store the above-mentioned three-dimensional model network in a data storage server.

[0010] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner of the above-mentioned first aspect.

[0011] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation manner of the above-mentioned first aspect is implemented.

[0012] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: through the three-dimensional model generation method based on polarization information of some embodiments of the present disclosure, the way of expressing other complex surface features of the object and the applicability to objects of complex media are improved. Specifically, the reason for the poor way of solving other complex surface features of the object is that the light in the three-dimensional reconstruction technology based on the neural radiation field is mainly transmitted in a straight line, and the effect of expressing other complex surface features of the object (such as transparency) is poor. And because the pixel color is predicted by volume rendering of direct light, when the pixel color of the object of the complex medium is predicted, the predicted color that does not match the actual color is generated, resulting in poor applicability for objects of complex media. Based on this, the three-dimensional model generation method based on polarization information of some embodiments of the present disclosure, first, receives each polarization picture information, wherein each polarization picture information in the above-mentioned each polarization picture information includes a polarization intensity picture, a polarization angle picture and a color picture. Thus, a picture with polarization information can be obtained. Then, based on the above-mentioned polarization picture information, each camera pose information is generated, wherein each camera pose information in the above-mentioned camera pose information includes camera intrinsic parameter information, camera extrinsic parameter information and camera origin information. Thus, the pose information of the camera corresponding to each polarization picture in the world coordinate system can be obtained. Then, based on the above-mentioned camera pose information, each sampling light direction information is generated. Thus, the direction of each sampling light can be obtained. Secondly, based on the above-mentioned sampling light direction information and the pre-trained signed distance network, each sampling point is generated. Thus, the sampling point related to the three-dimensional model can be obtained. Then, the sampling points that meet the preset surface point conditions in the above-mentioned sampling points are determined as surface points to obtain each surface point. Thus, the surface point of the three-dimensional model surface can be obtained. Then, based on the above-mentioned surface points and the pre-trained light decomposition network, each light reflection prediction information is generated, wherein each light reflection prediction information in the above-mentioned light reflection prediction information includes mirror reflection prediction information and diffuse reflection component prediction information. Thus, the light reflection related prediction information of each surface point can be obtained. Secondly, based on the above-mentioned surface points and the pre-trained color network, each pixel color prediction information is generated. Thus, the pixel color related prediction information of each surface point can be obtained. Then, based on the above-mentioned light reflection prediction information, the pre-stored coordinate system transformation matrix information and the pre-stored polarization matrix information, each light reflection information is generated, wherein each light reflection information in the above-mentioned light reflection information includes mirror reflection information, diffuse reflection component information and mirror vector information. Thus, the light reflection related information of each surface point can be obtained. Then, based on the above-mentioned surface points, the above-mentioned each sampling light direction information and the above-mentioned each camera pose information, the camera pose group of each surface point is generated. Thus, the camera pose group of each surface point under different camera poses can be obtained.Secondly, based on the camera pose groups of each surface point, each viewpoint inconsistency point is generated. Thus, the viewpoint inconsistency point can be obtained. Then, based on the surface points, each reflection confidence information is generated. Thus, each reflection confidence information can be obtained. Then, based on the pixel color prediction information, each light reflection information, each viewpoint inconsistency point, each reflection confidence information and each polarization picture information, a three-dimensional model network is generated. Thus, a network model for three-dimensional model generation can be obtained. Finally, the three-dimensional model network is stored in a data storage server. Thus, the three-dimensional model network can be stored as a network used for three-dimensional model generation. Also, because the three-dimensional model network is generated by calculating each light reflection information, a more accurate reconstruction result that conforms to physical reality is achieved, so that the model can express other complex surface features of the object. The effect is improved. Also, because the light reflection information is generated using polarization matrix information, the propagation mode of light can be obtained, so that the pixel color can be better predicted, and a predicted color that matches the actual color is generated, thereby improving the applicability of objects in complex media. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0014] Figure 1 is a flow chart of some embodiments of a method for generating a three-dimensional model based on polarization information according to the present disclosure;

[0015] Figure 2 is a flowchart of generating a three-dimensional model network in some embodiments of the present disclosure;

[0016] Figure 3 is a schematic structural diagram of some embodiments of a three-dimensional model generating device based on polarization information according to the present disclosure;

[0017] Figure 4 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0018] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0019] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0020] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0021] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0022] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0023] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0024] Figure 1 The process 100 of some embodiments of the method for generating a three-dimensional model based on polarization information according to the present disclosure is shown. The method for generating a three-dimensional model based on polarization information comprises the following steps:

[0025] Step 101, receiving each polarization picture information.

[0026] In some embodiments, the execution subject (e.g., a computing device) of the three-dimensional model generation method based on polarization information may receive various polarization picture information. Among them, the above-mentioned various polarization picture information may be pictures taken at various angles for characterizing the target object that needs to be three-dimensionally modeled. Each polarization picture information in the above-mentioned various polarization picture information may include a polarization intensity picture, a polarization angle picture, and a color picture. The above-mentioned polarization intensity picture may be a picture for characterizing the polarization intensity of the image. The above-mentioned polarization intensity picture may be a DoLP (Degree of Linear Polarization) picture. The above-mentioned polarization angle picture may be a picture for characterizing the polarization angle of the image. The above-mentioned polarization angle picture may be an AoLP (Angle of Linear Polarization) picture. The above-mentioned color picture may be a picture for characterizing the color of the image. The above-mentioned color picture may be an RGB picture.

[0027] Step 102: Generate each camera pose information based on each polarization picture information.

[0028] In some embodiments, the above-mentioned execution subject can generate each camera pose information based on the above-mentioned each polarization picture information. Among them, each camera pose information in the above-mentioned each camera pose information can be information related to the pose of the camera in the world coordinate system when shooting the polarization picture information. Each camera pose information in the above-mentioned each camera pose information can include camera intrinsic parameter information, camera extrinsic parameter information and camera origin information. The above-mentioned camera intrinsic parameter information can be used to characterize the intrinsic parameters (Intrinsics) of the camera. The above-mentioned camera extrinsic parameter information can be used to characterize the extrinsics (Extrinsics) of the camera. In practice, for each polarization picture in the above-mentioned each polarization picture, the above-mentioned execution subject can input the above-mentioned polarization picture information into the computer vision software to obtain the camera pose information. Among them, the above-mentioned computer vision software can be Colmap software.

[0029] Step 103: Generate each sampling light direction information based on each camera posture information.

[0030] In some embodiments, the execution subject may generate each sampling light direction information based on each camera position information, wherein each sampling light direction information in the each sampling light direction information may be used to characterize the direction of the sampling light.

[0031] In some optional implementations of some embodiments, the execution subject may generate each sampling light direction information based on each camera pose information through the following steps:

[0032] In the first step, for each of the above-mentioned camera pose information, perform the following steps:

[0033] In a first sub-step, the polarization image information corresponding to the camera posture information is determined as the target polarization image information.

[0034] In the second sub-step, for each pixel in the color image included in the target polarization image information, the following steps are performed:

[0035] Sub-step 1: Generate world coordinate information based on the camera extrinsic information and the pixel points included in the camera pose information. The world coordinate information can be the coordinates of the pixel points in the world coordinate system. In practice, the execution subject can input the camera extrinsic information and the pixel points into a first preset formula to obtain the world coordinate information. The first preset formula can be world coords =pose×pixel points .world coords is the world coordinate information. pose is the camera external parameter information. pixel points is the coordinate of the pixel in the above color image. × is matrix multiplication.

[0036] Sub-step 2: Generate initial light direction information based on the world coordinate information and the camera origin information included in the camera pose information. The initial light direction information may be used to characterize the direction of the sampling light. In practice, the execution subject may input the world coordinate information and the camera origin information into a second preset formula to obtain the initial light direction information. The second preset formula may be v=world coords -cam loc v is the initial light direction information. world coords It is the world coordinate information. loc The camera origin information.

[0037] Sub-step three: normalize the initial light direction information to obtain sampled light direction information. In practice, the execution subject may use a normalization function to normalize the initial light direction information to obtain sampled light direction information. The normalization function may be a normalize() function.

[0038] Step 104: Generate each sampling point based on the direction information of each sampling light and the pre-trained signed distance network.

[0039] In some embodiments, the execution subject may generate each sampling point based on the above-mentioned sampling light direction information and the pre-trained signed distance network. Among them, each of the above-mentioned sampling points may be related information for characterizing a point on the above-mentioned sampling light direction information in the world coordinate system. The above-mentioned sampling point may include sampling point coordinates, distance information, feature information and surface point information. The above-mentioned sampling point coordinates may be coordinates for characterizing the sampling point in the world coordinate system. The above-mentioned distance information may be used to characterize the distance from the sampling point to the surface of the object. The above-mentioned feature information may be used to characterize the features corresponding to the above-mentioned sampling point. The above-mentioned surface point information may be text information for characterizing whether the above-mentioned sampling point is a surface point. The above-mentioned signed distance network may be a multi-layer perceptron network. The above-mentioned signed distance network may include eight linear layers for extracting feature information, each layer being a 256-dimensional linear layer. The input of the above-mentioned signed distance network is the sampling point coordinate x in space, and the output result is the distance d from the sampling point x to the surface of the object, and the feature z corresponding to each sampling point.

[0040] In the process of adopting technical solutions to solve the above technical problems, the following problems are often accompanied:

[0041] After obtaining the sampling light direction, only using the points obtained by the light direction to perform three-dimensional modeling on the object results in some points that are not helpful for modeling being used as modeling points, thus making the effect of the established three-dimensional model poor.

[0042] Faced with the above technical problems, we decided to adopt the following solutions:

[0043] In some optional implementations of some embodiments, the execution subject may generate each sampling point based on the above-mentioned each sampling light direction information and a pre-trained signed distance network through the following steps:

[0044] In the first step, for each piece of sampled light direction information in the above-mentioned sampled light direction information, the following steps are performed:

[0045] In the first sub-step, the camera pose information corresponding to the above-mentioned sampled light direction information is determined as the target camera pose information.

[0046] The second sub-step is to generate sampling point coordinates based on the camera origin information and the sampling light direction information included in the target camera pose information. In practice, first, the execution subject may determine the light passing through the sampling direction information of the camera origin information as the target light. Then, the execution subject may randomly sample a point from the target light as the sampling point coordinates.

[0047] The third sub-step is to execute the following sampling point loop steps based on the sampling point coordinates:

[0048] Sub-step 1: Input the sampling point coordinates into the above-mentioned signed distance network to obtain distance information and feature information. The above-mentioned distance information can be used to characterize the distance from the sampling point to the surface of the object. The above-mentioned feature information can be used to characterize the features corresponding to the above-mentioned sampling points. In practice, the above-mentioned execution entity can input the above-mentioned sampling point coordinates into the above-mentioned signed distance network to obtain distance information and feature information.

[0049] Sub-step 2: Based on the above distance information, the above sampling light direction information and the sampling point coordinates, generate the sampling point coordinates to update the sampling point coordinates. In practice, the above execution entity can input the above distance information, the above sampling light direction information and the above sampling point coordinates into a third preset formula to obtain the updated sampling point coordinates. Among them, the above third preset formula can be x'=x+d*v. x' is the updated sampling point coordinates. X is the sampling point coordinates. d is the distance information. V is the sampling light direction information.

[0050] Sub-step three, in response to determining that the number of cycles of the sampling point cycle step does not meet the preset sampling point cycle condition, the distance information and the feature information are updated, and based on the updated sampling point coordinates, the sampling point cycle step is executed again. The preset sampling point cycle condition may be that the number of cycles of the sampling point cycle step is greater than a preset cycle threshold. The preset cycle threshold may be a pre-set value, which is not limited here.

[0051] Sub-step 4: In response to determining that the number of cycles of the sampling point cycle step satisfies the preset sampling point cycle condition, the updated sampling point coordinates, the updated distance information, and the updated feature information are integrated to obtain the sampling point. In practice, the execution entity may combine the updated sampling point coordinates, the updated distance information, and the updated feature information into a sampling point.

[0052] The fourth sub-step is, in response to determining that the distance information included in the above sampling point meets the preset distance threshold condition, executing the following steps:

[0053] Sub-step 1: determining the first preset surface point information as the surface point information. The preset distance threshold condition may be that the distance information satisfies the preset distance threshold. The preset distance threshold may be a preset value, which is not limited here. The first preset surface point information may be text information used to characterize that the sampling point is a surface point. For example, the first preset surface point information may be "Y".

[0054] Sub-step two: adding the surface point information to the sampling point to update the sampling point.

[0055] In a fifth sub-step, in response to determining that the distance information included in the sampling point does not meet the preset distance threshold condition, executing the following steps:

[0056] Sub-step 1: determining the second preset surface point information as the surface point information. The second preset surface point information may be text information used to indicate that the sampling point is not a surface point. For example, the second preset surface point information may be "N".

[0057] Sub-step two: adding the surface point information to the sampling point to update the sampling point.

[0058] The above technical solution and its related contents, as an inventive point of an embodiment of the present disclosure, solve the problem that "after obtaining the sampling light direction, only the points obtained by the light direction are used to perform three-dimensional modeling of the object, resulting in some points that are not helpful for modeling as modeling points, thereby making the quality of the established three-dimensional model poor". The factors that lead to the poor quality of the three-dimensional model are often as follows: after obtaining the sampling light direction, only the points obtained by the light direction are used to perform three-dimensional modeling of the object, resulting in some points that are not helpful for modeling as modeling points. If the above factors are solved, the effect of improving the quality of the three-dimensional model can be achieved. In order to achieve this effect, the present disclosure first performs the following steps for each sampling light direction information in the above-mentioned sampling light direction information: determining the camera pose information corresponding to the above-mentioned sampling light direction information as the target camera pose information. Then, based on the camera origin information included in the above-mentioned target camera pose information and the above-mentioned sampling light direction information, the sampling point coordinates are generated. Thus, a randomly generated sampling point coordinate in the sampling light direction can be obtained. Then, based on the sampling point coordinates, the following sampling point loop steps are performed: the sampling point coordinates are input into the above-mentioned signed distance network to obtain distance information and feature information. Secondly, based on the distance information, the sampling light direction information and the sampling point coordinates, the sampling point coordinates are generated to update the sampling point coordinates. Thus, the sampling point coordinates can be updated to obtain better sampling point coordinates. Then, in response to determining that the number of cycles of the sampling point loop step does not meet the preset sampling point loop condition, the distance information and feature information are updated, and based on the updated sampling point coordinates, the sampling point loop step is performed again. Thus, the sampling point coordinates that do not meet the loop condition can be re-looped. Then, in response to determining that the number of cycles of the sampling point loop step meets the preset sampling point loop condition, the updated sampling point coordinates, the updated distance information and the updated feature information are integrated to obtain the sampling point. Thus, the optimal sampling point can be obtained. Secondly, in response to determining that the distance information included in the sampling point meets the preset distance threshold condition, the following steps are performed: the first preset surface point information is determined as the surface point information. Then, the surface point information is added to the sampling point to update the sampling point. Then, in response to determining that the distance information included in the above sampling point does not meet the above preset distance threshold condition, the following steps are performed: second preset surface point information is determined as surface point information. Finally, the above surface point information is added to the above sampling point to update the above sampling point. Thus, it can be obtained that the surface point information can be added to the sampling point. Also, because the sampling point coordinates are iterated step by step through a loop. Therefore, the obtained sampling point coordinates and the real coordinates of the three-dimensional object are more accurate. Also, because the distance information included in the sampling point is judged by the preset distance threshold condition, the sampling point of the surface point information is obtained, and the quality of the established three-dimensional model is improved.

[0059] Step 105: Determine the sampling points that meet the preset surface point conditions among the sampling points as surface points to obtain the surface points.

[0060] In some embodiments, the execution entity may determine the sampling points that meet the preset surface point conditions among the above-mentioned sampling points as surface points to obtain the surface points. The preset surface point condition may be that the surface point information included in the above-mentioned sampling points is "Y". Each of the above-mentioned surface points may be used to characterize the sampling points that meet the preset surface point condition.

[0061] Step 106 , generating respective light reflection prediction information based on respective surface points and the pre-trained light decomposition network.

[0062] In some embodiments, the execution subject may generate each light reflection prediction information based on each surface point and a pre-trained light decomposition network. Each of the light reflection prediction information may be a Stokes vector corresponding to the surface point. The Stokes vector is a 4-dimensional vector. Since the circular polarization represented by the fourth dimension is relatively weak, only the Stokes vectors of the first three dimensions are considered. Each of the light reflection prediction information may include specular reflection prediction information and diffuse reflection component prediction information. The Stokes vector may be divided into a diffuse reflection component and a specular reflection component. The specular reflection prediction information may be the specular reflection component included in the Stokes vector. The diffuse reflection component prediction information may be the diffuse reflection component included in the Stokes vector. The light decomposition network may be a neural network including three multi-layer perceptron networks. Each of the multi-layer perceptron networks may include 4 linear layers, each of which is a 256-dimensional linear layer. The first multi-layer perceptron network may be used to predict the first dimension of the Stokes vector. The first dimension of the Stokes vector includes the first dimension of the specular reflection component and the first dimension of the diffuse reflection component. The second multilayer perceptron network can be used to predict the second and third dimensions of the specular reflection component included in the output Stokes vector. The third multilayer perceptron network can be used to predict the second and third dimensions of the diffuse reflection component included in the output Stokes vector.

[0063] In some optional implementations of some embodiments, the execution subject may generate each light reflection prediction information based on each surface point and a pre-trained light decomposition network through the following steps:

[0064] In the first step, for each of the above surface points, perform the following steps:

[0065] The first sub-step is to input the surface point into the light decomposition network to obtain specular reflection prediction information and diffuse reflection component prediction information. In practice, the execution subject may input the surface point into the light decomposition network to obtain specular reflection prediction information and diffuse reflection component prediction information.

[0066] The second sub-step is to integrate the specular reflection prediction information and the diffuse reflection component prediction information to obtain the light reflection prediction information. In practice, the execution subject may combine the specular reflection prediction information and the diffuse reflection component prediction information into the light reflection prediction information.

[0067] Step 107, generating color prediction information of each pixel based on each surface point and the pre-trained color network.

[0068] In some embodiments, the execution entity may generate color prediction information of each pixel based on the surface points and the pre-trained color network. The pixel color prediction information may be prediction information for characterizing the pixel color corresponding to the surface point. The color network may be a multi-layer perceptron network. The color network may include eight linear layers, each of which is a 256-dimensional linear layer. The input of the color network may be the coordinate x of the sampling point in space, and the output may be the predicted rgb color.

[0069] In some optional implementations of some embodiments, the execution subject may generate color prediction information of each pixel based on the surface points and the pre-trained color network through the following steps:

[0070] In the first step, for each of the surface points, the surface point is input into the color network to obtain pixel color prediction information. In practice, the execution subject may input the surface point into the color network to obtain pixel color prediction information.

[0071] Step 108 : generating each light reflection information based on each light reflection prediction information, pre-stored coordinate system transformation matrix information and pre-stored polarization matrix information.

[0072] In some embodiments, the execution subject may generate each light reflection information based on the above-mentioned light reflection prediction information, the pre-stored coordinate system transformation matrix information and the pre-stored polarization matrix information. Among them, the above-mentioned coordinate system transformation matrix information may be a rotation matrix for characterizing the coordinate system transformation. The above-mentioned polarization matrix information may be a Muller matrix. The above-mentioned polarization matrix information may include a specular reflection polarization matrix and a diffuse reflection polarization matrix. The above-mentioned specular reflection polarization matrix may be a Muller matrix representing specular reflection. The above-mentioned diffuse reflection polarization matrix may be a Muller matrix representing diffuse reflection. Each of the above-mentioned light reflection information may be used to characterize the processed light reflection prediction information. Each of the above-mentioned light reflection information may include specular reflection information, diffuse reflection component information and specular vector information. The above-mentioned specular reflection information may be the processed specular reflection prediction information. The above-mentioned diffuse reflection component information may be the processed diffuse reflection component prediction information. The above-mentioned specular vector information may be a Stokes vector.

[0073] In some optional implementations of some embodiments, the execution subject may generate each light reflection information based on the above-mentioned each light reflection prediction information, the pre-stored coordinate system transformation matrix information and the pre-stored polarization matrix information through the following steps:

[0074] In the first step, for each light reflection prediction information in the above light reflection prediction information, the following steps are performed:

[0075] The first sub-step is to generate the mirror reflection information based on the mirror reflection prediction information included in the light reflection prediction information, the pre-stored coordinate system change matrix information and the mirror reflection polarization matrix included in the polarization matrix information. The mirror reflection information may be a vector used to characterize the information related to the mirror reflection in the camera coordinate system. The camera coordinate system may be the coordinate system where the camera is located, which is not limited here. The coordinate system change matrix information may be a coordinate transformation matrix used to characterize the changes between the various coordinate systems. The various coordinate systems may be the world coordinate system, the Mueller coordinate system or the camera coordinate system. The Mueller coordinate system may be the coordinate system where the Mueller matrix is ​​located. The camera coordinate system may be the coordinate system where the camera is located. In practice, the execution entity may input the mirror reflection prediction information, the coordinate system change matrix information and the mirror reflection polarization matrix into a fourth preset formula to obtain the mirror reflection information. The fourth preset formula may be Specular reflection information. is the specular reflection prediction information. i is the coordinate transformation matrix from the world coordinate system to the Muller coordinate system. ris the mirror reflection polarization matrix. R o is the coordinate transformation matrix from the Mueller coordinate system to the world coordinate system. c It is the coordinate transformation matrix from the world coordinate system to the camera coordinate system.

[0076] The second sub-step is to generate diffuse reflection component information based on the diffuse reflection component prediction information included in the light reflection prediction information, the coordinate system change matrix information, and the diffuse reflection polarization matrix included in the polarization matrix information. The diffuse reflection component information can be a vector used to characterize diffuse reflection-related information in the camera coordinate system. In practice, the execution subject can input the diffuse reflection component prediction information, the coordinate system change matrix information, and the diffuse reflection polarization matrix into a fifth preset formula to obtain the diffuse reflection component information. The fifth preset formula can be It is the diffuse reflection component information. Predicts information for the diffuse component. R i is the coordinate transformation matrix from the world coordinate system to the Muller coordinate system. d is the diffuse reflection polarization matrix. R o is the coordinate transformation matrix from the Mueller coordinate system to the world coordinate system. c It is the coordinate transformation matrix from the world coordinate system to the camera coordinate system.

[0077] The third sub-step is to generate specular vector information based on the specular reflection information and the diffuse reflection component information. In practice, the execution subject may input the specular reflection information and the diffuse reflection component information into a sixth preset formula to obtain the specular vector information. The sixth preset formula may be Among them, Stokes out It is the mirror vector information. Prediction information for specular reflection. It is the diffuse reflection component information.

[0078] The fourth sub-step is to integrate the specular reflection information, the diffuse reflection component information and the specular vector information to obtain light reflection information. In practice, the execution subject may combine the specular reflection information, the diffuse reflection component information and the specular vector information into light reflection information.

[0079] Step 109: Generate a camera pose group for each surface point based on each surface point, each sampling light direction information and each camera pose information.

[0080] In some embodiments, the execution subject may generate a camera pose group for each surface point based on the surface points, the sampling light direction information and the camera pose information. Each surface point camera pose group in the surface point camera pose group may be the camera pose information for each surface point corresponding to the surface points.

[0081] In some optional implementations of some embodiments, the execution subject may generate a camera pose group for each surface point based on the surface points, the sampling light direction information, and the camera pose information by the following steps:

[0082] In the first step, for each of the above surface points, perform the following steps:

[0083] In the first sub-step, the sampling light direction information corresponding to the surface point is determined as the reference sampling light direction information.

[0084] In the second sub-step, the camera pose information corresponding to the reference sampling light direction information is determined as the reference camera pose information.

[0085] In the third sub-step, based on the reference camera pose information, the camera pose information that satisfies the preset camera condition among the camera pose information is determined as a surface point camera pose group. The preset camera condition may be that the polarization image information corresponding to the other camera pose information other than the reference camera pose information can observe the surface point. In practice, the execution subject may determine the camera pose information that satisfies the preset camera condition among the camera pose information as a surface point camera pose group.

[0086] Step 110, generating each perspective inconsistency point based on each surface point camera pose group.

[0087] In some embodiments, the execution subject may generate each perspective inconsistency point based on each surface point camera pose group, wherein each of the perspective inconsistency points may be a surface point where pixel color prediction information of the surface point corresponding to the surface point camera pose group changes dramatically.

[0088] In some optional implementations of some embodiments, the execution subject may generate each perspective inconsistency point based on each surface point camera pose group through the following steps:

[0089] In the first step, for each of the above surface point camera pose groups, perform the following steps:

[0090] In the first sub-step, the surface point corresponding to the surface point camera pose group is determined as the target surface point.

[0091] In the second sub-step, for each camera pose information in the above surface point camera pose group, perform the following steps:

[0092] Sub-step 1: Generate camera surface points based on the target surface points and the camera extrinsic information included in the camera pose information. The camera surface points may be coordinates in the camera coordinate system. In practice, the execution subject may input the sampling point coordinates included in the target surface points and the camera extrinsic information into the seventh preset formula to obtain the camera surface points. The seventh preset formula may be Y=pose -1 *X. Y is the camera surface point. pose is the camera external parameter information. pose -1 is the inverse matrix of the camera's external parameter information. X is the coordinate of the sampling point.

[0093] Sub-step two, based on the camera surface point and the camera intrinsic parameter information included in the camera pose information, generate pixel coordinates. The pixel coordinates may be coordinates used to characterize the target surface point in the pixel coordinate system in the color image. The pixel coordinate system may be a coordinate system used to characterize the color image, which is not limited here. In practice, the execution entity may input the camera surface point and the camera intrinsic parameter information into the eighth preset formula to obtain pixel coordinates. The eighth preset formula may be (u, v) = intrinsic parameter * Y. (u, v) is the pixel coordinate. is the camera intrinsic parameter information. Y is the camera surface point.

[0094] Sub-step three: determining the polarization image information corresponding to the above-mentioned camera posture information as the reference polarization image information.

[0095] Sub-step 4: Generate target surface point color information based on the pixel coordinates and the color image included in the reference polarization image information. The target surface point color information may be the value of the color corresponding to the pixel point corresponding to the pixel coordinates in the color image. In practice, the execution subject may determine the color corresponding to the pixel point corresponding to the pixel coordinates in the color image as the target surface point color information.

[0096] In the third sub-step, the obtained variance of the color information of each target surface point is determined as the target surface point variance.

[0097] In a fourth sub-step, in response to determining that the variance of the target surface point satisfies a preset color variance condition, the target surface point is determined as a perspective inconsistent point. The preset color variance condition may be that the variance of the target surface point is greater than a preset variance threshold. The preset variance threshold may be a preset value, which is not limited here.

[0098] Step 111: Generate each reflection confidence information based on each surface point.

[0099] In some embodiments, the execution entity may generate various reflection confidence information based on the various surface points, wherein each reflection confidence information in the various reflection confidence information may be used to characterize the confidence of the intensity of the reflected light.

[0100] In some optional implementations of some embodiments, the execution subject may generate each reflection confidence information based on each surface point through the following steps:

[0101] In the first step, for each of the above surface points, perform the following steps:

[0102] In the first sub-step, the sampling light direction information corresponding to the surface point is determined as the target sampling light direction information.

[0103] In the second sub-step, the polarization image information corresponding to the surface point is determined as the target polarization image information.

[0104] The third sub-step is to generate a surface point normal vector based on the surface point and the signed distance network. The surface point normal vector can be used to represent the normal vector of the surface point on the surface of the photographed object. In practice, the execution subject can derive the sampling point coordinates included in the surface point through the signed distance network to obtain the surface point normal vector.

[0105] The fourth sub-step is to generate incident light intensity information based on the target polarization image information. The incident light intensity information may be used to characterize the intensity of light received by the camera. In practice, the execution subject may convert the color image included in the target polarization image information into a grayscale image to obtain a grayscale color image. Then, the execution subject may normalize the grayscale color image to obtain the incident light intensity information.

[0106] The fifth sub-step is to generate target incident light direction information based on the target sampling light direction information. The target incident light direction information may be the reverse direction used to characterize the sampling light direction. In practice, the execution subject may determine the reverse direction of the target sampling light direction information as the target incident light direction information.

[0107] The sixth sub-step is to generate target refracted light direction information based on the target incident light direction information. The target refracted light direction information may be the refraction direction of the target incident light direction information. In practice, the execution subject may obtain the target refracted light direction information by applying the law of refraction to the target incident light direction information.

[0108] The seventh sub-step is to generate a light coefficient based on the pre-stored refractive index information, the surface point normal vector, the target incident light direction information and the target refracted light direction information. The refractive index information may be used to characterize the refractive index in the target medium. The refractive index information may include the external medium refractive index and the object refractive index. The external medium refractive index may be the refractive index of air. The object refractive index may be the refractive index of the photographed object. The light coefficient may be a Fresnel coefficient. In practice, the execution entity may input the refractive index information, the target incident light direction information and the target refracted light direction information into the ninth preset formula to obtain the light coefficient. The ninth preset formula may be F is the light coefficient. η i is the refractive index of the external medium. i is the incident light direction information of the target. n is the surface point normal vector. η t V is the refractive index of the object. t Provides refracted light direction information for the target.

[0109] In the eighth sub-step, based on the light coefficient and the incident light intensity information, the reflected light intensity information is generated. The reflected light intensity information can be used to characterize the intensity of the light reflected by the object. In practice, the execution subject can input the light coefficient and the incident light intensity information into a tenth preset formula to obtain the reflected light intensity information. The tenth preset formula can be I r =F·I i I r is the reflected light intensity information. F is the light coefficient. i It is the incident light intensity information.

[0110] The ninth sub-step is to generate refracted light intensity information based on the light coefficient and the incident light intensity information. The refracted light intensity information may be used to characterize the intensity of the refracted light of the object. In practice, the execution subject may input the light coefficient and the incident light intensity information into an eleventh preset formula to obtain the refracted light intensity information. The eleventh preset formula may be: t =(1-F)·I i I t is the refracted light intensity information. F is the light coefficient. i It is the incident light intensity information.

[0111] The tenth sub-step is to generate reflection confidence information based on the above-mentioned reflected light intensity information and the above-mentioned refracted light intensity information. In practice, the above-mentioned execution subject can input the above-mentioned reflected light intensity information and the above-mentioned refracted light intensity information into a twelfth preset formula to obtain the reflection confidence information. Among them, the above-mentioned twelfth preset formula can be W is the reflection confidence information. r It is the reflected light intensity information. t It is the intensity information of refracted light.

[0112] Step 112, generating a three-dimensional model network based on each pixel color prediction information, each light reflection information, each viewing angle inconsistency point, each reflection confidence information and each polarization image information.

[0113] In some embodiments, the execution entity may generate a three-dimensional model network based on the color prediction information of each pixel, the light reflection information, the perspective inconsistency points, the reflection confidence information and the polarization picture information. The three-dimensional model network may be a neural network that provides data information for subsequent three-dimensional model generation. In practice, the execution entity may use the color prediction information of each pixel, the light reflection information, the perspective inconsistency points, the reflection confidence information and the polarization picture information as parameters for neural network training. The neural network is optimized and trained using the Adam optimizer. The difference between the color prediction information of each pixel and the true color is used as the loss of the neural network. Through the back-propagation algorithm, as the number of iterations increases, the neural network finally converges and the training is completed to obtain a three-dimensional model network. The specific process of generating the three-dimensional model network can be referred to. Figure 2 .

[0114] Figure 2 It is a flowchart of generating a three-dimensional model network in some embodiments of the present disclosure. Among them, "calibration of camera pose" can be steps 101-102. "Calculating the direction of light emitted by the camera" can be step 103. "Random sampling of sampling points" can be step 104. "Sending the sampling points to the signed distance network to obtain the distance d and the feature f" can be step 104. "Using the light decomposition network to obtain the specular reflection component and the diffuse reflection component" can be steps 106 and 108. "Using the color network to predict the color of the pixel point" can be step 107. "Determining the visibility of the surface point" can be step 105. "Calculating the color change variance var" can be step 110. "Calculating the loss and back propagating with the stokes component and RGB value collected by the camera" can be step 109 and steps 111-112.

[0115] Step 113, storing the three-dimensional model network in a data storage server.

[0116] In some embodiments, the execution entity may store the three-dimensional model network in a data storage server, wherein the data storage server may be a server for managing and storing models and model-related information.

[0117] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the three-dimensional model generation method based on polarization information of some embodiments of the present disclosure, the way to solve other complex surface features of objects and the applicability to objects of complex media are improved. Specifically, the reason for the poor way to solve other complex surface features of objects is that the light in the three-dimensional reconstruction technology based on neural radiation field is mainly transmitted in a straight line, and the effect of solving other complex surface features of objects (such as transparency) is poor. And because the pixel color is predicted by volume rendering of direct light, the applicability to objects of complex media is poor. Based on this, the three-dimensional model generation method based on polarization information of some embodiments of the present disclosure first receives each polarization picture information, wherein each polarization picture information in the above polarization picture information includes a polarization intensity picture, a polarization angle picture and a color picture. Thus, a picture with polarization information can be obtained. Then, based on the above polarization picture information, each camera pose information is generated, wherein each camera pose information in the above camera pose information includes camera intrinsic parameter information, camera extrinsic parameter information and camera origin information. Thus, the pose information of the camera corresponding to each polarization picture in the world coordinate system can be obtained. Then, based on the above-mentioned camera pose information, the direction information of each sampled light is generated. Thus, the direction of each sampled light can be obtained. Secondly, based on the above-mentioned sampled light direction information and the pre-trained signed distance network, each sampling point is generated. Thus, the sampling points related to the three-dimensional model can be obtained. Then, the sampling points that meet the preset surface point conditions among the above-mentioned sampling points are determined as surface points to obtain each surface point. Thus, the surface points of the three-dimensional model surface can be obtained. Then, based on the above-mentioned surface points and the pre-trained light decomposition network, each light reflection prediction information is generated, wherein each light reflection prediction information in the above-mentioned light reflection prediction information includes mirror reflection prediction information and diffuse reflection component prediction information. Thus, the light reflection related prediction information of each surface point can be obtained. Secondly, based on the above-mentioned surface points and the pre-trained color network, each pixel color prediction information is generated. Thus, the pixel color related prediction information of each surface point can be obtained. Then, based on the above-mentioned light reflection prediction information, the pre-stored coordinate system transformation matrix information and the pre-stored polarization matrix information, each light reflection information is generated, wherein each light reflection information in the above-mentioned light reflection information includes mirror reflection information, diffuse reflection component information and mirror vector information. Thus, the light reflection related information of each surface point can be obtained. Then, based on the above-mentioned surface points, the above-mentioned sampled light direction information and the above-mentioned camera pose information, each surface point camera pose group is generated. Thus, each surface point camera pose group under different camera poses can be obtained. Secondly, based on the above-mentioned surface point camera pose groups, each viewpoint inconsistency point is generated.Thus, the viewpoint inconsistency point can be obtained. Then, based on the above-mentioned surface points, each reflection confidence information is generated. Thus, each reflection confidence information can be obtained. Then, based on the above-mentioned pixel color prediction information, the above-mentioned light reflection information, the above-mentioned viewpoint inconsistency point, the above-mentioned reflection confidence information and the above-mentioned polarization picture information, a three-dimensional model network is generated. Thus, a network model for three-dimensional model generation can be obtained. Finally, the above-mentioned three-dimensional model network is stored in a data storage server. Thus, the three-dimensional model network can be stored as a network used for three-dimensional model generation. Also, because the three-dimensional model network is generated by calculating each light reflection information, a more accurate reconstruction result that conforms to physical reality is achieved, so that the effect of the model in solving other complex surface features of the object is improved. Also, because the light reflection information is generated using polarization matrix information, the propagation mode of light can be obtained, so that the pixel color can be better predicted, thereby improving the applicability to objects of complex media.

[0118] Further references Figure 3 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a three-dimensional model generation device based on polarization information. These device embodiments are similar to Figure 1 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices.

[0119] like Figure 2As shown, in some embodiments, the three-dimensional model generation device 300 based on polarization information includes: a receiving unit 301, a first generating unit 302, a second generating unit 303, a third generating unit 304, a determining unit 305, a fourth generating unit 306, a fifth generating unit 307, a sixth generating unit 308, a seventh generating unit 309, an eighth generating unit 310, a ninth generating unit 311, a tenth generating unit 312 and a storage unit 313. The receiving unit 301 is configured to receive various polarization picture information, wherein each polarization picture information in the above-mentioned various polarization picture information includes a polarization intensity picture, a polarization angle picture and a color picture; the first generating unit 302 is configured to generate various camera pose information based on the above-mentioned various polarization picture information, wherein each camera pose information in the above-mentioned various camera pose information includes camera intrinsic parameter information, camera extrinsic parameter information and camera origin information; the second generating unit 303 is configured to generate various sampling light direction information based on the above-mentioned various camera pose information; the third generating unit 304, configured to generate each sampling point based on the above-mentioned each sampling light direction information and the pre-trained signed distance network; the determination unit 305, configured to determine the sampling points that meet the preset surface point conditions among the above-mentioned each sampling point as surface points, and obtain each surface point; the fourth generation unit 306, configured to generate each light reflection prediction information based on the above-mentioned each surface point and the pre-trained light decomposition network, wherein each light reflection prediction information in the above-mentioned each light reflection prediction information includes specular reflection prediction information and diffuse reflection component prediction information; the fifth generation unit 307, configured to The system is configured to generate color prediction information of each pixel based on the surface points and the pre-trained color network. The sixth generation unit 308 is configured to generate light reflection information based on the light reflection prediction information, the pre-stored coordinate system transformation matrix information and the pre-stored polarization matrix information, wherein each light reflection information in the light reflection information includes mirror reflection information, diffuse reflection component information and mirror vector information. The seventh generation unit 309 is configured to generate camera pose groups of each surface point based on the surface points, the sampling light direction information and the camera pose information. The eighth generation unit 310 is configured to generate perspective inconsistency points based on the surface point camera pose groups. The ninth generation unit 311 is configured to generate reflection confidence information based on the surface points. The tenth generation unit 312 is configured to generate a three-dimensional model network based on the pixel color prediction information, the light reflection information, the perspective inconsistency points, the reflection confidence information and the polarization picture information. The storage unit 313 is configured to store the three-dimensional model network in a data storage server.

[0120] It is understood that the units described in the device 300 are similar to those described in the reference Figure 1 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the device 300 and the units contained therein, and will not be described in detail here.

[0121] Reference below Figure 4 , which shows a structural schematic diagram of an electronic device 400 suitable for implementing some embodiments of the present disclosure. Figure 4 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0122] like Figure 4 As shown, the electronic device 400 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 408 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the electronic device 400 are also stored. The processing device 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0123] Typically, the following devices may be connected to the I / O interface 405: an input device 406 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 407 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 408 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 409. The communication device 409 may allow the electronic device 400 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 4 The electronic device 400 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 4 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0124] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 409, or installed from the storage device 408, or installed from the ROM 402. When the computer program is executed by the processing device 401, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0125] It should be noted that the computer-readable medium recorded in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0126] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0127] The computer-readable medium may be included in the electronic device; or it may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: receives various polarization picture information, wherein each polarization picture information in the various polarization picture information includes a polarization intensity picture, a polarization angle picture, and a color picture; generates various camera pose information based on the various polarization picture information, wherein each camera pose information in the various camera pose information includes camera intrinsic parameter information, camera extrinsic parameter information, and camera origin information; generates various sampling light direction information based on the various camera pose information; generates various sampling points based on the various sampling light direction information and a pre-trained signed distance network; determines the sampling points in the various sampling points that meet the preset surface point conditions as surface points to obtain various surface points; generates various light reflection prediction information based on the various surface points and a pre-trained light decomposition network, wherein each light reflection prediction information in the various light reflection prediction information Including mirror reflection prediction information and diffuse reflection component prediction information; based on the above-mentioned surface points and a pre-trained color network, generating each pixel color prediction information; based on the above-mentioned light reflection prediction information, pre-stored coordinate system transformation matrix information and pre-stored polarization matrix information, generating each light reflection information, wherein each light reflection information in the above-mentioned light reflection information includes mirror reflection information, diffuse reflection component information and mirror vector information; based on the above-mentioned surface points, the above-mentioned each sampling light direction information and the above-mentioned each camera pose information, generating each surface point camera pose group; based on the above-mentioned each surface point camera pose group, generating each viewpoint inconsistency point; based on the above-mentioned each surface point, generating each reflection confidence information; based on the above-mentioned each pixel color prediction information, the above-mentioned each light reflection information, the above-mentioned each viewpoint inconsistency point, the above-mentioned each reflection confidence information and the above-mentioned each polarization picture information, generating a three-dimensional model network; storing the above-mentioned three-dimensional model network in a data storage server.

[0128] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0129] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0130] The units described in some embodiments of the present disclosure may be implemented by software or by hardware. The described units may also be arranged in a processor, for example, may be described as: a processor includes a receiving unit, a first generating unit, a second generating unit, a third generating unit, a determining unit, a fourth generating unit, a fifth generating unit, a sixth generating unit, a seventh generating unit, an eighth generating unit, a ninth generating unit, a tenth generating unit and a storage unit. Among them, the names of these units do not constitute a limitation on the units themselves in some cases, for example, the receiving unit may also be described as a "unit for receiving information of each polarization picture".

[0131] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0132] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.

Claims

1. A method for generating a three-dimensional model based on polarization information, comprising: Receiving various polarization picture information, wherein each polarization picture information in the various polarization picture information includes a polarization intensity picture, a polarization angle picture and a color picture; Based on the polarization picture information, generate each camera pose information, wherein each camera pose information in the camera pose information includes camera intrinsic parameter information, camera extrinsic parameter information and camera origin information; Based on the position information of each camera, generate direction information of each sampling light; Generate each sampling point based on the direction information of each sampling light and a pre-trained signed distance network; Determine the sampling points that meet the preset surface point conditions among the sampling points as surface points to obtain the surface points; Based on the surface points and the pre-trained light decomposition network, generating respective light reflection prediction information, wherein each light reflection prediction information in the respective light reflection prediction information includes specular reflection prediction information and diffuse reflection component prediction information; Generate color prediction information of each pixel based on each surface point and a pre-trained color network; Generate each light reflection information based on each light reflection prediction information, pre-stored coordinate system transformation matrix information and pre-stored polarization matrix information, wherein each light reflection information in each light reflection information includes specular reflection information, diffuse reflection component information and specular vector information; Generate a camera pose group for each surface point based on each surface point, each sampling light direction information and each camera pose information; Based on the camera pose groups of each surface point, generating each perspective inconsistency point; Based on the respective surface points, generating respective reflection confidence information; Generate a three-dimensional model network based on the color prediction information of each pixel, the reflection information of each light, the inconsistency points of each viewing angle, the reflection confidence information and the polarization picture information; The three-dimensional model network is stored in a data storage server.

2. The method according to claim 1, wherein: The generating each sampling light direction information based on each camera pose information includes: For each piece of camera pose information, perform the following steps: Determine the polarization picture information corresponding to the camera posture information as the target polarization picture information; For each pixel point in the color image included in the target polarization image information, the following steps are performed: Generate world coordinate information based on the camera extrinsic parameter information included in the camera pose information and the pixel point; Generate initial light direction information based on the world coordinate information and the camera origin information included in the camera pose information; The initial light direction information is normalized to obtain sampled light direction information.

3. The method according to claim 1, wherein: The generating each light reflection prediction information based on each surface point and a pre-trained light decomposition network includes: For each of the surface points, the following steps are performed: Inputting the surface point into the light decomposition network to obtain specular reflection prediction information and diffuse reflection component prediction information; The specular reflection prediction information and the diffuse reflection component prediction information are integrated to obtain the light reflection prediction information.

4. The method according to claim 1, wherein: The generating of color prediction information of each pixel based on each surface point and a pre-trained color network comprises: For each of the surface points, the surface point is input into the color network to obtain pixel color prediction information.

5. The method according to claim 1, wherein: The polarization matrix information includes a mirror reflection polarization matrix and a diffuse reflection polarization matrix; and the generating of each light reflection information based on the reflection prediction information of each light, the pre-stored coordinate system transformation matrix information and the pre-stored polarization matrix information includes: For each piece of light reflection prediction information in the individual pieces of light reflection prediction information, the following steps are performed: Generate specular reflection information based on specular reflection prediction information included in the light reflection prediction information, pre-stored coordinate system change matrix information, and specular reflection polarization matrix included in the polarization matrix information; Generate diffuse reflection component information based on diffuse reflection component prediction information included in the light reflection prediction information, the coordinate system change matrix information, and the diffuse reflection polarization matrix included in the polarization matrix information; Generate specular vector information based on the specular reflection information and the diffuse reflection component information; The mirror reflection information, the diffuse reflection component information and the mirror vector information are integrated and processed to obtain light reflection information.

6. The method according to claim 1, wherein: The generating of each surface point camera pose group based on each surface point, each sampling light direction information and each camera pose information comprises: For each of the surface points, the following steps are performed: Determining the sampling light direction information corresponding to the surface point as reference sampling light direction information; Determine the camera pose information corresponding to the reference sampling light direction information as the reference camera pose information; Based on the reference camera pose information, each piece of camera pose information that meets a preset camera condition among the each piece of camera pose information is determined as a surface point camera pose group.

7. The method according to claim 1, wherein: The generating each perspective inconsistency point based on each surface point camera pose group includes: For each surface point camera pose group in the surface point camera pose groups, perform the following steps: Determine the surface point corresponding to the surface point camera pose group as the target surface point; For each camera pose information in the surface point camera pose group, perform the following steps: Generate a camera surface point based on the target surface point and the camera extrinsic parameter information included in the camera pose information; Generate pixel coordinates based on the camera surface point and the camera intrinsic parameter information included in the camera pose information; Determining the polarization picture information corresponding to the camera posture information as reference polarization picture information; Generate target surface point color information based on the pixel coordinates and the color picture included in the reference polarization picture information; The obtained variance of the color information of each target surface point is determined as the target surface point variance; In response to determining that the target surface point variance satisfies a preset color variance condition, the target surface point is determined as a perspective-inconsistent point.

8. The method according to claim 1, wherein: The generating each reflection confidence information based on each surface point comprises: For each of the surface points, the following steps are performed: Determine the sampling light direction information corresponding to the surface point as the target sampling light direction information; Determining the polarization image information corresponding to the surface point as target polarization image information; generating a surface point normal vector based on the surface point and the signed distance network; Based on the target polarization image information, generating incident light intensity information; Based on the target sampling light direction information, generating target incident light direction information; Based on the target incident light direction information, generating target refracted light direction information; Generate a light coefficient based on pre-stored refractive index information, the surface point normal vector, the target incident light direction information and the target refracted light direction information; Generate reflected light intensity information based on the light coefficient and the incident light intensity information; Generate refracted light intensity information based on the light coefficient and the incident light intensity information; Reflection confidence information is generated based on the reflected light intensity information and the refracted light intensity information.

9. A three-dimensional model generation device based on polarization information, comprising: A receiving unit configured to receive various polarization picture information, wherein each polarization picture information in the various polarization picture information includes a polarization intensity picture, a polarization angle picture and a color picture; A first generating unit is configured to generate each camera pose information based on each polarization picture information, wherein each camera pose information in each camera pose information includes camera intrinsic parameter information, camera extrinsic parameter information and camera origin information; A second generating unit is configured to generate each sampling light direction information based on each camera posture information; A third generating unit is configured to generate each sampling point based on the direction information of each sampling light and a pre-trained signed distance network; a determining unit configured to determine the sampling points that meet the preset surface point conditions among the sampling points as surface points, to obtain the surface points; a fourth generating unit configured to generate respective light reflection prediction information based on the respective surface points and a pre-trained light decomposition network, wherein each of the respective light reflection prediction information includes specular reflection prediction information and diffuse reflection component prediction information; a fifth generating unit, configured to generate color prediction information of each pixel based on each surface point and a pre-trained color network; a sixth generating unit, configured to generate each light reflection information based on each light reflection prediction information, pre-stored coordinate system transformation matrix information and pre-stored polarization matrix information, wherein each light reflection information in the each light reflection information includes specular reflection information, diffuse reflection component information and specular vector information; A seventh generating unit is configured to generate a camera pose group for each surface point based on each surface point, each sampling light direction information and each camera pose information; An eighth generating unit is configured to generate each perspective inconsistency point based on each surface point camera pose group; a ninth generating unit, configured to generate respective reflection confidence information based on the respective surface points; a tenth generating unit, configured to generate a three-dimensional model network based on the respective pixel color prediction information, the respective light reflection information, the respective perspective inconsistency points, the respective reflection confidence information and the respective polarization picture information; The storage unit is configured to store the three-dimensional model network in a data storage server.

10. An electronic device comprising: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 8.

11. A computer readable medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images