A method, apparatus, device and medium for reducing image shooting requirements

Abstract

The application provides a method, device, equipment and medium for reducing image shooting requirements, and relates to the field of computer vision. The method comprises: collecting multi-view images of a scene to be rendered; sampling on a camera ray corresponding to a given pixel point to obtain a sampling point coordinate and a view direction; calculating a hour angle h, an declination δ' and a solar elevation angle θ s , a solar azimuth angle Φ s , obtaining a global light direction vector, and encoding; encoding the shooting time of the image; inputting the sampling point coordinate, the view direction, the light encoding and the time encoding into a neural network to obtain the color and the volume density of the sampling point under the corresponding view; obtaining the final predicted pixel color by using volume rendering, calculating the color loss; and updating the neural network parameters by back propagation. The application can weaken or even eliminate the influence of light changes on outdoor shooting, and reduce the shooting requirements of neural radiance field images.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically to a method, apparatus, device, and medium for reducing the requirements for image capture. Background Technology

[0002] Neural Radiation Field (NeRF) is a groundbreaking 3D reconstruction method. Unlike traditional explicit reconstruction methods that use point clouds, meshes, or voxels, NeRF uses hidden functions of neural networks to model 3D scenes. Given a set of camera poses and captured images, the neural network is trained to fit the captured images at known angles. Volume rendering is used to obtain the color of pixels, and the hidden functions are used to model the 3D scene.

[0003] Compared with conventional modeling methods, neural radiation field modeling has the following advantages: 1. The modeling time of neural radiation field modeling is less dependent on the number of images; 2. Neural radiation field modeling has high realism. Compared with conventional modeling methods, neural radiation field modeling can realistically reproduce vegetation, semi-transparent objects, reflective objects, water surfaces, etc.; 3. Neural radiation field modeling does not depend on feature points in the image. Therefore, in local scenes where feature points are sparse, conventional modeling methods may miss some parts of the object, forming jagged edges.

[0004] However, traditional neural radiation fields require extremely high image capture standards, especially outdoors. Shooting must be done within a relatively short timeframe to ensure that the ambient light intensity and sun angle remain largely consistent, thus maintaining uniform shadow length and direction for objects in the scene from the same camera position. This requires a significant amount of manpower for taking the photographs. Summary of the Invention

[0005] In view of this, embodiments of this application provide a method, apparatus, device, and medium for reducing image capture requirements, so as to reduce or even eliminate the impact of lighting changes on outdoor shooting, thereby reducing the requirements for capturing images of neural radiation fields.

[0006] This application provides the following technical solution: a method for reducing image capture requirements, comprising:

[0007] The camera captures images of the scene to be rendered from multiple perspectives.

[0008] The camera pose parameters are obtained from the acquired image. The camera ray corresponding to the viewpoint of the given pixel is sampled to obtain the coordinates of the sampling point and the viewpoint direction. The camera ray is marked as r(t) = o + td, where o is the origin of the ray, d is the viewpoint direction, and t is the length of the camera ray.

[0009] According to the current shooting time and the GPS coordinates of the image, a hour angle h and a declination δ' are calculated;

[0010] A solar elevation angle θ and a solar azimuth angle Φ are calculated through the hour angle h and the declination δ'; s and the solar azimuth angle Φ s A global light direction vector is obtained through the solar elevation angle θ s and the solar azimuth angle Φ s , and the global light direction vector is encoded to obtain a light encoding;

[0011] The current shooting time of the image is encoded to obtain a time encoding;

[0012] The coordinates and the view direction of each sampling point, the light encoding and the time encoding are input into a neural network, and color and volume density of the sampling point under the corresponding view are output;

[0013] A predicted pixel color is obtained by accumulating the color and the volume density of each sampling point in a volume rendering manner, and a color loss is calculated according to the predicted pixel color and the real color of the image;

[0014] The color loss is input into the neural network, and the parameters of the neural network are updated through back propagation to obtain an optimized neural radiance field model.

[0015] According to an embodiment of the present application, the hour angle h is calculated according to the current shooting time and the GPS coordinates of the image, including:

[0016] The hour angle h is calculated by using the following formula:

[0017] h = 15 * (T + (Λ - 120°) / 15° - 12)

[0018] Wherein, T is the current shooting time, and Λ is the longitude of the GPS coordinates.

[0019] According to an embodiment of the present application, the declination δ' is calculated according to the current shooting time and the GPS coordinates of the image, including:

[0020] The accumulated day N is determined from the current shooting time, and the declination δ' is calculated from the accumulated day N by using the following formula:

[0021]

[0022] According to an embodiment of the present application, the solar elevation angle θ and the solar azimuth angle Φ are calculated through the hour angle h and the declination δ'; s and the solar azimuth angle Φ s , including:

[0023] Sun elevation angle θ s = arcsin (cosh cos δ' cos Φ + sin δ' sin Φ) ;

[0024] Sun azimuth angle

[0025] wherein Φ is the latitude of the GPS coordinate.

[0026] According to an embodiment of the present application, the global illumination direction vector is:

[0027] (-cos θ s sin φ s , cos θ s cos φ s , sin θ s )

[0028] wherein the center point of the reference system of the vector is (0, 0, 0) point, and the radius r is defined as 1.

[0029] According to an embodiment of the present application, the global illumination direction vector is encoded to obtain the illumination encoding, comprising:

[0030] The global illumination direction vector is encoded by using the third order spherical harmonic function to obtain the illumination encoding of the 27-dimensional vector matrix.

[0031] According to an embodiment of the present application, the color of each sampling point is accumulated to obtain the predicted pixel color, comprising:

[0032] The predicted pixel color is obtained by the following integral formula

[0033]

[0034]

[0035] wherein i = (1, 2, 3... N) represents the i-th sampling point, j = (1, 2, 3... (i-1)) represents the j-th sampling point, δ i and δ j respectively represent the distance between two sampling points, δ j = t j+1 -t j , δ i = t i+1 -t i , t i represents the distance from the camera to the i-th sampling point, t j represents the distance from the camera to the j-th sampling point, t i+1 represents the distance from the camera to the i+1-th sampling point, and t j+1denotes the distance of the camera to the j+1th sampling point, σ i denotes the volume density of the i th sampling point, c i denotes the color, T i denotes the volume density of the sampling point i remaining, σ j denotes the volume density of the j th sampling point.

[0036] Another aspect of the present application also provides a device for reducing image shooting requirements, comprising:

[0037] An image acquisition module is configured to acquire multi-view images of a scene to be rendered by a camera;

[0038] A sampling module is configured to obtain camera pose parameters according to the acquired images, sample on a camera ray corresponding to a given pixel point and a view angle to obtain coordinates and a view direction of a sampling point; wherein the camera ray is marked as r(t) = o + td, o is the origin of the ray, d is the view direction, and t is the length of the camera ray;

[0039] A processing module is configured to calculate a hour angle h and a declination δ' according to a current shooting time and GPS coordinates of the image;

[0040] The processing module is further configured to calculate a solar elevation angle θ s and a solar azimuth angle Φ s according to the hour angle h and the declination δ' and obtain a global light direction vector according to the solar elevation angle θ s and the solar azimuth angle Φ s , encode the global light direction vector to obtain a light encoding;

[0041] The processing module is further configured to encode the current shooting time of the image to obtain a time encoding;

[0042] The processing module is further configured to input the coordinates and the view direction of each sampling point, the light encoding and the time encoding into a neural network to output the color and the volume density of the sampling point under the corresponding view angle;

[0043] A loss calculation module is configured to accumulate the color of each sampling point to obtain a predicted pixel color in a volume rendering manner, and calculate a color loss according to the predicted pixel color and an actual color of the image;

[0044] A model updating module is configured to input the color loss into the neural network, update parameters of the neural network through back propagation, and obtain an optimized neural radiance field model.

[0045] Another aspect of the present application also provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method for reducing image shooting requirements when executing the computer program.

[0046] Another aspect of the present application also provides a computer readable storage medium, which stores a computer program for executing the method for reducing image shooting requirements.

[0047] Compared with the prior art, the above at least one technical solution adopted by the embodiments of the present application can achieve the beneficial effects at least including: the embodiments of the present application aim at the problem of high image shooting requirements of the existing neural radiance field image, and through encoding the shooting time and the solar elevation angle and inputting them into the neural radiance field network (i.e. the neural radiance field model) for training, the model (i.e. the neural radiance field network) learns the illumination information, which can weaken the influence caused by the problem of inconsistent illumination information in the neural radiance field network due to the change of image shooting time, weaken or even exclude the influence of illumination change on the neural radiance field network to reconstruct the scene, improve the quality of the neural radiance field network to reconstruct the scene, and thus is conducive to reducing the image shooting requirements. BRIEF DESCRIPTION OF DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0049] Figure 1 is a method flow diagram for reducing image shooting requirements according to an embodiment of the present application;

[0050] Figure 2 is a structural block diagram of a device for reducing image shooting requirements according to an embodiment of the present application;

[0051] Figure 3 is a structural block diagram of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0052] The embodiments of the present application will be described in detail below with reference to the drawings.

[0053] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the drawings and in combination with the embodiments, and the technical solutions of the present application will be clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0054] As shown in Figure 1 , the embodiment of the present application provides a method for reducing image shooting requirements, comprising:

[0055] S1. Multi-view image acquisition of a scene to be rendered by a camera;

[0056] S2. Obtaining camera pose parameters according to the acquired images, sampling on the camera ray corresponding to the given pixel point, obtaining the coordinates and view direction of the sampling points; wherein the camera ray is marked as r(t)=o+td, o is the origin of the ray, d is the view direction, and t is the length of the camera ray;

[0057] S3. Calculating the hour angle h and the declination δ' according to the current shooting time and GPS coordinates of the image;

[0058] S4. Calculating the solar elevation angle θ s and the solar azimuth angle Φ s by the hour angle h and the declination δ', and obtaining the global light direction vector by the solar elevation angle θ s and the solar azimuth angle Φ s , encoding the global light direction vector to obtain the light encoding;

[0059] S5. Converting the time minutes and seconds in the current shooting time of the image to p seconds, using the formula:

[0060] γ(p)=(sin(2 0 πp),cos(2 0 πp),…,sin(2 L-1 πp),cos(2 L-1 πp))

[0061] and encoding the p seconds to obtain the time encoding, wherein L is 8;

[0062] S6. Inputting the coordinates and view direction of each sampling point, the light encoding and the time encoding into a neural network, and outputting the color and volume density of the sampling point under the corresponding view angle;

[0063] S7. Cumulative color and body density of each sampling point are obtained by using volume rendering, and a final predicted color of a pixel is obtained, and a color loss is calculated according to the color predicted by each pixel and the color of the real image;

[0064] S8. The color loss is input into the neural network, and the parameters of the neural network are updated by back propagation, and an optimized neural radiation field model is obtained.

[0065] The embodiment of the present application encodes time and solar elevation angle, and inputs them into the neural network, so that the time information within a day can enable the model to learn short-time and short-range illumination changes, and the information of the solar elevation angle can enable the model to learn long-time and long-range illumination changes, thereby weakening or even eliminating the influence of illumination changes.

[0066] In the embodiment of the present application, illumination encoding is first calculated. According to the current shooting Beijing time T and GPS coordinates (longitude Λ, latitude Φ) obtained during shooting, the hour angle h is calculated, including:

[0067] The hour angle h is calculated by using the following formula:

[0068] First, the time difference ΔT is calculated from the change in longitude:

[0069] ΔT = (Λ-120°) / 15°

[0070] Then, the true solar time ST is calculated from the Beijing time T and the time difference ΔT:

[0071] ST = T + ΔT

[0072] And because the corresponding hour angle is 15 degrees per hour, the hour angle h can be calculated from the true solar time:

[0073] h = 15*(ST-12)

[0074] The above formulas are combined together, that is:

[0075] h = 15*(T+(Λ-120°) / 15°-12)

[0076] The accumulated day N is determined from the current shooting Beijing time T, that is, the day of the year, and because the shooting time of the image includes the month and day, the accumulated day N can be calculated from the month and day.

[0077] The solar declination angle is a seasonal change due to the tilt of the earth's rotation axis and the earth's revolution around the sun, and can be calculated from the angular velocity of the sun 360° / 365 and the accumulated day N by the following formula:

[0078]

[0079] In the embodiment of the present application, the solar elevation angle θ is calculated by the hour angle h and the declination δ' s and the solar azimuth angle Φ s , comprising:

[0080] The solar elevation angle θ s = arcsin (cosh cos δ' cos Φ + sin δ' sin Φ).

[0081] The solar azimuth angle Φ

[0082] Wherein, Φ is the latitude of the GPS coordinate; the solar elevation angle θ s is the angle between the incident direction of the sunlight and the ground plane; and the solar azimuth angle Φ s is the angle between the projection of the sunlight on the ground plane and the local meridian.

[0083] The global illumination direction vector is obtained by the solar elevation angle θ s and the solar azimuth angle Φ s , and the global illumination direction vector is:

[0084] (-cos θ s sin φ s , cos θ s cos φ s , sin θ s )

[0085] Wherein, the center point of the reference system of the vector is (0, 0, 0), and the radius r is defined as 1.

[0086] The global illumination direction vector is encoded by using the third-order spherical harmonic function, and the illumination encoding of the 3*9 = 27-dimensional vector matrix is obtained.

[0087] Next, the time encoding is calculated. The time, minute and second in the current shooting time of the image are converted into p seconds, and the formula is:

[0088] γ (p) = (sin (2 0 πp), cos (2 0 πp), …, sin (2 L-1 πp), cos (2 L-1 πp))

[0089] And the p seconds are encoded to obtain the time encoding, wherein L is the length of the encoding, and L is 8 in the embodiment.

[0090] After obtaining the illumination encoding and the time encoding, the coordinates and the perspective direction of each sampling point, and the illumination encoding and the time encoding are input into a neural network, and color and body density of the sampling point under the corresponding perspective are output; by using a volume rendering mode, color and body density of each sampling point are accumulated to obtain a final predicted pixel color;

[0091] wherein the integral formula of the final predicted pixel color is simplified as:

[0092]

[0093] wherein,

[0094] wherein, i=(1, 2, 3...N) represents the i-th sampling point, j=(1, 2, 3... (i-1)) represents the j-th sampling point, δ i , δ j respectively represent distances between two sampling points, δ j =t j+1 -t j , δ i =t i+1 -t i , t i represents a distance from the camera to the i-th sampling point, t j represents a distance from the camera to the j-th sampling point, t i+1 represents a distance from the camera to the i+1-th sampling point, t j+1 represents a distance from the camera to the j+1-th sampling point, σ i represents a body density of the i-th sampling point, c i represents color, T i represents a remaining body density of the sampling point i, σ j represents a body density of the j-th sampling point.

[0095] Color loss is calculated according to the color predicted by each pixel and the color of the real image:

[0096]

[0097] wherein, R is a set of light rays.

[0098] Finally, the neural network parameters are updated by back propagation, and the neural radiation field model is optimized.

[0099] As shown in FIG. 1, Figure 2 the present application also provides a device 100 for reducing image shooting requirements, comprising:

[0100] Image acquisition module 101 is used to acquire images of the scene to be rendered from multiple perspectives using a camera;

[0101] The sampling module 102 is used to obtain camera pose parameters based on the acquired image, and to sample on the camera ray corresponding to the viewpoint of a given pixel to obtain the coordinates of the sampling point and the viewpoint direction; wherein, the camera ray is marked as r(t)=o+td, o is the origin of the ray, d is the viewpoint direction, and t is the length of the camera ray.

[0102] The processing module 103 is used to calculate the hour angle h and declination δ' based on the current shooting time and GPS coordinates of the image;

[0103] The processing module 103 is further configured to calculate the solar altitude angle θ using the hour angle h and the declination δ′. s And solar azimuth Φ s Then, through the solar altitude angle θ s And solar azimuth Φ s Obtain the global illumination direction vector, encode the global illumination direction vector to obtain the illumination code;

[0104] The processing module 103 is further configured to encode the current shooting time of the image to obtain a time code;

[0105] The processing module 103 is further configured to input the coordinates and viewing direction of each sampling point, as well as the illumination encoding and the time encoding, into the neural network, and output the color and volume density of the sampling point under the corresponding viewing angle.

[0106] The loss calculation module 104 is used to accumulate the color of each sampling point using volume rendering to obtain the predicted pixel color, and calculate the color loss based on the predicted pixel color and the real color of the image.

[0107] The model update module 105 is used to input the color loss into the neural network and update the parameters of the neural network through backpropagation to obtain an optimized neural radiation field model.

[0108] In one embodiment, a computer device is provided, such as Figure 3 As shown, it includes a memory 201, a processor 202, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any of the above-described methods for reducing image capture requirements.

[0109] Specifically, the computer device can be a computer terminal, a server, or a similar computing device.

[0110] In the present embodiment, a computer-readable storage medium storing a computer program that executes any of the above-described methods of reducing image capturing requirements is provided.

[0111] In particular, computer-readable storage media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology for information storage. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer-readable storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device. According to the definition herein, computer-readable storage media does not include transitory media such as modulated data signals and carriers.

[0112] Obviously, those skilled in the art should understand that each module or step of the above-mentioned embodiments of the present application can be realized by a general computing device, which can be concentrated on a single computing device or distributed on a network composed of multiple computing devices, and can be realized by program codes executable by a computing device, so that they can be stored in a storage device and executed by a computing device, and in some cases, the steps shown or described can be executed in different order, or they can be manufactured into individual integrated circuit modules, or multiple modules or steps can be manufactured into a single integrated circuit module. Thus, the embodiments of the present application are not limited to any particular hardware and software combination.

[0113] The above only describes specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any changes or replacements within the technical scope disclosed by the present application can be easily thought of by those skilled in the art, which should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method of reducing image capture requirements, characterized by, The method comprises the following steps: Multi-view image acquisition of a scene to be rendered is performed by a camera; Camera pose parameters are obtained according to the acquired image, and a camera ray corresponding to a given pixel point is sampled to obtain the coordinates and the view direction of a sampling point; wherein the camera ray is marked as r(t) = o + td, o is the origin of the ray, d is the view direction, and t is the length of the camera ray; A hour angle h and a declination δ' are calculated according to the current shooting time and the GPS coordinates of the image; The solar elevation angle θ is calculated from the hour angle h and the declination δ' s and the solar azimuth angle Φ s The global illumination direction vector is obtained from the solar elevation angle θ s and the solar azimuth angle Φ s The illumination encoding is obtained by encoding the global illumination direction vector. The current shooting time of the image is encoded to obtain a time code; The coordinates and the view direction of each sampling point, the light code and the time code are input into a neural network to output the color and the volume density of the sampling point under the corresponding view; The color and the volume density of each sampling point are accumulated in a volume rendering mode to obtain a predicted pixel color, and a color loss is calculated according to the predicted pixel color and the real color of the image; The color loss is input into the neural network, and the parameters of the neural network are updated through back propagation to obtain an optimized neural radiance field model.

2. The method of claim 1, wherein, The hour angle h is calculated according to the current shooting time and the GPS coordinates of the image, comprising: The hour angle h is calculated by the following formula: h = 15 * (T + (Λ - 120°) / 15° - 12) Wherein, T is the current shooting time, and Λ is the longitude of the GPS coordinates.

3. The method of claim 1, wherein, The declination δ' is calculated according to the current shooting time and the GPS coordinates of the image, comprising: The accumulated day N is determined from the current shooting time, and the declination δ' is calculated from the accumulated day N by the following formula:

4. The method of claim 1, wherein, The solar elevation angle θ is calculated from the hour angle h and the declination δ' s and the solar azimuth angle Φ s comprising: Sun elevation angle θ s = arcsin (cosh cos δ' cos Φ + sin δ' sin Φ); solar azimuth Wherein, Φ is the latitude of the GPS coordinates.

5. The method of claim 1, wherein, The global light direction vector is: (-cos θ s sin φ s , cos θ s cos φ s , sin θ s ) Wherein, the center point of the reference system of the vector is the (0, 0, 0) point, and the radius r is defined as 1.

6. The method of claim 1, wherein, The global light direction vector is encoded to obtain a light code, comprising: The global light direction vector is encoded by a third-order spherical harmonic function to obtain a 27-dimensional vector matrix light code.

7. The method of claim 1, wherein, The color of each sampling point is accumulated to obtain a predicted pixel color, comprising: The predicted pixel color is obtained by the following integral formula where i = (1, 2, 3...N) represents the ith sample point, j = (1, 2, 3... (i-1)) represents the jth sample point, δ i , δ j represent the distance between two sample points, δ j = t j+1 -t j , δ i = t i+1 -t i , t i represents the distance from the camera to the ith sample point, t j represents the distance from the camera to the jth sample point, t i+1 represents the distance from the camera to the ith+1 sample point, t j+1 represents the distance from the camera to the jth+1 sample point, σ i represents the volume density of the ith sample point, c i represents the color, T i represents the volume density remaining for sample point i, σ j represents the volume density of the jth sample point.

8. An apparatus for reducing image capture requirements, the apparatus comprising: The method comprises the following steps: An image acquisition module is configured to perform multi-view image acquisition of a scene to be rendered by a camera; A sampling module is configured to obtain camera pose parameters according to the acquired image, and sample a camera ray corresponding to a given pixel point to obtain the coordinates and the view direction of a sampling point; wherein the camera ray is marked as r(t) = o + td, o is the origin of the ray, d is the view direction, and t is the length of the camera ray; A processing module is configured to calculate a hour angle h and a declination δ' according to the current shooting time and the GPS coordinates of the image; The processing module is further configured to calculate a solar elevation angle θ by using the hour angle h and the declination δ' s and a solar azimuth angle Φ s Further, a global illumination direction vector is obtained by using the solar elevation angle θ s and the solar azimuth angle Φ s The global illumination direction vector is encoded to obtain an illumination encoding. The processing module is further configured to encode the current shooting time of the image to obtain a time code; The processing module is further configured to input the coordinates and the view direction of each sampling point, the light code and the time code into a neural network to output the color and the volume density of the sampling point under the corresponding view; A loss calculation module is configured to accumulate the color of each sampling point in a volume rendering mode to obtain a predicted pixel color, and calculate a color loss according to the predicted pixel color and the real color of the image. A model updating module is configured to input the color loss into the neural network, update parameters of the neural network by back propagation, and obtain an optimized neural radiance field model.

9. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the method for reducing image shooting requirements in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program for implementing the method for reducing image shooting requirements in any one of claims 1 to 7.

Images