Image processing method and device, chip, electronic equipment and storage medium
The camera parameters are calculated through the NeRF model and image data, and the rendered images are automatically generated, which solves the problem of low efficiency in scene subject generation in the prior art, and improves the diversity and accuracy of the training data of the computer vision model.
Patent Information
- Application Number
- CN202310791117.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-29
- Publication Date
- 2024-12-31
AI Technical Summary
In the prior art, training computer vision models requires collecting and labeling images in various scenarios, resulting in low efficiency in generating scene subjects and inability to support the diversity of different application scenarios.
The camera parameters are calculated using NeRF model and image data, and the scene data from the expanded viewing angle is expanded by interpolation processing, and a large number of rendered images are automatically generated to reduce the need for scene construction and labeling.
It improves image rendering generation efficiency, reduces complexity, supports the diversity requirements of different application scenarios, and enhances the diversity and accuracy of training data of computer vision models.
Smart Images
Figure CN119228903A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of image processing, and in particular to an image processing method, device, chip, electronic device and storage medium. Background Art
[0002] In related technologies, in order to train computer vision models, it is necessary to collect and annotate images under various scene configurations and attributes to obtain scene subjects. In the process of collecting and annotating images, it is necessary to build different scenes, place scene subjects (such as dolls, cups, furniture, etc.) in different scenes, and collect scene images containing scene subjects. Then, manually adjust the attribute parameters of different scenes to perform manual annotation to generate scene subjects, and the efficiency of scene subject generation is low. Summary of the invention
[0003] The present disclosure provides an image processing method, device, chip, electronic device and storage medium to solve the problems in the related art and improve the generation efficiency of rendered images.
[0004] A first aspect of the present disclosure provides an image processing method, the method comprising:
[0005] Based on a Neural Radiance Fields (NeRF) model and image data, calculating camera parameters corresponding to when the image data is generated using the NeRF model; the camera parameters include camera extrinsic parameters;
[0006] Performing interpolation processing on the camera extrinsic parameters to obtain target camera parameters;
[0007] The NeRF model and the target camera parameters are used to perform NeRF rendering on the images of the scene subject at multiple viewing angles to obtain a target rendered image.
[0008] In the above scheme, the NeRF model and the target camera parameters are used to perform NeRF rendering on the images of the scene subject at multiple perspectives to obtain the target rendered image, including:
[0009] Acquire subject images of the scene subject at different viewing angles;
[0010] The target rendered image is generated based on the target camera parameters and each subject image using the NeRF model.
[0011] In the above solution, the use of the NeRF model to generate the target rendered image based on the target camera parameters and each subject image includes:
[0012] Combining different subject images and preset backgrounds to obtain at least one scene combination information;
[0013] The target rendered image is generated by utilizing the NeRF model based on the target camera parameters and each scene combination information.
[0014] In the above scheme, the method further includes:
[0015] Acquire parameter information of the preset background from the background parameter information existing in the NeRF model;
[0016] Alternatively, in response to a configuration operation on background parameters, parameter information of the preset background is generated.
[0017] In the above scheme, the method further includes:
[0018] Using the output information of the computer vision model, updating the parameters of the multilayer perceptron (MLP) of the NeRF model to obtain an updated NeRF model;
[0019] The target rendered image is updated using the updated NeRF model.
[0020] In the above solution, the interpolation process is performed on the camera external parameters to obtain the target camera parameters, including:
[0021] Perform bilinear interpolation processing or grid interpolation processing on the camera extrinsic parameters to obtain the target camera parameters.
[0022] In the above scheme, the method further includes:
[0023] Using the first device, acquiring scene images of the scene subject at multiple angles in a preset scene;
[0024] Get the image data corresponding to each scene image.
[0025] A second aspect of the present disclosure provides an image processing device, the device comprising:
[0026] A first processing unit, configured to calculate, based on a NeRF model and image data, camera parameters corresponding to when the image data is generated by using the NeRF model; the camera parameters include camera extrinsic parameters;
[0027] A second processing unit is used to perform interpolation processing on the camera extrinsic parameters to obtain target camera parameters;
[0028] The third processing unit is used to perform NeRF rendering on the images of the scene subject at multiple viewing angles by using the NeRF model and the target camera parameters to obtain a target rendered image.
[0029] The third aspect embodiment of the present disclosure proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in the first aspect embodiment of the present disclosure.
[0030] The fourth aspect embodiment of the present disclosure proposes a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the method described in the first aspect embodiment of the present disclosure.
[0031] A fifth aspect embodiment of the present disclosure provides a computer program product, including a computer program, which executes the method described in the first aspect embodiment of the present disclosure when a processor is used to execute the computer program.
[0032] The sixth aspect embodiment of the present disclosure proposes a chip, which includes one or more interface circuits and one or more processors; the interface circuit is used to receive signals from a memory of an electronic device and send signals to the processor, the signals including computer instructions stored in the memory, and when the processor executes the computer instructions, the electronic device executes the method described in the first aspect embodiment of the present disclosure.
[0033] In summary, the image processing method, device, chip, electronic device and storage medium proposed in the present invention, by using the Raw data of the image acquisition device to infer the camera parameters at different angles, and interpolating the camera external parameters to expand the scene data at different perspectives, can generate a large amount of scene data of the virtual rendering environment for the NeRF model, so that a large number of rendered images can be automatically generated using the NeRF model. Since there is no need to build a large number of scenes, and there is no need to collect and annotate images of the scene subjects in different scenes multiple times, the complexity of the image rendering process is reduced and the efficiency of rendering image generation is greatly improved.
[0034] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.
[0036] Figure 1 A schematic diagram of an application scenario provided by an embodiment of the present disclosure;
[0037] Figure 2 A flowchart of an image processing method provided by an embodiment of the present disclosure;
[0038] Figure 3 A flowchart of an image processing method provided by an embodiment of the present disclosure;
[0039] Figure 4 A schematic diagram of a flow chart of a sample image generation method provided in an application embodiment of the present disclosure;
[0040] Figure 5 A schematic diagram of the structure of an image processing device provided by an embodiment of the present disclosure;
[0041] Figure 6 A schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure;
[0042] Figure 7 A schematic diagram of the structure of a chip provided in an embodiment of the present disclosure. DETAILED DESCRIPTION
[0043] Embodiments of the present disclosure are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present disclosure, and should not be construed as limiting the present disclosure.
[0044] In order to improve the accuracy of the output results of the computer vision model, it is often necessary to use a large number of scene subjects to train the computer vision model. In the process of generating scene subjects, it is necessary to collect and annotate sample images under various scene configurations and attributes. This process is not only time-consuming, but also difficult to ensure that the distribution of the captured scene subjects can be well mapped to the target domain of the application scenario.
[0045] In order to solve the above problems, the scene subject can be generated by synthetic data, that is, different scene subjects are combined with different scenes, and public data sets are constructed by collecting images of scene subjects in different scenes, and sample data is obtained from the public data sets. However, in the process of synthesizing data, it is often necessary to build different scenes and manually adjust the properties of each scene through manual annotation. Since a large amount of random data needs to be rendered to train computer vision models, the generation process of the scene subject is highly complex and the generation efficiency is low. At the same time, since only public data sets can be used and the data type is single, it is impossible to support the usage requirements of different application scenarios.
[0046] The NeRF model is a neural network-based renderer that can generate high-quality scene views using virtual rendering scenes. Different virtual scenes can be constructed by controlling rendering parameters in the NeRF model, such as lighting, material, albedo, appearance, etc. Compared with traditional renderers, the NeRF model has the advantages of differential rendering and the ability to construct virtual rendering scenes. Therefore, it is widely used in various graphics and visual task processing.
[0047] On this basis, in order to solve the problems existing in the related art, the present disclosure proposes an image processing method, device, chip, electronic device and storage medium, based on the NeRF model and image data, calculates the camera parameters corresponding to the image data generated by the NeRF model; the camera parameters include camera external parameters; the camera external parameters are interpolated to obtain target camera parameters; the NeRF model and the target camera parameters are used to perform NeRF rendering on the images of the scene subject at multiple perspectives to obtain target rendered images. The solution provided by the present disclosure, by using the Raw data of the image acquisition device to infer the camera parameters at different angles, and interpolating the camera external parameters to expand the scene data at different perspectives, can generate a large amount of scene data of the virtual rendering environment for the NeRF model, so that a large number of rendered images can be automatically generated using the NeRF model. Since there is no need to build a large number of scenes, and there is no need to collect and annotate the images of the scene subject in different scenes multiple times, the complexity of the image rendering process is reduced, and the efficiency of rendering image generation is greatly improved.
[0048] Before introducing the detailed solution of the present disclosure, the scenario to which the solution of the present disclosure is applied is described first. Figure 1 FIG. 1 is an application scenario diagram of an image processing method in an embodiment. Figure 1 As shown, the application scenario includes an electronic device 104, in which a camera module and several applications can be installed. The application can initiate an image acquisition instruction to acquire an image, and the camera module acquires the image 102. Among them, the camera module can include a front camera module and / or a rear camera module. Finally, the target image is sent to the target application. Among them, the electronic device 104 can be a smart phone, a tablet computer, a personal digital assistant, a wearable device, etc.
[0049] In some optional embodiments, the electronic device may also be a vehicle-mounted device or an Internet of Vehicles device, such as a smart car, etc. In the present disclosure, only a smart phone is taken as an example, but it does not limit the scope of the present disclosure.
[0050] A camera can be installed on an electronic device, and images can be acquired through the installed camera. Cameras can be divided into laser cameras, visible light cameras and other types according to the different images acquired. Laser cameras can acquire images formed by irradiating lasers onto objects, and visible light cameras can acquire images formed by irradiating visible light onto objects. Several cameras can be installed on an electronic device, and the installation locations are not limited. For example, a camera can be installed on the front panel of an electronic device, and two cameras can be installed on the back panel. The camera can also be installed inside the electronic device in an embedded manner, and then the camera can be opened by rotating or sliding. Specifically, a front camera and a rear camera can be installed on an electronic device. The front camera and the rear camera can acquire images from different perspectives. Generally, the front camera can acquire images from the front perspective of the electronic device, and the rear camera can acquire images from the back perspective of the electronic device.
[0051] It should be understood that in the present disclosure, the front camera or the rear camera is only used as an example to distinguish the shooting angles of different cameras, rather than to limit the functions of multiple cameras. The multiple cameras in the present disclosure can be rear cameras or front cameras at the same time, which is not limited in the present disclosure.
[0052] Several applications can be installed in electronic devices. Applications refer to software written for a certain application purpose in electronic devices. Electronic devices can provide services for users through applications. When an application needs to capture an image, it will initiate an image capture instruction. The electronic device will call the camera module according to the image capture instruction to capture the image. Image capture instructions refer to instructions used to trigger image capture operations.
[0053] The electronic device is also provided with a processor, and the image processing module in the processor can perform correction processing on the image captured by the camera module, for example, execute the image processing method provided in the present disclosure.
[0054] Figure 2 An image processing method is provided in an embodiment of the present disclosure. Figure 2 As shown, the method may include:
[0055] Step 201: Based on a NeRF model and image data, calculate camera parameters corresponding to when the image data is generated by using the NeRF model; the camera parameters include camera extrinsic parameters.
[0056] In some embodiments, the image data may be image data in a raw format, specifically, the image data may be image data in a Bayer raw format; in actual application, the image data may be used as the output of the NeRF model, and then the camera extrinsic parameters input to the NeRF model when the output of the NeRF model is the image data are deduced; here, since different image data correspond to different viewing angles, that is, the corresponding acquisition angles of the scene subject are different, the camera extrinsic parameters input to the NeRF model at different angles can be deduced through the image data corresponding to the scene subject at different angles.
[0057] In some embodiments, the camera extrinsic parameters may be calculated by camera calibration and image matching.
[0058] Step 202: performing interpolation processing on the camera extrinsic parameters to obtain target camera parameters.
[0059] In some embodiments, the camera extrinsic parameters may be interpolated by bilinear interpolation, grid interpolation, or the like.
[0060] In practical applications, since the camera extrinsic parameters include the relative spatial position of the scene subject and the camera, more scene data from different perspectives can be obtained by interpolating the camera extrinsic parameters to expand the camera extrinsic parameters.
[0061] Step 203: using the NeRF model and the target camera parameters, NeRF rendering is performed on the images of the scene subject at multiple viewing angles to obtain a target rendered image.
[0062] In some embodiments, the NeEF model may be used to perform NeRF rendering on images of the scene subject at multiple perspectives based on the target camera parameters and preset camera internal parameters to obtain a target rendered image.
[0063] In some embodiments, a large amount of scene data can be obtained through the camera intrinsic parameters and the expanded camera extrinsic parameters, so that a large amount of rendered images can be generated, that is, a large amount of rendered images can be synthesized to provide a large amount of rendered image resources for different image application scenarios, for example, a large amount of sample images can be provided for training computer vision models.
[0064] In summary, the image processing method proposed in the present invention utilizes the Raw data of the image acquisition device to infer the camera parameters at different angles, and interpolates the camera external parameters to expand the scene data at different perspectives. It can generate a large amount of scene data of the virtual rendering environment for the NeRF model, so that a large number of rendered images can be automatically generated using the NeRF model. Since there is no need to build a large number of scenes and there is no need to collect and annotate images of the scene subjects in different scenes multiple times, the complexity of the image rendering process is reduced and the efficiency of rendering image generation is greatly improved.
[0065] Figure 3 An image processing method is provided in an embodiment of the present disclosure. Figure 3 As shown, the method may include:
[0066] Step 301: Based on a NeRF model and image data, calculate camera parameters corresponding to when the image data is generated by using the NeRF model; the camera parameters include camera extrinsic parameters.
[0067] In some embodiments, the image data may be image data in a raw format, specifically, the image data may be image data in a Bayer raw format; in actual application, the image data may be used as the output of the NeRF model, and then the camera extrinsic parameters input to the NeRF model when the output of the NeRF model is the image data may be deduced; here, since different image data correspond to different viewing angles, that is, the corresponding acquisition angles of the scene subject are different, the camera extrinsic parameters input to the NeRF model at different angles may be deduced through the image data corresponding to the scene subject at different angles, that is, the camera extrinsic parameters at different angles; the NeRF model may also be referred to as a NeRF rendering engine.
[0068] In some embodiments, the camera extrinsic parameters may be calculated by using camera calibration and image matching.
[0069] In some embodiments, the image data may include Bayer Raw format data corresponding to the scene image of the subject.
[0070] Based on this, in some embodiments, the method may further include:
[0071] Using the first device, acquiring scene images of the scene subject at multiple angles in a preset scene;
[0072] Get the image data corresponding to each scene image.
[0073] In some embodiments, multiple scene subjects (for example, dolls, water cups and other objects of different materials) can be selected first, and after setting corresponding exposure, diagonal and other parameters, the first device can be used to collect scene images of the scene subjects at different angles; scene images and corresponding Bayer Raw format data can also be obtained from existing public data sets, that is, scene images and corresponding Bayer Raw format data can be obtained from constructed scene image data sets and Bayer Raw format data levels; since different subjects and backgrounds can be combined according to actual application requirements, sample generation requirements for different application scenarios can be supported; the scene subject can also be called a subject, which is not limited in the embodiments of the present disclosure.
[0074] In some embodiments, the first device may be a device capable of generating image data, specifically, a device capable of generating Bayer Raw format data. The first device may be a mobile phone camera or a SLR camera. The embodiments of the present disclosure are not limited to this as long as the function can be realized.
[0075] In actual applications, when using a mobile phone camera to capture scene images, since the imaging process pipeline in the mobile phone often only provides end-to-end results, it is necessary to first enable the root permission of the mobile phone so that the intermediate processing results in the pipeline, including Raw format data, can be freely obtained; at the same time, the dump interface of Bayer Raw format data can also be opened to obtain the intermediate processing results of the pipeline through the dump interface.
[0076] Step 302: interpolating the camera extrinsic parameters to obtain target camera parameters.
[0077] In some embodiments, the target camera parameters include the interpolated camera extrinsic parameters.
[0078] In some embodiments, the camera extrinsic parameters may be interpolated by bilinear interpolation, grid interpolation, or the like.
[0079] In practical applications, since the camera extrinsic parameters include the relative spatial position of the scene subject and the camera, the camera extrinsic parameters are expanded by interpolating the camera extrinsic parameters, and the expanded target camera parameters are input into the NeRF model to obtain scene data from more perspectives.
[0080] Step 303: Obtain subject images of the scene subject at different viewing angles.
[0081] In actual application, the subject image can be extracted from the scene image obtained in step 301; specifically, the semantic subject (i.e., the scene subject) can be segmented and identified by pixel set according to the semantic identifier of the scene subject, so as to obtain the scene subject in the scene image; wherein the semantic identifier can be the semantic information of the scene subject, such as people, snacks, buildings, etc.
[0082] It should be noted that step 303 may be performed before executing steps 301 to 302, or after executing steps 301 to 302, or simultaneously with executing steps 301 to 302, which is not limited in the present embodiment.
[0083] Step 304: Generate a target rendered image using the NeRF model based on the target camera parameters and each subject image.
[0084] In some embodiments, the camera intrinsic parameters may be preset, and the NeEF model may be used to perform NeRF rendering on the images of the scene subject at multiple perspectives based on the target camera parameters and the preset camera intrinsic parameters to obtain a target rendered image.
[0085] In some embodiments, target camera parameters at different angles, camera intrinsic parameters and the subject image may be input into a NeRF model, and the NeRF model renders the subject image at the corresponding angle using the intrinsic parameters and the target camera parameters to generate the target rendered image.
[0086] In some embodiments, the acquired subject image and different backgrounds can be combined to generate rendered images with different perspectives, backgrounds, and subjects, providing a large number of rendered image resources for different image application scenarios. For example, a large number of sample images can be provided for training computer vision models, thereby increasing the diversity of the computer vision model training set and improving the accuracy of the model output results.
[0087] Based on this, in some embodiments, the generating the target rendered image by using the NeRF model based on the target camera parameters and each subject image may include:
[0088] Combining different subject images and preset backgrounds to obtain at least one scene combination information;
[0089] The target rendered image is generated by utilizing the NeRF model based on the target camera parameters and each scene combination information.
[0090] In some embodiments, the preset background may be a template of a background scene that already exists in the rendering engine (ie, the NeRF model), or may be a scene customized according to application requirements, such as a low-light scene, a high-contrast scene, and the like.
[0091] Based on this, in some embodiments, the method may further include:
[0092] Acquire parameter information of the preset background from the background parameter information existing in the NeRF model;
[0093] Alternatively, in response to a configuration operation on background parameters, parameter information of the preset background is generated.
[0094] In some embodiments, the existing background parameter information in the NeRF model can be directly called, and new background parameter information can be customized for specific application scenarios and input into the NeRF model so that the NeRF model can combine it with different scene subjects to generate a large amount of rendering data.
[0095] In some embodiments, the generated target rendered image can be used to train a computer vision model; in order to improve the accuracy of the output results of the NeRF model, the data of the computer vision model can be migrated to downstream tasks, such as semantic segmentation, key point detection and other tasks, and combined with the execution results of the downstream tasks, the generation parameters of the NeRF model can be jointly optimized.
[0096] Based on this, in some embodiments, the method may further include:
[0097] Using the output information of the computer vision model, updating the MLP parameters of the NeRF model to obtain an updated NeRF model;
[0098] The target rendered image is updated using the updated NeRF model.
[0099] In some embodiments, the target rendered image generated by the NeRF model can be used in downstream task processing, and the MLP parameters of the NeRF model can be adjusted according to the downstream task processing results to improve the accuracy of the NeRF model output results, that is, the accuracy of the generated target rendering results.
[0100] In some embodiments, when optimizing the MLP parameters in the NeRF model, the parameters of the downstream task processing model may be optimized simultaneously to improve the accuracy of the output results of the NeRF model.
[0101] In some embodiments, by adjusting different MLP parameters in the NeRF model, different image quality defects in the output results of the NeRF model can be corrected, thereby improving the accuracy of the output results, that is, the image quality of the target rendered image, thereby achieving the generation of a large number of high-quality rendered images. For example, by fine-tuning the fitting parameters of the NeRF model, the NeRF model can better fit more data, thereby solving the fitting problem that occurs under high-frequency sampling of the image.
[0102] In summary, the image processing method proposed in the present invention utilizes the Raw data of the image acquisition device to infer the camera parameters at different angles, and interpolates the camera external parameters to expand the scene data at different perspectives. It can generate a large amount of scene data of the virtual rendering environment for the NeRF model, so that a large number of rendered images can be automatically generated using the NeRF model. Since there is no need to build a large number of scenes and there is no need to collect and annotate images of the scene subjects in different scenes multiple times, the complexity of the image rendering process is reduced and the efficiency of rendering image generation is greatly improved.
[0103] The technical solution of the present disclosure is further described in detail below in conjunction with specific application examples.
[0104] The present disclosure provides a sample image generation method in an application embodiment; for ease of understanding, Figure 4 A schematic diagram of a flow chart of a sample image generation method provided by an application embodiment of the present disclosure is shown; the method comprises:
[0105] Step 1: Build the scene, then proceed to step 2.
[0106] Specifically, subjects with different surface materials are selected, such as dolls, water cups, etc., and the subjects are placed in fixed positions.
[0107] Step 2: Store the raw data, and then proceed to step 3.
[0108] Specifically, first enable the root permission of the mobile phone, open the raw format data dump interface, set the corresponding exposure, focus and other parameters, collect data and save it in the mobile phone storage space; after completion, obtain the data collected by the mobile phone through the computer that communicates with the mobile phone.
[0109] When collecting data, a mobile phone is used to shoot the scene where the subject is located at different angles to obtain scene images; the scene images are collected at 5 to 10 angles, and a scene image is collected at each angle.
[0110] Step 3: Use the NeRF rendering engine for reverse rendering to infer the internal and external parameters of the camera, and then proceed to step 4.
[0111] Specifically, the NeRF rendering engine and the collected raw data are used to render the scene. That is, the collected raw data is used as the output data of the NeRF rendering engine, and the internal and external parameters of the camera at different angles are calculated by image matching and camera calibration.
[0112] Step 4: Render to get more scene data, then proceed to step 5.
[0113] Specifically, bilinear interpolation method, grid interpolation method and other methods are used to interpolate the camera extrinsic parameters, expand the camera extrinsic parameters, and input the expanded camera extrinsic parameters into the NeRF rendering engine to obtain a large amount of scene data.
[0114] Step 5: Send the preset camera intrinsic parameters, the expanded camera extrinsic parameters and the corresponding scene image to the scene NeRF rendering engine to generate rendering data, and then execute step 6.
[0115] Here, the 3D game rendering engine Unity can be used as a rendering engine, and the scene data obtained in step 4 can be used for large-scale rendering.
[0116] Among them, based on the scene data and semantic identification of the subject obtained in steps 1 to 4, different targets, that is, different subjects, can be extracted from the scene image; then, the extracted subject is placed in different background scenes to perform three-dimensional point cloud rendering calculation; wherein, the background scene in which the subject is placed is a template of a background scene that already exists in the rendering engine, or is a new background scene customized according to a specific application scenario, such as a low-light scene, a high-contrast scene, etc.; on this basis, the camera intrinsic parameters and camera extrinsic parameters of the virtual camera in the rendering engine can be used to further render the image and obtain a large number of different types of sample data.
[0117] The NeRF rendering engine is an implicit scene representation. Compared with point clouds, meshes, and voxels, which allow you to directly see the 3D model, you cannot see the 3D model in NeRF rendering. The mapping formula corresponding to the NeRF rendering engine is:
[0118] (σ,c)=gθ(x,d);
[0119] Among them, the input of the mapping is x and d, x∈R3, representing the coordinates of the three-dimensional space point, d∈S2, representing the observation angle; the output of the mapping is σ and c, σ∈R+, representing opacity, and c∈R3, representing the RGB color value.
[0120] Opacity represents the differential probability that a light ray ends at an infinitesimal particle at position x. The color C(r) of a camera ray r(t)=o+td with near boundary tn and far boundary tf can be expressed as:
[0121] C(r)=∫tntfT(t)σ(r(t))c(r(t),d)dt;
[0122] Among them, T(t) represents the cumulative transmittance along the light from tn to t, that is, the probability that the light propagates from tn to t without encountering any other particles (still alive particles); T(t) can be expressed by the following formula:
[0123] T(t)=exp(-∫tntσ(r(s))ds);
[0124] Therefore, to synthesize a complete picture from the rendering of the NeRF rendering engine, it is necessary to calculate the integral C(r) for the light passing through each pixel of the virtual camera to obtain the color value of the pixel; in the application embodiment of the present disclosure, a layered sampling method is used to numerically estimate the integral.
[0125] Step 6: Jointly optimize the generation parameters of the NeRF rendering engine based on the downstream tasks, and then execute step 7.
[0126] Specifically, the generated rendering data is migrated to downstream tasks, such as semantic segmentation tasks, key point detection tasks, etc., and then the MLP parameters of the NeRF rendering engine are adjusted and optimized according to the execution results of the downstream tasks.
[0127] Step 7: Iterate over the data.
[0128] Specifically, return to step 1 to generate sample data using the optimized NeRF rendering engine.
[0129] Here, the optimized NeRF rendering engine can correct local image problems that are easily captured by the human eye, such as high-frequency details, lighting changes, etc. By optimizing the iterative process, the accuracy of the NeRF rendering engine's output results can be improved, and the image quality of the rendering results can be improved.
[0130] The training process of the NeRF rendering engine is further explained below.
[0131] When training the NeRF rendering engine, the input data is pictures of the same scene taken at different locations, the camera poses used to take these pictures, the camera intrinsic parameters, and the scope of the scene; if the image dataset lacks the true values of the camera parameters, the SFM (full name of English is Structure From Motion) reconstruction solution COLMAP is used to estimate the required parameters as the true values of the camera parameters.
[0132] In the process of training NeRF to render new images, each position is first input into MLP to generate opacity and RGB color values, and then different positions are taken and these values are synthesized into a complete image using volume rendering technology; because the volume rendering function is differentiable, the NeRF scene representation can be trained and optimized by minimizing the difference between the synthetic image and the real image rendered in the previous step; in this way, when a NeRF training is completed, a model represented by the weights of MLP can be obtained. Among them, a model only contains the information of the scene and does not have the ability to generate pictures of other scenes.
[0133] In addition, NeRF can also be optimized in two aspects:
[0134] (1) Position encoding: Position encoding is similar to Fourier transform, which maps low-dimensional input to high-dimensional space to improve the network's ability to capture high-frequency information;
[0135] (2) Hierarchical sampling for volume rendering: A more efficient sampling strategy is used to reduce the computational overhead of the estimated integral and speed up training.
[0136] In summary, the solution provided by the application embodiment of the present disclosure can automatically generate a large amount of sample data according to application requirements, and at the same time, it does not require manual labeling, thereby improving the accuracy of the generated sample data.
[0137] Figure 5 FIG. 5 is a schematic diagram of the structure of an image processing device 500 provided in an embodiment of the present disclosure. Figure 5 As shown, the image processing device comprises:
[0138] The first processing unit 501 is used to calculate, based on the NeRF model and the image data, camera parameters corresponding to when the image data is generated by using the NeRF model; the camera parameters include camera extrinsic parameters;
[0139] A second processing unit 502 is used to perform interpolation processing on the camera extrinsic parameters to obtain target camera parameters;
[0140] The third processing unit 503 is used to perform NeRF rendering on the images of the scene subject at multiple viewing angles by using the NeRF model and the target camera parameters to obtain a target rendered image.
[0141] In some embodiments, the third processing unit 503 is specifically configured to:
[0142] Acquire subject images of the scene subject at different viewing angles;
[0143] The target rendered image is generated based on the target camera parameters and each subject image using the NeRF model.
[0144] In some embodiments, the third processing unit 503 is specifically configured to:
[0145] Combining different subject images and preset backgrounds to obtain at least one scene combination information;
[0146] The target rendered image is generated by utilizing the NeRF model based on the target camera parameters and each scene combination information.
[0147] In some embodiments, the third processing unit 503 may also be used to:
[0148] Acquire parameter information of the preset background from the background parameter information existing in the NeRF model;
[0149] Alternatively, in response to a configuration operation on background parameters, parameter information of the preset background is generated.
[0150] In some embodiments, the apparatus may further include a fourth processing unit, wherein the fourth processing unit is configured to:
[0151] Using the output information of the computer vision model, updating the MLP parameters of the NeRF model to obtain an updated NeRF model;
[0152] The target rendered image is updated using the updated NeRF model.
[0153] In some embodiments, the second processing unit 502 may be specifically configured to:
[0154] Perform bilinear interpolation processing or grid interpolation processing on the camera extrinsic parameters to obtain the target camera parameters.
[0155] In some embodiments, the first processing unit 501 may also be used to:
[0156] Using the first device, acquiring scene images of the scene subject at multiple angles in a preset scene;
[0157] Get the image data corresponding to each scene image.
[0158] Since the device provided in the embodiment of the present disclosure corresponds to the methods provided in the above-mentioned embodiments, the implementation of the method is also applicable to the device provided in the embodiment and will not be described in detail in this embodiment.
[0159] In the embodiments provided in the present application, the methods and devices provided in the embodiments of the present application are introduced. In order to implement the functions in the methods provided in the embodiments of the present application, the electronic device may include a hardware structure and a software module, and implement the functions in the form of a hardware structure, a software module, or a hardware structure plus a software module. A function of the functions may be executed in the form of a hardware structure, a software module, or a hardware structure plus a software module.
[0160] Figure 6 is a block diagram of an electronic device 600 for implementing the above-mentioned image processing method according to an exemplary embodiment. For example, the electronic device 600 may be a mobile phone, a computer, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.
[0161] Reference Figure 6 , the electronic device 600 may include one or more of the following components: a processing component 602 , a memory 604 , a power component 606 , a multimedia component 608 , an audio component 610 , an input / output (I / O) interface 612 , a sensor component 614 , and a communication component 616 .
[0162] The processing component 602 generally controls the overall operation of the electronic device 600, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 602 may include one or more processors 620 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 602 may include one or more modules to facilitate the interaction between the processing component 602 and other components. For example, the processing component 602 may include a multimedia module to facilitate the interaction between the multimedia component 608 and the processing component 602.
[0163] The memory 604 is configured to store various types of data to support operations on the electronic device 600. Examples of such data include instructions for any application or method operating on the electronic device 600, contact data, phone book data, messages, pictures, videos, etc. The memory 604 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.
[0164] The power supply component 606 provides power to the various components of the electronic device 600. The power supply component 606 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 600.
[0165] The multimedia component 608 includes a screen that provides an output interface between the electronic device 600 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 608 includes a front camera and / or a rear camera. When the electronic device 600 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.
[0166] The audio component 610 is configured to output and / or input audio signals. For example, the audio component 610 includes a microphone (MIC), and when the electronic device 600 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 604 or sent via the communication component 616. In some embodiments, the audio component 610 also includes a speaker for outputting audio signals.
[0167] I / O interface 612 provides an interface between processing component 602 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.
[0168] The sensor assembly 614 includes one or more sensors for providing various aspects of status assessment for the electronic device 600. For example, the sensor assembly 614 can detect the open / closed state of the electronic device 600, the relative positioning of components, such as the display and keypad of the electronic device 600, and the sensor assembly 614 can also detect the position change of the electronic device 600 or a component of the electronic device 600, the presence or absence of user contact with the electronic device 600, the orientation or acceleration / deceleration of the electronic device 600, and the temperature change of the electronic device 600. The sensor assembly 614 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 614 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 614 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.
[0169] The communication component 616 is configured to facilitate wired or wireless communication between the electronic device 600 and other devices. The electronic device 600 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, 4G LTE, 5G NR (NewRadio) or a combination thereof. In an exemplary embodiment, the communication component 616 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 616 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.
[0170] In an exemplary embodiment, the electronic device 600 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.
[0171] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 604 including instructions, and the above instructions can be executed by the processor 620 of the electronic device 600 to complete the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.
[0172] An embodiment of the present disclosure further proposes a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the image processing method described in the above embodiment of the present disclosure.
[0173] An embodiment of the present disclosure further provides a computer program product, including a computer program, which executes the image processing method described in the above embodiment of the present disclosure when a processor is used.
[0174] The embodiment of the present disclosure also provides a chip, which can be seen in Figure 7 Schematic diagram of the chip structure shown. Figure 7 The chip shown includes a processor 701 and an interface circuit 702. The number of the processor 701 can be one or more, and the number of the interface circuit 702 can be one or more.
[0175] Optionally, the chip also includes a memory 703, which is used to store necessary computer programs and data; the interface circuit 702 is used to receive signals from the memory 703 and send signals to the processor 701, and the signals include computer instructions stored in the memory 703. When the processor 701 executes the computer instructions, the electronic device executes the image processing method described in the above embodiments of the present disclosure.
[0176] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.
[0177] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.
[0178] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.
[0179] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processing module, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (control method), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary, and then stored in a computer memory.
[0180] It should be understood that the various parts of the embodiments of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0181] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.
[0182] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium. The above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.
[0183] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.
Claims
1. An image processing method, characterized in that: The method comprises: Based on the neural radiation field NeRF model and the image data, calculating the camera parameters corresponding to when the image data is generated by using the NeRF model; the camera parameters include camera external parameters; Performing interpolation processing on the camera extrinsic parameters to obtain target camera parameters; The NeRF model and the target camera parameters are used to perform NeRF rendering on the images of the scene subject at multiple viewing angles to obtain a target rendered image.
2. The method according to claim 1, characterized in that The NeRF model and the target camera parameters are used to perform NeRF rendering on the images of the scene subject at multiple viewing angles to obtain a target rendered image, including: Acquire subject images of the scene subject at different viewing angles; The target rendered image is generated based on the target camera parameters and each subject image using the NeRF model.
3. The method according to claim 2, characterized in that The step of generating the target rendered image by using the NeRF model based on the target camera parameters and each subject image includes: Combining different subject images and preset backgrounds to obtain at least one scene combination information; The target rendered image is generated by utilizing the NeRF model based on the target camera parameters and each scene combination information.
4. The method according to claim 3, characterized in that The method further comprises: Acquire parameter information of the preset background from the background parameter information existing in the NeRF model; Alternatively, in response to a configuration operation on background parameters, parameter information of the preset background is generated.
5. The method according to any one of claims 1 to 4, characterized in that The method further comprises: Using the output information of the computer vision model, updating the multi-layer perceptron (MLP) parameters of the NeRF model to obtain an updated NeRF model; The target rendered image is updated using the updated NeRF model.
6. The method according to any one of claims 1 to 4, characterized in that The interpolation process is performed on the camera external parameters to obtain target camera parameters, including: Perform bilinear interpolation processing or grid interpolation processing on the camera extrinsic parameters to obtain the target camera parameters.
7. The method according to any one of claims 1 to 4, characterized in that The method further comprises: Using the first device, acquiring scene images of the scene subject at multiple angles in a preset scene; Get the image data corresponding to each scene image.
8. An image processing device, characterized in that: The device comprises: A first processing unit, configured to calculate, based on a NeRF model and image data, camera parameters corresponding to when the image data is generated by using the NeRF model; the camera parameters include camera extrinsic parameters; A second processing unit is used to perform interpolation processing on the camera extrinsic parameters to obtain target camera parameters; The third processing unit is used to perform NeRF rendering on the images of the scene subject at multiple viewing angles by using the NeRF model and the target camera parameters to obtain a target rendered image.
9. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.
10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 7.
11. A chip, characterized in that: It comprises one or more interface circuits and one or more processors; the interface circuit is used to receive a signal from a memory of an electronic device and send the signal to the processor, the signal includes a computer instruction stored in the memory, and when the processor executes the computer instruction, the electronic device executes the method described in any one of claims 1 to 7.