Image processing method and device and electronic equipment

Abstract

The embodiment of the invention discloses an image processing method and device and electronic equipment. The method comprises the following steps: acquiring a plurality of to-be-processed images acquired by a plurality of cameras; based on the plurality of to-be-processed images and the full-view neural light field, obtaining depth information and color information of pixel points in the plurality of to-be-processed images; obtaining an offline rendering image based on the depth information, the color information and preset virtual camera parameters; and sending the off-line rendered image to the head-mounted display device, so that the head-mounted display device performs three-dimensional reconstruction based on the off-line rendered image. According to the mode, the plurality of cameras are used, so that the plurality of to-be-processed images can represent the result of observing the target object at different visual angles, and the plurality of cameras are distributed on the movable object in a surrounding manner, so that the six-degree-of-freedom expression of the target object in the space can be recovered through off-line image rendering; therefore, the quality of three-dimensional reconstruction is improved, and the immersion of a user is improved.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. patent application No. 63 / 513,740, filed with the U.S. Patent Office on July 14, 2023, and entitled “SYSTEM AND METHOD FOR HYBRID 6DOF PHOTO / VIDEO CAPTURE ON MIXED REALITY DEVICES WITH MULTIPLE INPUTS,” the entire contents of which are incorporated by reference into this application. Technical Field

[0003] The present application relates to the field of computer technology, and more specifically, to an image processing method, device and electronic device. Background Art

[0004] With the development of science and technology, virtual reality, mixed reality, extended reality and other related technologies have begun to become research hotspots. In the application process of virtual reality, mixed reality, extended reality and other related technologies, three-dimensional reconstruction is particularly important. In related methods, images can be collected by a monocular camera or a binocular camera, and then three-dimensional reconstruction is performed based on the collected images. However, the three-dimensional reconstruction results of related methods still have the problem of poor user immersion. Summary of the invention

[0005] In view of the above problems, the present application proposes an image processing method, device and electronic device to improve the above problems.

[0006] In a first aspect, the present application provides an image processing method, which is applied to an electronic device, and the method includes: acquiring multiple images to be processed acquired by multiple cameras, wherein the multiple cameras are distributed around a movable object; based on the multiple images to be processed and a full-view neural light field, obtaining depth information and color information of pixel points in the multiple images to be processed; based on the depth information, the color information and preset virtual camera parameters, obtaining an offline rendered image, wherein the preset virtual camera parameters represent a user perspective corresponding to a head-mounted display device; and sending the offline rendered image to the head-mounted display device so that the head-mounted display device performs three-dimensional reconstruction based on the offline rendered image.

[0007] In a second aspect, the present application provides an image processing device that runs on an electronic device, the device comprising: a unit for acquiring an image to be processed, for acquiring a plurality of images to be processed captured by a plurality of cameras, the plurality of cameras being distributed around a movable object; a pixel information acquisition unit, for obtaining depth information and color information of pixels in the plurality of images to be processed based on the plurality of images to be processed and a full-view neural light field; a rendering image generation unit, for obtaining an offline rendered image based on the depth information, the color information and preset virtual camera parameters, the preset virtual camera parameters representing a user perspective corresponding to a head-mounted display device; an image sending unit, for sending the offline rendered image to the head-mounted display device, so that the head-mounted display device performs three-dimensional reconstruction based on the offline rendered image.

[0008] In a third aspect, the present application provides an electronic device comprising one or more processors and a memory; one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the above-mentioned method.

[0009] In a fourth aspect, the present application provides a computer-readable storage medium, in which program code is stored, wherein the above method is executed when the program code is run.

[0010] The present application provides an image processing method, device, electronic device and storage medium. After acquiring multiple images to be processed captured by multiple cameras distributed around a movable object, the method obtains depth information and color information of pixel points in the multiple images to be processed based on the multiple images to be processed and a full-view neural light field; obtains an offline rendered image based on the depth information, the color information and preset virtual camera parameters, and the preset virtual camera parameters represent the user perspective corresponding to a head-mounted display device; and sends the offline rendered image to the head-mounted display device so that the head-mounted display device performs three-dimensional reconstruction based on the offline rendered image. Through the above method, the depth information and color information of the pixels in the multiple images to be processed can be obtained based on the multiple images to be processed and the full-view neural light field, and then the offline rendered image can be obtained based on the depth information, color information and preset virtual camera parameters, and the offline rendered image is sent to the head display device for three-dimensional reconstruction. Among them, since multiple cameras are used, the multiple images to be processed can represent the observation results of the target object from different perspectives, and the multiple cameras are distributed around a movable object, so that the motion of the target object observed by the multiple cameras at any position and any posture in the space can be restored through the offline rendered image (in the three-dimensional coordinate system, the movement freedom along the three rectangular coordinate axes of x, y, and z and the rotation freedom around these three coordinate axes), that is, the six degrees of freedom (Degree of Freedom, DoF) information of the target object is obtained, thereby improving the quality of three-dimensional reconstruction and further improving the user's immersion. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0012] Figure 1 A flowchart of an image processing method proposed in an embodiment of the present application is shown;

[0013] Figure 2 A schematic diagram of a movable object proposed in the present application is shown;

[0014] Figure 3 A schematic diagram of a spherical surface proposed in the present application is shown;

[0015] Figure 4 A schematic diagram of a light ray passing through a double sphere model proposed in the present application is shown;

[0016] Figure 5 A schematic diagram of a multi-layer perceptron model proposed in this application is shown;

[0017] Figure 6 A flowchart of an image processing method proposed in another embodiment of the present application is shown;

[0018] Figure 7 A schematic diagram of a sliding time window proposed in the present application is shown;

[0019] Figure 8 A flowchart of an image processing method proposed in another embodiment of the present application is shown;

[0020] Fig. 9 A schematic diagram of a head-mounted display device proposed in the present application is shown;

[0021] Fig.10 A schematic diagram showing a viewing angle of a head-mounted display device proposed in the present application is shown;

[0022] Fig.11 A schematic diagram showing the overall business process of an image processing method proposed in this application;

[0023] Fig.12 A structural block diagram of an image processing device proposed in an embodiment of the present application is shown;

[0024] Fig.13 A structural block diagram of an electronic device proposed in this application is shown;

[0025] Fig.14 It is a storage unit of an embodiment of the present application for storing or carrying a program code for implementing the image processing method according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0027] With the continuous development of social productivity and science and technology, various industries have an increasing demand for virtual reality, mixed reality, extended reality and other related technologies, making related technologies a research hotspot. In the application process of virtual reality, mixed reality, extended reality and other related technologies, three-dimensional reconstruction is particularly important. In related methods, images can be collected by a monocular camera or a binocular camera, and then three-dimensional reconstruction can be performed based on the collected images.

[0028] The inventors found in the relevant research that the 3D reconstruction results of the relevant methods still have the problem of poor user immersion. For example, when the images collected by a monocular camera are used for 3D reconstruction, the quality of the reconstructed 3D image is not high due to the limited viewing angle of the monocular camera. For another example, when the images collected by a binocular camera are used for 3D reconstruction, although the images are collected from two different viewing angles, there may be parallax blur, which makes the depth information inaccurate, thereby affecting the quality of the reconstructed 3D image.

[0029] Therefore, the inventors proposed an image processing method, device and electronic device in the present application. After obtaining multiple images to be processed captured by multiple cameras distributed around a movable object, the depth information and color information of the pixel points in the multiple images to be processed are obtained based on the multiple images to be processed and the full-view neural light field; based on the depth information, the color information and preset virtual camera parameters, an offline rendered image is obtained, and the preset virtual camera parameters represent the user perspective corresponding to the head-mounted display device; the offline rendered image is sent to the head-mounted display device, so that the head-mounted display device performs three-dimensional reconstruction based on the offline rendered image. Through the above method, the depth information and color information of the pixels in the multiple images to be processed can be obtained based on the multiple images to be processed and the full-view neural light field, and then the offline rendered image can be obtained based on the depth information, color information and preset virtual camera parameters, and the offline rendered image is sent to the head display device for three-dimensional reconstruction. Among them, since multiple cameras are used, the multiple images to be processed can represent the observation results of the target object from different perspectives, and the multiple cameras are distributed around a movable object, so that the motion of the target object observed by the multiple cameras at any position and any posture in the space can be restored through the offline rendered image, that is, the six-degree-of-freedom information of the target object is obtained, thereby improving the quality of three-dimensional reconstruction and further improving the user's immersion.

[0030] See also Figure 1 , an image processing method provided by an embodiment of the present application is applied to an electronic device, the method comprising:

[0031] S110: Acquire a plurality of images to be processed that are captured by a plurality of cameras, where the plurality of cameras are distributed around a movable object.

[0032] The image to be processed may refer to an image used for three-dimensional reconstruction of a target object (such as a scene, a person, a single object, etc.). Figure 2As shown, the movable object can be a hemisphere, and the movable object 10 can be set on a bracket 12 with a pulley 11 at the bottom and adjustable height. Multiple cameras can be distributed on the hemisphere, and the total viewing angle range of the multiple cameras can be 360°, that is, the target object that needs to be three-dimensionally reconstructed can be observed at 360 degrees. Exemplarily, the height range of the movable object can be 1174mm to 2324mm, and there can be 40 cameras.

[0033] As a method, the movable object can send the multiple images to be processed to the electronic device after collecting the multiple images to be processed, so that the electronic device can obtain the multiple images to be processed.

[0034] Among them, electronic devices can refer to devices with image processing capabilities, such as computers, tablets, mobile phones, etc.

[0035] Optionally, at the same acquisition moment, each camera on the movable object may acquire an image, thereby obtaining a plurality of images to be processed at the acquisition moment.

[0036] Optionally, in some implementations, the electronic device and the movable object may be the same device, that is, the movable object may not only capture the image to be processed, but also process the captured image to be processed.

[0037] S120: Based on the multiple images to be processed and the full-view neural light field, obtain depth information and color information of pixel points in the multiple images to be processed.

[0038] Among them, the neural light field (Neural Light Field, NeuLF) can be used to describe the light in the scene corresponding to the image to be processed. Each light ray can be described by a directional ray. Since the total viewing angle of multiple cameras in this application can be 360°, the full-view neural light field in this application can be called NeuLF-360.

[0039] The full-view neural light field may include an image pre-processing module, a multilayer perceptron (MLP) model, and an image post-processing module. The image pre-processing module may be used to convert multiple input images to be processed into light parameters, which may be used to describe the direction of the light. The image pre-processing module may include a double sphere model, in which the two spheres in the double sphere model are spheres in different coordinate systems, and each sphere may be Figure 3The grid shape shown in the figure, each grid can correspond to a pixel point, and the two spherical surfaces can be obtained by combining the pixel surface, the lens pupil surface and the main lens surface in pairs. The multi-layer perceptron model can be used to obtain the color information and depth information of the pixel points in the image to be processed based on the light parameters. The image post-processing module can be used to obtain an offline rendered image based on the color information and depth information, and the offline rendered image can be used to reconstruct the three-dimensional structure of the target object.

[0040] In the embodiment of the present application, since the target object can be observed 360°, the entire pixel surface formed by multiple images to be processed, the lens pupil surface and the main lens surface corresponding to multiple cameras can be spherical surfaces, thereby obtaining a double-spherical model. Compared with the prior art that can only observe the target object forward in a limited area to obtain a binocular image or a monocular image, the present application can observe the target object 360°, so that more detailed information of the target object can be observed. Therefore, the double-spherical model can obtain light parameters under different viewing angles, and thus a higher quality offline rendered image can be obtained.

[0041] As a method, light parameters can be obtained based on camera parameters of multiple cameras, multiple images to be processed and image pre-processing modules. The light parameters can characterize the direction of light. Based on the light parameters and a multi-layer perceptron model, the depth information and color information of the pixels in the image to be processed can be obtained.

[0042] Among them, camera parameters can include intrinsic parameters and extrinsic parameters. The intrinsic parameters (Camera Intrinsics) can refer to the 3*3 parameter matrix that is fixed since leaving the factory. The parameter matrix can include focal length (fx, fy), principal point coordinates (cx, cy), and distortion parameters. The extrinsic parameters can be used to describe the movement of the camera in a static scene. The extrinsic parameters can refer to the rotation matrix and translation vector of the camera (the position and posture of the camera can be described by its rotation matrix R and translation vector t).

[0043] Optionally, the position information of multiple cameras can be determined based on the camera parameters of the multiple cameras; the direction of the light can be obtained based on the position information of the multiple cameras and the position information of pixels in the multiple images to be processed; the intersection position information can be obtained based on the direction of the light and the double-sphere model, and the intersection position information can be used as the light parameter, and the intersection position information can refer to the position information of the intersection of the light and the two spheres in the double-sphere model.

[0044] For example, the top / bottom view of the double sphere model can be Figure 4As shown, each ray may correspond to a pixel point, and the starting position information of the ray may be the same as the position information of the camera that collects the pixel point. The direction of the ray may be obtained through the position information of the pixel point and the corresponding camera, and then the intersection position information may be obtained based on the direction of the ray and the double sphere model, and the intersection position information may be used as the ray parameter. For example, the intersection position information may be (ui, vi) of the first sphere and (si, ti) of the second sphere, and the ray parameter may be (ui, vi, si, ti).

[0045] Optional, such as Figure 5 As shown, the multi-layer perceptron model can include two branch structures, one of which can be used to output the color information of the pixel points, and the other branch structure can be used to output the depth information of the pixel points.

[0046] S130: Obtain an offline rendered image based on the depth information, the color information, and preset virtual camera parameters, where the preset virtual camera parameters represent a user perspective corresponding to a head mounted display device.

[0047] Among them, the preset virtual camera parameters can be used to determine the position information of the virtual camera, and the position information can represent a viewpoint, that is, the user perspective corresponding to the head display device, so as to obtain an offline rendered image under the viewpoint.

[0048] As a method, an offline rendered image can be obtained based on depth information, color information, preset virtual camera parameters and an image post-processing module.

[0049] Optionally, the image post-processing module may obtain color information of each pixel in the offline rendered image through image rendering technology and color information.

[0050] Optionally, the image rendering technology may be the rendering technology in NeuLF, the volume rendering technology (Volume Rendering), etc. Among them, the rendering technology in NeuLF can realize direct rendering after one inference, which can save time and improve the real-time performance of 3D reconstruction, so that when the user interacts with the 3D reconstruction result, it can respond faster, thereby improving the user experience; while the volume rendering technology needs to be rendered in the form of multiple integrations, so that more image details can be obtained, and the quality of offline rendered images can be improved.

[0051] S140: Send the offline rendered image to a head mounted display device, so that the head mounted display device performs three-dimensional reconstruction based on the offline rendered image.

[0052] Among them, the head-mounted display device can refer to MR (Mixed Reality) head-mounted display device, XR (Extended Reality) head-mounted display device, VR (Virtual Reality) head-mounted display device, etc.

[0053] As a method, the offline rendered image can be sent to the head-mounted display device, so that the head-mounted display device performs three-dimensional reconstruction based on the offline rendered image.

[0054] Optionally, you can get the 3D reconstruction result through OpenGL.

[0055] Optionally, when the real-time requirement is not high, steps S110 to S130 can be executed in advance, and when the head display device sends a request to obtain an offline rendered image, the pre-processed offline rendered image can be sent to the head display device, so that there is sufficient time to observe the target object from multiple perspectives and perform more sophisticated processing on the processed image, so as to provide the user with better visual quality.

[0056] An image processing method provided by the present embodiment obtains a plurality of images to be processed captured by a plurality of cameras distributed around a movable object, and then obtains depth information and color information of pixel points in the plurality of images to be processed based on the plurality of images to be processed and a full-view neural light field; obtains an offline rendered image based on the depth information, the color information and preset virtual camera parameters, wherein the preset virtual camera parameters represent the user perspective corresponding to a head-mounted display device; and sends the offline rendered image to the head-mounted display device, so that the head-mounted display device performs three-dimensional reconstruction based on the offline rendered image. Through the above method, the depth information and color information of the pixels in the multiple images to be processed can be obtained based on the multiple images to be processed and the full-view neural light field, and then the offline rendered image can be obtained based on the depth information, color information and preset virtual camera parameters, and the offline rendered image is sent to the head display device for three-dimensional reconstruction. Among them, since multiple cameras are used, the multiple images to be processed can represent the observation results of the target object from different perspectives, and the multiple cameras are distributed around a movable object, so that the motion of the target object observed by the multiple cameras at any position and any posture in the space can be restored through the offline rendered image, that is, the six-degree-of-freedom information of the target object is obtained, thereby improving the quality of three-dimensional reconstruction and further improving the user's immersion.

[0057] See also Figure 6 , an image processing method provided by an embodiment of the present application is applied to an electronic device, the method comprising:

[0058] S210: Acquire images to be processed respectively captured by a plurality of cameras in a continuous time to obtain the plurality of images to be processed.

[0059] The continuous time may include multiple acquisition moments, and at the same acquisition moment, each camera on the movable object may acquire an image to be processed.

[0060] As a method, the images to be processed captured by multiple cameras in a continuous time can be obtained to obtain multiple images to be processed and the capture time of each image to be processed.

[0061] S220: Divide the plurality of images to be processed into a plurality of groups of images to be processed, each group of images to be processed including an image to be processed captured by the plurality of cameras at the same capture time.

[0062] As one approach, multiple images to be processed are divided into multiple groups of images to be processed based on the acquisition time of each image to be processed.

[0063] S230: Based on the camera parameters of the multiple cameras, the multiple groups of images to be processed and the image pre-processing module, obtain light parameters corresponding to each of the multiple groups of images to be processed.

[0064] As a method, light parameters corresponding to each of the multiple groups of images to be processed can be obtained based on camera parameters of multiple cameras, multiple groups of images to be processed and an image pre-processing module.

[0065] S240: Sort the light parameters corresponding to the plurality of groups of images to be processed in order of acquisition time from earliest to latest, to obtain a plurality of groups of light parameters to be input.

[0066] As a method, the light parameters corresponding to the multiple groups of images to be processed can be sorted in order from the earliest to the latest acquisition time to obtain multiple groups of light parameters to be input.

[0067] S250: Based on the sliding time window, the multiple groups of light parameters to be input and the multi-layer perceptron model, obtain depth information and color information corresponding to the multiple groups of light parameters to be input.

[0068] As a way, Figure 7 As shown, the light parameters to be input to the multi-layer perceptron model each time can be controlled by a sliding time window, so as to obtain multiple sets of depth information and color information corresponding to the light parameters to be input. The depth information and color information corresponding to the light parameters to be input can be understood as the depth information and color information of the pixel points in the image to be processed corresponding to each set of light parameters.

[0069] S260: Obtain an offline rendered image based on the depth information, the color information, and preset virtual camera parameters, where the preset virtual camera parameters represent a user perspective corresponding to the head-mounted display device.

[0070] As a method, multiple sets of offline rendering images corresponding to the input light rays may be obtained based on the depth information and color information corresponding to the multiple sets of input light rays, so as to obtain offline rendering images corresponding to different acquisition moments.

[0071] In the embodiment of the present application, due to the superior image processing capability of the electronic device, it is possible to perform complex rendering tasks, improve the quality of visual output, and reduce the computational burden of the head-mounted display device itself.

[0072] S270: Send the offline rendered image to a head mounted display device, so that the head mounted display device performs three-dimensional reconstruction based on the offline rendered image.

[0073] As a method, offline rendered images corresponding to different acquisition times can be sent to a head-mounted display device, so that the head-mounted display device performs three-dimensional reconstruction based on the offline rendered images corresponding to different acquisition times, and displays the three-dimensional reconstruction results in sequence based on the acquisition time, thereby generating a video.

[0074] The present embodiment provides an image processing method, through the above-mentioned method, based on multiple images to be processed and a full-view neural light field, the depth information and color information of the pixel points in the multiple images to be processed can be obtained, and then based on the depth information, color information and preset virtual camera parameters, an offline rendered image is obtained, and the offline rendered image is sent to the head display device for three-dimensional reconstruction, wherein, due to the use of multiple cameras, the multiple images to be processed can represent the observation results of the target object from different perspectives, and the multiple cameras are distributed around a movable object, so that the motion of the target object observed by the multiple cameras at any position and any posture in space can be restored through the offline rendered image, that is, the six-degree-of-freedom information of the target object is obtained, thereby improving the quality of three-dimensional reconstruction and further improving the user's immersion. In addition, in the present embodiment, by introducing a sliding time window to indicate the time dimension, the offline rendered images at different acquisition times can be obtained based on multiple groups of light parameters to be input in a sequence from the first to the last acquisition time, which can improve the sensitivity of the reconstruction process to the time and the dynamic performance of the target object, and the six-degree-of-freedom video can be generated based on the offline rendered images at different acquisition times.

[0075] See also Figure 8 , an image processing method provided in an embodiment of the present application is applied to a head-mounted display device, the method comprising:

[0076] S310: Obtain an offline rendered image, wherein the offline rendered image is obtained by the electronic device based on preset virtual camera parameters, depth information and color information of pixel points in multiple images to be processed, wherein the depth information and the color information are obtained based on the multiple images to be processed and a full-view neural light field, and the multiple images to be processed are collected by multiple cameras distributed around a movable object.

[0077] Among them, Fig. 9 As shown, the head display device may be provided with multiple cameras, which may include four color cameras and one depth camera, wherein the four color cameras are evenly distributed on both sides of the head display device, and the depth camera is distributed in the middle of the head display device. The viewing angle of the head display device may be as follows: Fig.10 shown.

[0078] As a method, after generating an offline rendered image, the electronic device can send the offline rendered image to a head-mounted display device, so that the head-mounted display device can obtain the offline rendered image.

[0079] In the embodiment of the present application, since the head-mounted display device itself is provided with multiple cameras, the head-mounted display device can also capture images, thereby obtaining depth information and color information of pixel points in the images captured by the multiple cameras; based on the depth information, color information and preset virtual camera parameters of the pixel points in the images captured by the multiple cameras, an online rendered image is obtained; based on the depth information, color information and preset virtual camera parameters of the pixel points in the images captured by the multiple cameras, a reference image is obtained; based on the reference image and a hole-filling algorithm, an online rendered image is obtained.

[0080] Optionally, the color information collected by the color camera and the depth information collected by the depth camera can be synthesized, and then through post-processing such as sharpening and smoothing, the depth information and color information of the pixel points in the images collected by multiple cameras can be obtained.

[0081] Optionally, the information missing areas in the online rendered image can be filled by a hole filling algorithm, so as to obtain a more complete online rendered image, thereby improving the quality of 3D reconstruction and enhancing the immersive experience.

[0082] S320: Perform three-dimensional reconstruction based on the offline rendered image.

[0083] As a way, 3D reconstruction can be performed based on offline rendered images and opengl.

[0084] As another way, 3D reconstruction is performed based on offline rendered images, online rendered images and OpenGL.

[0085] As another method, 3D reconstruction can be performed based on online rendered images and OpenGL. Since images can be collected in real time through multiple cameras of the head-mounted display device, and the collected images can be directly processed online through the head-mounted display device, the real-time nature of data processing and the rapid response to user needs can be improved, thereby improving the user's sense of immersion. In addition, 3D reconstruction can be performed even when the computer device is unavailable or the user has a wide range of movement, thereby reducing the limitations of 3D reconstruction.

[0086] The present embodiment provides an image processing method, through the above-mentioned method, since the offline rendered image is obtained to be generated by the electronic device, and the electronic device obtains the depth information and color information of the pixel points in the multiple images to be processed based on the multiple images to be processed and the full-view neural light field, and then obtains the offline rendered image based on the depth information, color information and preset virtual camera parameters, and sends the offline rendered image to the head display device for three-dimensional reconstruction, wherein, since multiple cameras are used, the multiple images to be processed can represent the observation results of the target object from different perspectives, and the multiple cameras are distributed around a movable object, so that the motion of the target object observed by the multiple cameras at any position and any posture in the space can be restored through the offline rendered image, that is, the six-degree-of-freedom information of the target object is obtained, thereby improving the quality of the three-dimensional reconstruction, and then improving the user's immersion. In addition, in this embodiment, three-dimensional reconstruction based on offline rendered images and online rendered images can improve the quality of the three-dimensional reconstruction results, thereby improving the user's immersion.

[0087] In order to better understand the solution of the embodiment of the present application, the overall business process of the image processing method is introduced below.

[0088] See also Fig.11 In this application, three-dimensional reconstruction can be performed by combining an electronic device with a head-mounted display device or by only using a head-mounted display device. In the case where the electronic device is combined with the head-mounted display device, the electronic device can first obtain multiple images to be processed based on step S1, and then execute step S2: obtain color information and depth information of pixel points in multiple images to be processed based on multiple images to be processed and the full-view neural light field, and then execute step S3: obtain an offline rendered image based on the color information, depth information and preset virtual camera parameters, and send the offline rendered image to the head-mounted display device, so that the head-mounted display device can obtain the offline rendered image based on step S4, and then execute step S5: perform three-dimensional reconstruction based on the offline rendered image, and display the three-dimensional reconstruction result to the user.

[0089] When only the head-mounted display device is used, the head-mounted display device can obtain the online color information and online depth information of the pixel points in the image captured by the head-mounted display device based on step S6, and then execute step S7: obtain the online rendered image based on the online color information, the online depth information and the preset virtual camera parameters, and then execute step S8: perform three-dimensional reconstruction based on the online rendered image.

[0090] Optionally, in a manner where the electronic device is combined with a head-mounted display device, step S9 may also be performed: performing three-dimensional reconstruction based on the offline rendered image and the online image.

[0091] See also Fig.12 The present application provides an image processing device 600, which runs on an electronic device. The device 600 includes:

[0092] The to-be-processed image acquisition unit 610 is used to acquire a plurality of to-be-processed images captured by a plurality of cameras, where the plurality of cameras are distributed around a movable object;

[0093] A pixel information acquisition unit 620, configured to obtain depth information and color information of pixels in the plurality of images to be processed based on the plurality of images to be processed and the full-view neural light field;

[0094] A rendering image generating unit 630, configured to obtain an offline rendering image based on the depth information, the color information and preset virtual camera parameters, wherein the preset virtual camera parameters represent a user perspective corresponding to a head mounted display device;

[0095] The image sending unit 640 is used to send the offline rendered image to a head mounted display device, so that the head mounted display device performs three-dimensional reconstruction based on the offline rendered image.

[0096] As a method, the full-view neural light field includes an image pre-processing module and a multi-layer perceptron model, and the pixel information acquisition unit 620 is specifically used to obtain light parameters based on the camera parameters of the multiple cameras, the multiple images to be processed and the image pre-processing module, and the light parameters characterize the direction of the light; based on the light parameters and the multi-layer perceptron model, the depth information and the color information are obtained.

[0097] Optionally, the image pre-processing module includes a double-sphere model, the two spheres in the double-sphere model are spheres in different coordinate systems, and the pixel information acquisition unit 620 is specifically used to determine the position information of the multiple cameras based on the camera parameters of the multiple cameras; obtain the direction of the light based on the position information of the multiple cameras and the position information of the pixels in the multiple images to be processed; obtain the intersection position information based on the direction of the light and the double-sphere model, and use the intersection position information as the light parameter, and the intersection position information may refer to the position information of the intersection of the light and the two spheres in the double-sphere model.

[0098] As a method, the image to be processed acquisition unit 610 is specifically used to acquire the images to be processed captured by multiple cameras respectively in a continuous time to obtain the multiple images to be processed; divide the multiple images to be processed into multiple groups of images to be processed, each group of images to be processed includes an image to be processed captured by the multiple cameras at the same acquisition time.

[0099] As a method, the pixel information acquisition unit 620 is specifically used to obtain the light parameters corresponding to each of the multiple groups of images to be processed based on the camera parameters of the multiple cameras, the multiple groups of images to be processed and the image pre-processing module.

[0100] Optionally, the pixel information acquisition unit 620 is specifically used to sort the light parameters corresponding to the multiple groups of images to be processed in order of acquisition time from first to last, so as to obtain multiple groups of light parameters to be input; based on the sliding time window, the multiple groups of light parameters to be input and the multi-layer perceptron model, the depth information and color information corresponding to the multiple groups of light parameters to be input are obtained.

[0101] As a method, the full-view neural light field includes an image post-processing module, and the rendering image generation unit 630 is specifically used to obtain the offline rendered image based on the depth information, the color information, preset virtual camera parameters and the image post-processing module.

[0102] The following will be combined Fig.13 An electronic device provided by the present application is described.

[0103] See also Fig.13 Based on the above-mentioned image processing method and device, the embodiment of the present application also provides another electronic device 100 that can execute the above-mentioned image processing method. The electronic device 100 includes a processor 102, a memory 104 and a communication module 106, wherein the memory 104 stores a program that can execute the content of the above-mentioned embodiment, and the processor 102 can execute the program stored in the memory 104.

[0104] Among them, the processor 102 may include one or more processing cores. The processor 102 uses various interfaces and lines to connect various parts within the entire electronic device 100, and executes various functions and processes data of the electronic device 100 by running or executing instructions, programs, code sets or instruction sets stored in the memory 104, and calling data stored in the memory 104. Optionally, the processor 102 can be implemented in at least one hardware form of a network processor (Neural network Processing Unit, NPU), a digital signal processing (Digital Signal Processing, DSP), a field programmable gate array (Field-Programmable Gate Array, FPGA), and a programmable logic array (Programmable Logic Array, PLA). The processor 102 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processor (Graphics Processing Unit, GPU), a network processor (Neural network Processing Unit, NPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing display content; the NPU is responsible for processing multimedia data such as video and images; and the modem is used to process wireless communications. It is understandable that the above-mentioned modem may not be integrated into the processor 102, but may be implemented separately through a communication chip.

[0105] The memory 104 may include a random access memory (RAM), and may also include a read-only memory (RAM) and a double data rate synchronous dynamic random access memory (DDR). The memory 104 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 104 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data (such as a phone book, audio and video data, chat record data) created by the electronic device 100 during use.

[0106] The communication module 106 may be used to implement information exchange between the electronic device 100 and other devices, for example, to transmit device control instructions, operation request instructions, and status information acquisition instructions, etc. When the other devices are different devices, the corresponding communication modules 106 may be different.

[0107] Please refer to Fig.14 , which shows a structural block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable storage medium 1000 stores program codes, which can be called by a processor to execute the method described in the above method embodiment.

[0108] The computer readable storage medium 1000 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer readable storage medium 1000 includes a non-transitory computer-readable storage medium. The computer readable storage medium 1000 has storage space for program code 1010 that performs any method step in the above method. These program codes can be read from or written to one or more computer program products. The program code 1010 can be compressed, for example, in an appropriate form.

[0109] In summary, the present application provides an image processing method, device, and electronic device. After acquiring multiple images to be processed captured by multiple cameras distributed around a movable object, the method, device, and electronic device obtain depth information and color information of pixel points in the multiple images to be processed based on the multiple images to be processed and the full-view neural light field; obtain an offline rendered image based on the depth information, the color information, and preset virtual camera parameters, and the preset virtual camera parameters represent the user perspective corresponding to the head-mounted display device; and send the offline rendered image to the head-mounted display device so that the head-mounted display device performs three-dimensional reconstruction based on the offline rendered image. Through the above method, the depth information and color information of the pixels in the multiple images to be processed can be obtained based on the multiple images to be processed and the full-view neural light field, and then the offline rendered image can be obtained based on the depth information, color information and preset virtual camera parameters, and the offline rendered image is sent to the head display device for three-dimensional reconstruction. Among them, since multiple cameras are used, the multiple images to be processed can represent the observation results of the target object from different perspectives, and the multiple cameras are distributed around a movable object, so that the motion of the target object observed by the multiple cameras at any position and any posture in the space can be restored through the offline rendered image, that is, the six-degree-of-freedom information of the target object is obtained, thereby improving the quality of three-dimensional reconstruction and further improving the user's immersion.

[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An image processing method, characterized in that: Applied to electronic equipment, the method comprises: Acquire a plurality of images to be processed captured by a plurality of cameras, wherein the plurality of cameras are distributed around a movable object; Based on the multiple images to be processed and the full-view neural light field, obtaining depth information and color information of pixel points in the multiple images to be processed; Obtaining an offline rendered image based on the depth information, the color information, and preset virtual camera parameters, wherein the preset virtual camera parameters represent a user perspective corresponding to a head-mounted display device; The offline rendered image is sent to the head mounted display device, so that the head mounted display device performs three-dimensional reconstruction based on the offline rendered image.

2. The method according to claim 1, characterized in that The full-view neural light field includes an image pre-processing module and a multi-layer perceptron model. The depth information and color information of the pixel points in the multiple images to be processed are obtained based on the multiple images to be processed and the full-view neural light field, including: Based on the camera parameters of the multiple cameras, the multiple images to be processed and the image pre-processing module, a light parameter is obtained, where the light parameter represents the direction of the light; The depth information and the color information are obtained based on the light parameters and the multi-layer perceptron model.

3. The method according to claim 2, characterized in that The image pre-processing module includes a double sphere model, wherein the two spheres in the double sphere model are spheres in different coordinate systems, and the light parameters are obtained based on the camera parameters of the multiple cameras, the multiple images to be processed and the image pre-processing module, including: Determining position information of the multiple cameras based on camera parameters of the multiple cameras; Based on the position information of the multiple cameras and the position information of the pixels in the multiple images to be processed, obtaining the direction of the light; Based on the direction of the light and the double sphere model, the intersection position information is obtained and used as the light parameter. The intersection position information may refer to the position information of the intersection between the light and two spheres in the double sphere model.

4. The method according to claim 2, characterized in that: The step of acquiring a plurality of images to be processed acquired by a plurality of cameras comprises: Acquire images to be processed respectively captured by a plurality of cameras in a continuous time to obtain the plurality of images to be processed; The multiple images to be processed are divided into multiple groups of images to be processed, each group of images to be processed includes one image to be processed captured by the multiple cameras at the same capture time.

5. The method according to claim 4, characterized in that The obtaining of light parameters based on the camera parameters of the multiple cameras, the multiple images to be processed and the image pre-processing module, wherein the light parameters represent the direction of the light, includes: Based on the camera parameters of the multiple cameras, the multiple groups of images to be processed and the image pre-processing module, light parameters corresponding to each of the multiple groups of images to be processed are obtained.

6. The method according to claim 5, characterized in that The obtaining the depth information and the color information based on the light parameters and the multi-layer perceptron model includes: Sorting the light parameters corresponding to the plurality of groups of images to be processed in order of acquisition time from earliest to latest, to obtain a plurality of groups of light parameters to be input; Based on the sliding time window, the multiple groups of light parameters to be input and the multi-layer perceptron model, the depth information and color information corresponding to the multiple groups of light parameters to be input are obtained.

7. The method according to claim 1, characterized in that The full-view neural light field includes an image post-processing module, which obtains an offline rendered image based on the depth information, the color information and preset virtual camera parameters, wherein the preset virtual camera parameters represent the user perspective corresponding to the head display device, including: The offline rendered image is obtained based on the depth information, the color information, preset virtual camera parameters and the image post-processing module.

8. An image processing device, characterized in that: Running on an electronic device, the device comprises: A unit for acquiring images to be processed, used for acquiring a plurality of images to be processed captured by a plurality of cameras, wherein the plurality of cameras are distributed around a movable object; A pixel point information acquisition unit, used to obtain depth information and color information of pixel points in the multiple images to be processed based on the multiple images to be processed and the full-view neural light field; A rendering image generating unit, configured to obtain an offline rendering image based on the depth information, the color information and preset virtual camera parameters, wherein the preset virtual camera parameters represent a user viewing angle corresponding to a head mounted display device; The image sending unit is used to send the offline rendered image to a head-mounted display device, so that the head-mounted display device performs three-dimensional reconstruction based on the offline rendered image.

9. An electronic device, characterized in that: comprising one or more processors and memory; One or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program code, wherein when the program code is run, the method according to any one of claims 1 to 7 is executed.

Images