Image processing method, electronic equipment, chip system and readable storage medium

Abstract

The embodiment of the invention provides an image processing method, electronic equipment, a chip system and a readable storage medium, and the method can comprise the steps: determining camera internal and external parameter information and scene point cloud information according to a first image and a second image which are different in shooting visual angle in a target scene; determining first attribute information of a scene Gaussian ball based on internal and external parameters of the camera and the scene point cloud information; determining a target depth image of a scene Gaussian ball according to the first attribute information, the reference shooting view angle and the reference optical center position; obtaining second attribute information of a scene Gaussian ball based on the target depth image and the color truth value information; based on the reference shooting view angle and the second attribute information, generating a target image of which the shooting view angle is the reference shooting view angle in the target scene; the first depth image comprises a depth value from an intersection point of an observation ray and a scene Gaussian sphere to a reference optical center position; the end point position of the observation ray is the reference optical center position. Therefore, the synthesis quality of the new view angle image can be improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method, electronic device, chip system, and readable storage medium. Background Technology

[0002] Novel perspective synthesis refers to the process of obtaining the geometric representation and color attributes of a scene from multiple perspective images of the same scene, thereby generating a new perspective image of the same scene. Novel perspective synthesis is widely used in a wide range of fields, including augmented reality (AR), virtual reality (VR), autonomous driving, and 3D content generation.

[0003] Common novel perspective synthesis methods can be applied to a wide range of scenes, but the quality of the generated novel perspective images is highly dependent on the accuracy of scene geometry information (such as the position, shape, and size of objects in the scene) and the information from a dense set of known perspective images. Therefore, in scenarios with limited information such as sparse known perspective images (e.g., a small number of known perspective images), unknown camera intrinsic and extrinsic parameters, and unknown perspective order of perspective images, the quality of novel perspective images synthesized by the above methods is poor. Summary of the Invention

[0004] This application provides an image processing method, electronic device, chip system, and readable storage medium, which are beneficial for improving the quality of image synthesis from new perspectives.

[0005] In a first aspect, embodiments of this application provide an image processing method, which may include: determining camera intrinsic and extrinsic parameter information and scene point cloud information for a target scene based on a first image and a second image in a target scene, wherein the first image and the second image have different shooting perspectives; determining first attribute information of a scene Gaussian sphere based on the camera intrinsic and extrinsic parameters and the scene point cloud information; determining a target depth image of the scene Gaussian sphere based on the first attribute information, a reference shooting perspective, and a reference optical center position; wherein the reference shooting perspective is different from the shooting perspectives of the first image and the second image; the target depth image includes the depth value of the distance from the intersection point of the observation ray and the scene Gaussian sphere to the reference optical center position, the endpoint position of the observation ray is the reference optical center position, and the direction of the observation ray is within the viewing angle range of the reference shooting perspective; adjusting the first attribute information based on the target depth image and color ground truth information to obtain second attribute information of the scene Gaussian sphere; and generating a target image for the target scene based on the reference shooting perspective and the second attribute information, wherein the shooting perspective of the target image is the reference shooting perspective.

[0006] This application can generate a new perspective image, i.e., a target image, based on two images taken from different perspectives—a first image and a second image—of a target scene, even when the camera's intrinsic and extrinsic parameters and the order of viewpoints are unknown. Furthermore, based on the target depth image and true color information, the first attribute information of the scene's Gaussian sphere can be optimized and adjusted to obtain the second attribute information, thereby facilitating the generation of a target image that more closely resembles the real target scene and improving the image quality of the target image.

[0007] In one possible implementation, determining the target depth image of the scene Gaussian sphere based on the first attribute information, the reference shooting viewpoint, and the reference optical center position may include: determining the first depth image of the scene Gaussian sphere based on scene point cloud information, the first attribute information, the reference shooting viewpoint, the reference optical center position, and the depth prior image; performing edge extraction processing on the first depth image to determine the non-edge regions of the first depth image; and performing depth regularization constraint processing on the non-edge regions of the first depth image to obtain the target depth image.

[0008] This improves the accuracy of the first depth image. Furthermore, by applying depth regularization constraints to the non-edge regions of the first depth image, a target depth image with smoothly varying depth values ​​can be obtained. This improves the accuracy of the depth values ​​in the target depth image, allows the scene Gaussian sphere to more accurately describe the target scene, and ultimately makes the target image synthesized based on the scene Gaussian sphere more realistic.

[0009] In one possible implementation, determining a first depth image of a scene Gaussian sphere based on first attribute information, a reference shooting viewpoint, and a reference optical center position includes: inputting the first attribute information and the reference shooting viewpoint into a multilayer perceptron to obtain a first distance value output by the multilayer perceptron; wherein the first distance value is the distance between the intersection point and the reference optical center position; determining first position information based on the reference optical center position and the first distance value; the first position information is the position information of the intersection point of the observation ray and the scene Gaussian sphere; determining a second distance value based on the first position information and the first distance value; wherein the second distance value is the distance between the projection point of the intersection point on the camera orientation and the reference optical center position, the camera orientation being determined based on camera intrinsic and extrinsic parameters; determining a first depth value of the scene Gaussian sphere based on the second distance value; and determining a first depth image of the scene Gaussian sphere based on the first depth value.

[0010] This allows for a more accurate determination of the first depth image, which is beneficial for generating high-quality target images.

[0011] In one possible implementation, the method may further include: determining the camera orientation based on camera intrinsic and extrinsic parameters.

[0012] The camera orientation is closely related to the rotation component of the extrinsic parameter matrix in the camera's intrinsic and extrinsic parameters. The rotation component is typically a 3×3 matrix that describes the rotation of the camera coordinate system relative to the world coordinate system. The third row of this rotation matrix can be considered as the camera's orientation vector along the z-axis in the world coordinate system (after rotation). By analyzing this rotation matrix, the camera orientation can be determined. Accurately estimating and setting the camera orientation is beneficial for generating new perspective images consistent with the real scene.

[0013] In one possible implementation, the method may further include: performing monocular depth estimation on a monocular image to determine a first monocular depth image corresponding to the monocular image, wherein the monocular image is either a first image or a second image; performing physical scale registration on the first monocular depth image based on a reference sparse depth image to obtain a second monocular depth image; and performing fusion processing on the reference sparse depth image and the second monocular depth image to obtain a depth prior image.

[0014] The first monocular depth image is a depth image with high geometric accuracy; the reference sparse depth image is a sparse depth map with accurate physical scale. Using the reference sparse image, the physical scale of the first monocular depth image can be registered to obtain a second monocular depth image, aligning its physical scale with that of the sparse depth image. By fusing the second monocular depth image and the reference sparse depth image, a depth prior image with realistic physical scale and high geometric accuracy can be obtained. Thus, using the depth prior image helps improve the accuracy of the target depth image of the Gaussian sphere in the scene, thereby facilitating the generation of target images that closely resemble real-world target scenes.

[0015] In one possible implementation, the method may further include: determining monocular scene point cloud information of the monocular image; and determining a reference sparse depth image based on the monocular scene point cloud information of the monocular image.

[0016] In this way, the reference sparse depth image obtained by rendering and other processing of the monocular scene point cloud information can be used to perform physical scale registration processing on the first monocular depth image, thereby improving the accuracy of the target depth image.

[0017] In one possible implementation, the first attribute information includes first geometric attribute information and first color attribute information; the above-mentioned adjustment of the first attribute information based on the target depth image and color ground truth information to obtain the second attribute information of the scene Gaussian sphere may include: adjusting the first geometric attribute information based on the target depth image to obtain the second geometric attribute information; adjusting the first color attribute information based on the color ground truth information to obtain the second color attribute information; determining the second attribute information of the scene Gaussian sphere, wherein the second attribute information includes the second geometric attribute information and the second color attribute information.

[0018] Specifically, by using the target depth image, the first geometric attribute information of the Gaussian sphere of the scene can be optimized to obtain the second geometric attribute information, enabling the second attribute information to more accurately represent the geometric structure of the target scene. By using ground truth color information, the second color attribute information can be made to closely approximate the true colors of the target scene. This improves the image quality of the generated target image, making its depth and color closely resemble the real target scene.

[0019] In one possible implementation, the first geometrical attribute information of the scene Gaussian sphere includes one or more of the following: size information, orientation information, sphere center position information, and opacity; the first color attribute information includes color information.

[0020] In one possible implementation, the color truth information can be the color information of the first image and / or the second image. Optionally, the color truth information refers to the actual color information of the first image and / or the second image, such as actual RGB color information.

[0021] In one possible implementation, a target image in the target scene is generated based on a reference shooting viewpoint and second attribute information, including: based on the reference shooting viewpoint and second attribute information, and calling the target Gaussian rendering model to render the scene Gaussian sphere to generate the target image in the target scene.

[0022] As can be seen, the target Gaussian rendering model can be used to perform three-dimensional Gaussian splash differentiable rendering of the scene Gaussian sphere, which can achieve a high-quality representation of the target scene, thereby improving the quality of the target image.

[0023] In one possible implementation, the method may further include: determining a training depth prior image based on multiple training images, wherein the multiple training images are images taken from different perspectives in the same training scene; and training an initial Gaussian rendering model based on the training depth prior image and the multiple training images to obtain a target Gaussian rendering model.

[0024] Among them, training the deep prior image can supervise the process of training the initial Gaussian rendering model, which helps to ensure that the initial Gaussian rendering model can continuously improve its accuracy and generalization ability during training, thereby obtaining an effective target Gaussian rendering model.

[0025] In one possible implementation, a training depth prior image is determined based on multiple training images, including: performing single-view depth estimation on a first training image among the multiple training images to determine a first training monocular depth image corresponding to the first training image; performing physical scale registration on the first training monocular depth image based on a training reference sparse depth image to obtain a second training monocular depth image; and fusing the training reference sparse depth image and the second training monocular depth image to obtain the training depth prior image.

[0026] The first training monocular depth image is a depth image with high geometric accuracy; the training reference sparse depth image is a sparse depth map with accurate physical scale. Using this training reference sparse image, the physical scale of the first training monocular depth image can be registered, resulting in a second training monocular depth image whose physical scale is aligned with that of the training reference sparse depth image. By fusing the second training monocular depth image and the training reference sparse depth image, a training depth prior image with realistic physical scale and high geometric accuracy can be obtained. Therefore, when the initial Gaussian rendering model is trained based on the training depth prior image, the rendering capability of the target Gaussian rendering model can be improved, which is beneficial for generating target images that closely resemble real-world scenes.

[0027] In one possible implementation, the method may further include: determining scene point cloud information corresponding to a training scene based on multiple training images; and rendering the scene point cloud information corresponding to the training scene to obtain a training reference sparse depth image.

[0028] The geometric information of the training reference sparse depth image obtained by rendering the scene point cloud information corresponding to the training scene is not dense enough, but the physical scale of the reference sparse depth image is consistent with reality. The training reference sparse depth image obtained in this way can be used to adjust the first training monocular depth image to improve the accuracy of the physical scale of the first training monocular depth image.

[0029] Secondly, embodiments of this application provide an image processing apparatus that, when operating on an electronic device, enables the electronic device to execute the method described in the first aspect or any implementation thereof.

[0030] Thirdly, embodiments of this application provide an electronic device comprising: one or more processors and a memory; the memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, and the one or more processors invoking the computer instructions to cause the electronic device to perform the method described in the first aspect or any implementation thereof.

[0031] Fourthly, embodiments of this application provide a chip system, which includes at least one processor, a memory, and an interface circuit. The memory, the interface circuit, and the at least one processor are interconnected via circuits. The at least one memory stores program instructions. When the program instructions are executed by the processor, the chip system performs the method described in the first aspect or any implementation thereof.

[0032] Fifthly, embodiments of this application provide a computer program product containing instructions that, when the computer program product is run on an electronic device, cause the electronic device to perform the method described in the first aspect or any implementation thereof.

[0033] In a sixth aspect, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on an electronic device, cause the electronic device to perform the method described in the first aspect or any implementation thereof. Attached Figure Description

[0034] Figure 1 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application;

[0035] Figure 2 A schematic diagram of the software structure of an electronic device provided in an embodiment of this application;

[0036] Figure 3 A schematic diagram illustrating a novel perspective image synthesis method provided in an embodiment of this application;

[0037] Figure 4 A schematic flowchart of an image processing method provided in an embodiment of this application;

[0038] Figure 5 A schematic diagram illustrating another novel perspective image synthesis provided in an embodiment of this application;

[0039] Figure 6 This is a scene diagram illustrating a first depth value provided in an embodiment of this application;

[0040] Figure 7 This is a schematic diagram illustrating a process for determining a reference depth image, as provided in an embodiment of this application. Detailed Implementation

[0041] The embodiments of this application will now be described with reference to the accompanying drawings.

[0042] The terms "first," "second," "third," and "fourth," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0043] It should be understood that in this application, "at least one (item)" means one or more, and "more than one" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0044] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0045] As used in this specification, the terms "component," "module," "system," etc., are used to refer to computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. As illustrated, applications running on computing devices and computing devices can both be components. One or more components may reside in a process and / or an execution thread, and components may be located on a single computer and / or distributed among two or more computers. Furthermore, these components can be executed from various computer-readable media on which various data structures are stored. Components can communicate, for example, via local and / or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system, and / or a network, such as the Internet interacting with other systems via signals).

[0046] To better understand the technical solutions provided in the embodiments of this application, before describing the technical solutions in the embodiments of this application, the hardware structure of the electronic device to which the embodiments of this application are applicable will be described first in conjunction with the accompanying drawings.

[0047] Electronic devices may include mobile terminal devices (such as smartphones, tablets, laptops, etc.), augmented reality (AR) devices, virtual reality (VR) devices, wearable devices, in-vehicle devices, etc.

[0048] For example, please refer to Figure 1 , Figure 1 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. It should be understood that the electronic device 100 may have a... Figure 1 The more or fewer components shown can be combined into two or more components, or they can have different component configurations. Figure 1 The various components shown can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits.

[0049] Electronic device 100 may include: processor 110, external memory interface 120, internal memory 121, universal serial bus (USB) interface 130, charging management module 140, power management module 141, battery 142, antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194, and subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0050] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include... Figure 1 This may involve more or fewer components, or combining certain components, or splitting certain components, or different component arrangements. Figure 1 The components shown can be implemented in hardware, software, or a combination of both.

[0051] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0052] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to the instruction opcode and timing signals to complete the control of fetching and executing instructions.

[0053] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are being used repeatedly.

[0054] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0055] The interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0056] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0057] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Mini LED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 100 may include one or N displays 194, where N is a positive integer greater than 1.

[0058] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0059] The ISP is used to process data fed back from the camera. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's image sensor. The light signal is converted into an electrical signal, and the image sensor transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye.

[0060] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor.

[0061] A digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals.

[0062] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. Thus, electronic device 100 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0063] An NPU (Neural Processing Unit) is a computational processor for neural networks (NNs). By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0064] The external memory interface 120 can be used to connect an external memory card to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to perform data storage functions.

[0065] Internal memory 121 can be used to store computer executable program code, including instructions. Processor 110 executes various functional applications and data processing of electronic device 100 by running the instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application required for a function (such as a camera function), etc. The data storage area may store data created during the use of electronic device 100 (such as photos), etc. Furthermore, internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0066] Electronic device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0067] The pressure sensor 180A is used to sense pressure signals and can convert pressure signals into electrical signals.

[0068] The gyroscope sensor 180B can be used to determine the motion attitude of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 about three axes (i.e., the x, y, and z axes) can be determined by the gyroscope sensor 180B.

[0069] The magnetic sensor 180D includes a Hall effect sensor. The electronic device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip cover. The accelerometer 180E can detect the magnitude of the acceleration of the electronic device 100 in various directions (generally three axes). When the electronic device 100 is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the device's posture, and can be applied to applications such as screen orientation switching and pedometers.

[0070] A distance sensor 180F is used to measure distance. Electronic device 100 can measure distance via infrared or laser. In some embodiments, during a shooting scene, electronic device 100 can utilize the distance sensor 180F to measure distance for rapid focusing.

[0071] The proximity sensor 180G may include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode. The LED may be an infrared LED. The electronic device 100 emits infrared light outward through the LED. The electronic device 100 uses the photodiode to detect infrared reflected light from nearby objects in order to automatically turn off the screen to save power. The proximity sensor 180G can also be used in folding mode and pocket mode for automatic unlocking and locking of the screen.

[0072] The ambient light sensor 180L is used to sense the ambient light intensity. The electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the sensed ambient light intensity. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures.

[0073] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can utilize the characteristics of the collected fingerprints to achieve fingerprint unlocking, accessing application locks, taking photos with fingerprints, answering calls with fingerprints, etc.

[0074] Touch sensor 180K, also known as a "touch panel," can be located on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a "touchscreen." Touch sensor 180K detects touch operations applied to or near it. The touch sensor can then transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194.

[0075] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch-sensitive buttons. Electronic device 100 can receive button input and generate key signal inputs related to user settings and function control of electronic device 100.

[0076] In addition, an operating system runs on top of the aforementioned components. Examples include the iOS operating system, the Android open-source operating system, and the Windows operating system.

[0077] The operating system of electronic device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses the layered architecture Android system as an example to exemplify the software structure of electronic device 100. It should be noted that although this application embodiment uses the Android system as an example, its basic principles are equally applicable to electronic devices based on operating systems such as iOS or Windows.

[0078] Figure 2 This is a schematic diagram of the software structure of an electronic device provided in an embodiment of this application. The software structure adopts a layered architecture, which divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. Taking the Android system, which runs on an application processing unit (AP), as an example, in some embodiments, the Android system is divided into five layers, from top to bottom: the application layer, the application framework layer, the system runtime library layer, the hardware abstraction layer (HAL), and the kernel layer.

[0079] The application layer can include a series of application packages. These application packages can include applications such as camera, calendar, news, music, SMS, gallery, phone, Bluetooth, video, and maps. The application layer can also include a system user interface (system UI), which displays the interface of the electronic device 100, such as the desktop, camera interface, and photo album interface.

[0080] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes predefined functions. For example, it may include a window manager, content provider, view system, telephone manager, resource manager, notification manager, etc. The telephone manager is used to provide the calling functionality of the electronic device 100, such as managing call status (including connection, hang-up, etc.).

[0081] The system runtime library layer is divided into two parts: C / C++ libraries and the Android runtime library. The Android runtime includes the runtime environment, primarily responsible for the scheduling and management of the Android system. The C / C++ libraries mainly include the media framework, surface manager, 3D graphics engine, and 2D graphics engine. For example, the 3D graphics engine may include the Open Graphics Library (OpenGL); the 2D graphics engine may include the Skia Graphics Library (SGL).

[0082] The surface manager manages the display subsystem and provides fusion of 2D and 3D layers for multiple applications. The 3D graphics processing library implements 3D graphics drawing, image rendering, compositing, and layer processing. The 2D graphics engine is the drawing engine for 2D drawing.

[0083] The Hardware Abstraction Layer (HAL) isolates the application framework layer from the kernel layer, preventing the Android system from overly relying on the kernel layer. This allows the application framework layer to be developed without considering drivers. The HAL can include multiple functional modules, such as the Display HAL, Camera HAL, Audio HAL, and Sensor HAL modules.

[0084] The kernel layer is the layer between hardware and software. The kernel layer includes at least a display driver, a camera driver, an audio driver, a sensor driver, and a shared memory driver. In this embodiment, the display driver can be used to display two-dimensional and three-dimensional images. The camera driver can be used to capture photos or videos.

[0085] The following explanations will cover some of the terms used in the embodiments of this application to facilitate understanding by those skilled in the art.

[0086] 1. Depth map

[0087] Depth refers to the distance information in three-dimensional space inferred from a two-dimensional image through technical means. It can represent the distance between a point in a scene and the observer (such as a camera, video camera, or other shooting tool).

[0088] A depth map is a grayscale image that records the distance from each point in a scene to the observer. In other words, a depth map records information indicating the distance of a pixel to the observer. This distance information is usually represented in the depth map as a single-channel pixel value. Specifically, each pixel in the depth map corresponds to a specific point in the scene, and the value of this pixel reflects the distance of that point in the scene to the observer's plane, such as the distance from that point to the camera. The smaller the depth value of a pixel, the closer it is to the camera; the larger the depth value of a pixel, the farther it is from the camera.

[0089] 2. Camera pose

[0090] Camera pose, also known as camera orientation, refers to the camera's position and orientation in space. The camera's position is typically represented by a three-dimensional coordinate point. Camera orientation refers to the camera's orientation relative to a reference direction, and is usually described using a rotation matrix or rotation vector.

[0091] Camera pose estimation has wide applications in machine vision, robot navigation, and augmented reality. For example, in robot navigation, estimating the camera pose enables autonomous localization and path planning for the robot. In augmented reality, precisely calculating the camera pose allows virtual objects to be accurately superimposed onto a real-world scene.

[0092] 3. Camera internal and external parameters

[0093] Camera intrinsic parameters and extrinsic parameters are used to describe the internal characteristics of the camera and its position and orientation in three-dimensional space.

[0094] (1) Camera internal parameters

[0095] Camera intrinsic parameters are parameters that describe the internal properties of a camera. These parameters are typically fixed for a specific camera model and do not change over time. Intrinsic parameters include focal length, principal point (optical center) coordinates, and distortion coefficients.

[0096] Camera intrinsic parameters can be used to estimate camera pose. For example, in 3D reconstruction or camera pose estimation, camera intrinsic parameters are used to transform points on the image into the camera coordinate system in order to solve for the camera pose.

[0097] (2) Camera external parameters

[0098] Camera extrinsic parameters are parameters that describe the position and orientation of the camera in the world coordinate system. Camera extrinsic parameters can be used to directly describe the camera pose.

[0099] Camera extrinsic parameters typically include a rotation matrix R and a translation vector t, used to describe how points are transformed from the world coordinate system to the camera coordinate system. The translation vector t represents the position of the camera center in the world coordinate system. The rotation matrix R represents the camera's orientation and determines the relationship between the camera coordinate system and the world coordinate system.

[0100] 4. Point cloud

[0101] A point cloud is a data structure that represents objects in three-dimensional space, consisting of many discrete points. Each point in a point cloud represents a position in space, which can be represented by three-dimensional coordinates (X, Y, Z). Each point may also contain other attribute information, such as color information, normal vector, laser reflection intensity, etc.

[0102] Based on the number of points and the spacing between them, point clouds can be classified into sparse point clouds and dense point clouds. Dense point clouds contain a large number of points with small spacing between them, while sparse point clouds contain fewer points with larger spacing between them.

[0103] Point clouds have wide applications in many fields. For example, point clouds are the basic data for creating 3D models. By processing and analyzing point clouds, information such as the shape, size, and position of a target object can be identified, and accurate 3D models can be generated, which can then be used in product design, animation production, and other fields.

[0104] 5. Scene Gaussian Sphere

[0105] A Gaussian sphere is a three-dimensional spatial representation of a scene, constructed using the properties of the Gaussian distribution. In three-dimensional space, a Gaussian sphere can be viewed as a spherical region centered at a certain point with a radius of a positive real number. Gaussian spheres are commonly used to represent the probability distribution or attribute distribution at a specific location within a scene.

[0106] A scene Gaussian sphere can be used to represent attributes such as the shape, color, opacity, or transparency of objects in a scene. By adjusting the properties of the Gaussian sphere (such as position, covariance matrix, opacity, color, etc.), precise control over scene details can be achieved.

[0107] 6. Three-dimensional reconstruction (3D reconstruction)

[0108] 3D reconstruction is a technique for recovering 3D models or scenes from 2D images or data, encompassing multiple steps from data acquisition, preprocessing, 3D modeling to rendering. The 3D reconstruction process typically requires images from multiple perspectives as input, which can be obtained through cameras or other imaging devices. By processing and analyzing these images, information such as the object's contours, textures, and depth can be extracted, and a 3D model of the object can be constructed based on this information.

[0109] 3D reconstruction can be widely used in fields such as computer vision, robotics, medical imaging, virtual reality, and augmented reality.

[0110] 7. Novel View Synthesis (NVS)

[0111] New perspective synthesis refers to generating new images of the same scene or object from different perspectives based on multiple known images of the same scene. See also: Figure 3 Based on such Figure 3 Multiple images of the same scene shown can be synthesized from a new perspective to obtain a new perspective image of the scene.

[0112] For example, in generating new perspective images using novel perspective synthesis methods, it is usually necessary to reconstruct the scene or object in 3D to obtain information such as its 3D structure and shape. This information typically includes the object's geometry, positional relationships, and camera pose. By processing and analyzing this information, computer algorithms can be used to project the 3D model onto the new perspective, thereby generating a new 2D image.

[0113] Common novel perspective synthesis methods are applicable to a wide range of scenes, but the quality of the generated novel perspective images is highly dependent on the accuracy of scene geometry information (such as the position, shape, and size of objects in the scene) and the information from a dense set of known perspective images. Therefore, in constrained scenarios such as sparse known perspective images (e.g., a small number of known perspective images), unknown camera pose or camera intrinsic and extrinsic parameters, or unknown perspective order of perspective images, the quality of novel perspective images synthesized by the above methods is poor.

[0114] In view of this, embodiments of this application provide an image processing method, electronic device, chip system, and readable storage medium, which can be applied to sparse view scenarios to process images with unknown camera intrinsic and extrinsic parameters and unknown camera view order to generate new view images in the same scene. This is beneficial to improving the efficiency and accuracy of new view synthesis and improving the image quality of new view images.

[0115] Please see Figure 4 , Figure 4This is a schematic flowchart illustrating an image processing method provided in an embodiment of this application. The image processing method can be applied to an electronic device, and can be executed by the electronic device itself, by a processor within the electronic device, or by a chip or chip system within the electronic device that has processor functionality. For example... Figure 4 As shown, the image processing method may include, but is not limited to, the following steps:

[0116] S401, based on the first and second images of the target scene, determines the camera's intrinsic and extrinsic parameters and the scene's point cloud information.

[0117] The target scene refers to the specific environment, location, or situation that you wish to capture, record, or analyze. Shooting the target scene from different angles yields a first image and a second image. In other words, the first image and the second image are shot from different angles.

[0118] Optionally, the target scene may include natural landscape scenes, urban street scenes, indoor environment scenes, etc. For example, when using an electronic device to film the interior of a residence, the target scene is the interior of that residence, including elements such as the floor plan, furniture layout, and decoration style.

[0119] The first image and the second image can be photographs taken by the same shooting device (such as a smartphone or camera), or they can be photographs taken by different shooting devices. Alternatively, the first image and the second image can be images stored in the image library (such as a photo album) of an electronic device. The shooting order between the first image and the second image can be arbitrary; for example, the second image can be taken before the first image, and this application does not limit this.

[0120] Before performing step S401 on the first and second images, the viewing order between the first and second images, the camera intrinsic and extrinsic parameters when capturing the first image, and the camera extrinsic and extrinsic parameters when capturing the second image may be unknown. The viewing order refers to the order or method of viewpoint transition used when the images are presented. The camera extrinsic and extrinsic parameters when capturing the first or second image include camera intrinsic parameters and camera extrinsic parameters. Camera intrinsic parameters may include distance, principal point (optical center) coordinates, distortion coefficients, etc.; camera extrinsic parameters may include rotation matrix R and translation vector t.

[0121] Optionally, the first image and the second image can be images containing color information. For example, the color spaces of the first image and the second image may include red-green-blue (RGB) color space, lightness-color difference (YUV) color space, hue-saturation-value (HSV) color space, etc. Accordingly, the first image and the second image may include RGB images, YUV images, HSV images, etc.

[0122] The camera intrinsic and extrinsic parameters of the target scene, also known as the camera pose, are estimated based on the first and second images. These parameters determine the camera orientation. Accurate mapping between the first and second images is possible using the camera intrinsic and extrinsic parameters of the target scene, facilitating the accurate representation of the target scene in three-dimensional space.

[0123] Scene point cloud information refers to the point cloud used to describe a target scene in three-dimensional space. The scene point cloud information of a target scene includes multiple discrete points, and each point in the point cloud can include information such as its three-dimensional coordinates, color information, normal vector, and laser reflection intensity. The scene point cloud information of a target scene can be used to reconstruct three-dimensional objects or scene models in the real world, thus facilitating the accurate representation of the target scene in three-dimensional space.

[0124] In one possible implementation, the dense unconstrained stereo three-dimensional reconstruction (DUSt3R) method can be used to determine the scene point cloud information and camera intrinsic and extrinsic parameters of the current scene based on the first and second images.

[0125] DUSt3R is a 3D reconstruction framework capable of directly recovering the 3D point position information in the corresponding camera coordinate system from an unconstrained image set. For example, inputting the first and second images into the DUSt3R deep learning model can regress the initial point cloud images for each image. Through a global alignment strategy, the relative pose and depth information of the camera can be estimated. Transforming the two initial point cloud images to the same coordinate system optimizes their alignment, yielding the alignment result. From the alignment result, the scene point cloud information of the target scene can be extracted. This scene point cloud information contains the complete geometric structure of the target scene. Based on the parameter estimation during the global alignment process, the absolute pose and intrinsic parameters of the camera can be recovered, thus determining the camera's intrinsic and extrinsic parameters in the target scene.

[0126] S402 determines the first attribute information of the scene Gaussian sphere based on the camera's intrinsic and extrinsic parameters and scene point cloud information.

[0127] A Gaussian sphere is a mathematical model that uses the Gaussian distribution to represent the geometric structure and color information of a target scene. Specifically, a set of Gaussian spheres (or Gaussian models) can be used to approximate the objects and surfaces in a target scene. Each Gaussian sphere has a center point (also called the sphere center), a covariance matrix (describing shape and size), opacity, and color parameters.

[0128] The first attribute information of the scene Gaussian sphere refers to the attribute information of the initialized scene Gaussian sphere, including one or more of the following: center position information, size information, orientation information, opacity, and color information for each Gaussian sphere. Optionally, the first attribute information may include first geometric attribute information and first color attribute information. The first geometric attribute information includes one or more of the following: center position information, size information, orientation information, and opacity. The first color attribute information includes color information.

[0129] The location of the sphere's center can be represented by its three-dimensional coordinates. The size can be represented by the covariance matrix of the Gaussian sphere.

[0130] Orientation information refers to the orientation of a Gaussian sphere, that is, the directional characteristics presented by the normal mapping of each point on the Gaussian sphere. In other words, it refers to the geometric direction represented by the pattern (such as a large circle, a small circle, etc.) formed by the points mapped onto the Gaussian sphere. Orientation information is helpful in identifying and understanding the geometric characteristics of the surface of an object in a target scene, such as normal direction and surface curvature.

[0131] Opacity refers to the degree to which light passes through a Gaussian sphere, describing the extent to which an object blocks light. For example, a higher opacity of a Gaussian sphere indicates that the spatial location it describes is more likely to be the surface of an object in the target scene. Optionally, the degree to which light passes through a Gaussian sphere can also be represented by transparency.

[0132] Color information refers to the color of the Gaussian sphere, which can be represented by RGB values ​​or by spherical harmonics. This application does not limit the representation to either of these methods.

[0133] S403, based on the first attribute information, the shooting angle, and the reference optical center position, determines the target depth image of the scene Gaussian sphere.

[0134] The reference shooting perspective differs from the shooting perspective of the first image and the second image. Optionally, the reference shooting perspective can be a shooting perspective generated by an electronic device or a shooting perspective received by the user and selected by the user; this application does not limit this.

[0135] The target depth image of the scene Gaussian sphere includes the depth value of the intersection point of the observation ray and the scene Gaussian sphere from the reference optical center. The endpoint of the observation ray is the reference optical center, and the direction of the observation ray is within the field of view of the reference shooting angle.

[0136] In one possible implementation, the process of obtaining the target depth image may include the following steps: determining a first depth image of the scene's Gaussian sphere based on scene point cloud information, first attribute information, a reference shooting viewpoint, a reference optical center position, and a depth prior image; performing edge extraction processing on the first depth image to determine the non-edge regions of the first depth image; and performing depth regularization constraint processing on the non-edge regions of the first depth image to obtain the target depth image.

[0137] In one possible implementation, the depth prior image can be determined by the following steps: performing monocular depth estimation on the monocular image to determine the first monocular depth image corresponding to the monocular image, wherein the monocular image is either the first image or the second image; performing physical scale registration on the first monocular depth image based on the reference sparse depth image to obtain the second monocular depth image; and performing fusion processing on the reference sparse depth image and the second monocular depth image to obtain the depth prior image.

[0138] Optionally, the monocular scene point cloud information of the monocular image can be determined by rendering the monocular image. Based on the monocular scene point cloud information of the monocular image, the aforementioned reference sparse depth image can be determined. Optionally, the DUSt3R method can be used to render the monocular image to determine the monocular scene point cloud information.

[0139] Optionally, based on the same principle, the process for determining the depth prior image can be found in [reference needed]. Figure 7 And the following text Figure 7 Related descriptions.

[0140] During acquisition and transmission, the first and second images may be subject to various noise interferences, which can lead to inaccurate depth values ​​and thus affect the accuracy of the depth values ​​in the first depth image. The edge regions of the first depth image typically contain important features of the target scene, such as the outlines and shapes of objects; while non-edge regions are theoretically relatively smooth and do not contain much detailed information. Therefore, applying depth regularization constraints to non-edge regions can reduce the impact of noise on the first depth map while preserving image features, making the depth map smoother and thus improving the accuracy of the depth values.

[0141] Optionally, an edge extractor or edge extraction algorithm can be used to determine the non-edge regions of the first depth image.

[0142] In one possible implementation, the specific process of obtaining the first depth image may include the following steps: inputting first attribute information and a reference shooting viewpoint into a multilayer perceptron (MLP) network to obtain a first distance value output by the MLP network; wherein, the first distance value is the distance between the intersection point and the reference optical center position. Based on the reference optical center position and the first distance value, first position information can be determined. The first position information is the position information of the intersection point of the observation ray and the scene Gaussian sphere. Based on the first position information and the first distance value, a second distance value can be determined; wherein, the camera orientation can be determined based on the camera's intrinsic and extrinsic parameters, and the second distance value is the distance between the projection point of the intersection point on the camera orientation and the reference optical center position. Based on the second distance value, a first depth value of the scene Gaussian sphere can be determined. Based on the first depth value, a first depth image of the scene Gaussian sphere can be determined. The first depth value of the scene Gaussian sphere can also be referred to as the depth value of the scene Gaussian sphere from the reference optical center position.

[0143] Optionally, the first depth value of the scene Gaussian sphere can be equal to the second distance value. Optionally, after obtaining the second distance value, the second distance value can be validated to obtain a validated second distance value, and the validated second distance value can be used as the first depth value.

[0144] The first distance value, which is the distance between the reference optical center and the intersection point, is related to the direction of the observed ray, the size, center position, and orientation of the scene Gaussian sphere, etc. By inputting the direction of the observed ray, the size, center position, and orientation of the scene Gaussian sphere into the MLP network, the first distance value output by the MLP network can be obtained. This improves the accuracy of the first distance value.

[0145] For example, please see Figure 6 This is a scene illustration of a first depth value provided in an embodiment of this application. Figure 6 As shown, point O represents the reference optical center; ray d1 represents the observation ray, which is a ray emitted from the reference optical center O towards a pixel in the scene's Gaussian sphere, and its direction can be represented by the arrow direction of ray d1; the area enclosed by the ellipsoidal dashed box represents the scene's Gaussian sphere, with its center at μ; the intersection of the observation ray and the scene's Gaussian sphere is ψ. The distance between the reference optical center O and the intersection point ψ, i.e., the first distance value, can be represented as L1. The reference optical center O can be used as the origin of the coordinate axes. Based on the extrinsic parameter matrix in the camera's intrinsic and extrinsic parameters, the camera orientation can be determined. Assuming the camera orientation is as follows... Figure 6 Ray d2 in the diagram represents the point where the reference optical center O passes through the sphere center μ. Projecting the intersection point ψ onto the camera's orientation yields the projection point β. Thus, the depth of the intersection point ψ from the reference optical center is equal to the depth of the projection point β from the reference optical center, i.e., the second distance value. Where, as... Figure 6As shown, the second distance value can be represented as L2. In other words, the second distance value is the component of the first distance value in the camera orientation, which can be represented as Z, i.e., Z = L2.

[0146] Optionally, the second distance value can be determined using a projection function. Specifically, based on the position O of the reference optical center O, the first distance value L1, and the direction of the observed ray, i.e., the direction of ray d1, the position information of the projection point β can be determined, which can be expressed as (O+L1*d1). Using the projection function π(·), the component of the first distance value in the camera orientation direction can be obtained as π(O+L1*d1), which is the second distance value.

[0147] Optionally, in a special case, such as when the intersection of the observed ray and the scene Gaussian sphere is at the sphere's center μ, the first distance value equals the second distance value, which is the third distance value between the sphere's center and the reference optical center, as shown below. Figure 6 L3 is represented in the context. In this case, the first depth value of the scene's Gaussian sphere is L3.

[0148] S404, based on the target depth image and color ground truth information, adjust the first attribute information to obtain the second attribute information of the scene Gaussian sphere.

[0149] The first attribute information may include first geometric attribute information and first color attribute information.

[0150] Optionally, based on the target depth image, the first geometric attribute information in the first attribute information can be adjusted to obtain the second geometric attribute information of the scene Gaussian sphere. Based on the color ground truth information, the first color attribute information in the first attribute information can be adjusted to obtain the second color attribute information of the scene Gaussian sphere. Thus, based on the second geometric attribute information and the second color attribute information, the second attribute information of the scene Gaussian sphere can be determined. In other words, the second attribute information includes both the second geometric attribute information and the second color attribute information.

[0151] The second attribute information can be considered as the optimized and adjusted first attribute information, and may include one or more of the following: the size, orientation, center position, opacity, and color information of the scene Gaussian sphere. Specifically, the second geometric attribute information includes one or more of the adjusted size, orientation, center position, and opacity information of the scene Gaussian sphere. The second color attribute information includes the adjusted color information.

[0152] In one possible implementation, the color truth information can be the color information of the first image and / or the second image. Optionally, the color truth information refers to the actual color information of the first image and / or the second image, such as actual RGB color information.

[0153] In this way, adjusting the first geometric attribute information through the target depth image helps improve the accuracy of the scene Gaussian sphere's representation of the target scene's geometric structure. Adjusting the first color attribute information through the color ground truth information helps improve the accuracy and realism of the scene Gaussian sphere's representation of the target scene's color. Therefore, the above steps optimize the first attribute information, enabling the scene Gaussian sphere to more accurately describe the target scene, thus facilitating the acquisition of more realistic new perspective images based on the scene Gaussian sphere.

[0154] S405 generates a target image of the target scene based on a reference shooting perspective and second attribute information.

[0155] A target image is a new perspective image of a target scene, captured from a reference shooting perspective, thus differing from the shooting perspectives of the first and second images. In this way, different new perspective images can be generated using different reference shooting perspectives, which helps to obtain more comprehensive information about the target scene.

[0156] For example, such as Figure 5 As shown, the first image includes elements of the target scene such as the corridor, corridor railings, decorative paintings, and the front bedroom, and the shooting angle of the first image is relatively far from the front bedroom. The second image includes elements of the target scene such as the corridor, decorative paintings, part of the right-side bedroom, and the front bedroom, and the shooting angle of the second image is relatively close to the front bedroom. Figure 5 As shown, the target image obtained based on the first image and the second image contains the same content as the first image and the second image, but the shooting angle of the target image is located between the shooting angle of the first image and the shooting angle of the second image.

[0157] The target image is a color image whose color space is the same as that of the first and second images. For example, if the first and second images are RGB images, then the target image is also an RGB image.

[0158] In one possible implementation, 3D Gaussian splatting differentiable rendering technology can be used to render the Gaussian sphere of the scene to obtain the target image of the target scene. Specifically, based on the reference shooting perspective and second attribute information, and by calling the target Gaussian rendering model, the scene Gaussian sphere can be rendered to obtain the target image.

[0159] Among them, 3D Gaussian splatter differentiable rendering is a computer graphics technique that combines the principles of 3D Gaussian distribution and differentiable rendering to achieve high-quality real-time rendering of 3D scenes. Specifically, 3D Gaussian splatter technology uses a 3D Gaussian distribution to describe pixels in the target scene. Each pixel is treated as a Gaussian function with a specific location and covariance matrix, and these Gaussian functions together constitute a continuous representation of the target scene. During the rendering process, each Gaussian function is projected onto the screen, forming a blurred circular area (i.e., a "splatter"). The final rendered image is generated by superimposing and blending these circular areas.

[0160] Optionally, the target Gaussian rendering model can perform depth rendering on the scene Gaussian sphere using the following formula (1) to obtain the first depth image. For any pixel q in the target scene, mapping it to the world coordinate system yields the corresponding pixel p in the world coordinate system. The scene Gaussian sphere includes multiple Gaussian spheres, and the observation ray emitted from the reference optical center to pixel p will pass through one or more Gaussian spheres. For example, assume that the observation ray will pass through N Gaussian spheres (N is an integer greater than or equal to 1). Using formula (1), the contribution of each Gaussian sphere passed by the observation ray to the depth value and other attributes of the pixel can be calculated. Where D(p) represents the depth value of pixel p; Z i α represents the first depth value of the i-th Gaussian sphere through which the observed ray currently passes; i (p) represents the opacity of the i-th Gaussian sphere; α j (p) represents the opacity of the j-th Gaussian sphere. Here, the j-th Gaussian sphere represents the Gaussian sphere the observation ray passes through before passing the i-th Gaussian sphere. In formula (1)... This can be viewed as the weight of the i-th Gaussian sphere, which describes the contribution of the Gaussian sphere to attributes such as the depth value of pixel p. Based on formula (1), the contributions of each Gaussian sphere among the N Gaussian spheres through which the observation ray passes can be accumulated to obtain the depth value of pixel p.

[0161]

[0162] Optionally, the target Gaussian rendering model can render the scene Gaussian sphere using the following formula (2) to obtain the target image. For any pixel q in the target scene, mapping it to the world coordinate system yields the corresponding pixel p in the world coordinate system. The scene Gaussian sphere includes multiple Gaussian spheres, and the observation ray emitted from the reference optical center to pixel p will pass through one or more Gaussian spheres. For example, assume that the observation ray will pass through N Gaussian spheres (N is an integer greater than or equal to 1). Using formula (2), the contribution of each Gaussian sphere passed by the observation ray to the color attribute of the pixel can be calculated. Wherein, RGB(p) represents the RGB value of pixel p; c i This represents the RGB value of the i-th Gaussian sphere that the observed ray is currently passing through; α i (p) represents the opacity of the i-th Gaussian sphere; α j (p) represents the opacity of the j-th Gaussian sphere. Here, the j-th Gaussian sphere represents the Gaussian sphere the observation ray passes through before passing the i-th Gaussian sphere. In formula (2)... The weight can be viewed as the weight of the i-th Gaussian sphere, which describes the contribution of the Gaussian sphere to the color attribute of pixel p. Based on formula (1), the contribution of each Gaussian sphere among the N Gaussian spheres through which the observation ray passes can be accumulated to obtain the RGB value of pixel p.

[0163]

[0164] In one possible implementation, the target Gaussian rendering model can be trained on an initial Gaussian rendering model based on a training depth supervision image and multiple training images. These multiple training images are taken from the same training scene at different angles. Based on these multiple training images, a training depth prior image can be determined. Optionally, during model training, training color ground truth information can also be used to supervise the process. Thus, when the initial Gaussian rendering model is trained based on the training depth prior image and training color ground truth information, the rendering capability of the target Gaussian rendering model can be improved, which is beneficial for generating target images that closely resemble the real target scene.

[0165] Optionally, the training process of the Gaussian rendering model can be performed by an electronic device or by a remote server, and this application does not limit this.

[0166] Optionally, for details on how to determine the specific implementation of training depth prior images, please refer to [link / reference]. Figure 7 And the following text Figure 7 The description.

[0167] Optionally, training color ground truth information refers to the color information of the training image corresponding to the new perspective image generated during model training. For example, suppose there are 10 training images, referred to as training image 1 to training image 10, with shooting perspectives of view 1 to view 10 respectively. Suppose that during a training process, based on training image 1 and training image 2, an image with a shooting perspective of view 8 (called the new perspective training image) can be generated. Then, in this training process, training color ground truth information refers to the color information of training image 8. In this way, the color information of training image 8 can be used to evaluate whether the color of the new perspective training image is the same as the real color. Therefore, using training color ground truth information to supervise the training process of the initial Gaussian rendering model is beneficial to improving the accuracy and realism of the colors of the images rendered by the target Gaussian rendering model.

[0168] In this way, supervising the training process of the initial Gaussian rendering model using training depth prior images and training color ground truth information helps improve the accuracy of the target Gaussian rendering model in rendering depth maps. Based on this, the target Gaussian rendering model can render target images with realistic colors and depth values ​​that conform to the actual physical scale.

[0169] In one possible implementation, multiple new perspective images under different reference shooting angles can be determined based on the first image and the second image, and new perspective videos can be synthesized based on these new perspective images.

[0170] As can be seen, through the embodiments of this application, a new perspective image can be generated based on a very small number of viewpoint images of the target scene, even when the camera's internal and external parameters and viewpoint order are unknown. This improves the image quality of the new perspective image, ensuring its physical scale matches reality and accurately describes the target scene. For example, through the embodiments of this application, scene point cloud information and a scene Gaussian sphere that accurately represent the target scene can be obtained based on the first and second images, improving the efficiency and accuracy of 3D reconstruction of the target scene. Based on the reference shooting viewpoint, the attribute information of the scene Gaussian sphere, and the first depth value, a new perspective image—the target image—can be generated for the target scene, further improving its quality.

[0171] The following describes the specific implementation process for determining the reference depth image.

[0172] Please see Figure 7 , Figure 7 This is a schematic diagram illustrating a process for determining a reference depth image, as provided in an embodiment of this application. Figure 7 As shown, the process of determining the reference depth image may include, but is not limited to, the following steps:

[0173] S701 determines the scene point cloud information corresponding to the training scene based on multiple training images taken from different perspectives in the same training scene.

[0174] In this process, the intrinsic and extrinsic parameters of the camera corresponding to the training scene can also be determined.

[0175] The process of determining the scene point cloud information corresponding to the training scene follows the same principle as determining the scene point cloud information of the target scene. For details, please refer to [reference needed]. Figure 4 The description in S401 shown will not be repeated here.

[0176] S702 renders the scene point cloud information corresponding to the training scene to obtain a training reference sparse depth map.

[0177] The training reference sparse depth map is a sparse depth map with accurate physical scale. Using this training sparse map, the physical scale of the first training monocular depth map can be registered.

[0178] S703, perform single-view depth estimation on the first training image among multiple training images to determine the first training monocular depth map corresponding to the first training image.

[0179] Optionally, a monocular depth estimation method of any depth can be used to perform single-view depth estimation on the first training image. The first training monocular depth map is a dense depth image in which each pixel corresponds to a depth value, providing richer depth information and helping to accurately reflect the shape and position of objects in the training scene.

[0180] Optionally, multiple first training images can be selected from multiple training images, and single-view depth estimation can be performed on each first training image to obtain multiple training monocular depth maps. Then, the accuracy of the depth values ​​of the multiple training monocular depth maps is evaluated, and the training monocular depth map with the highest depth value accuracy is selected as the first training monocular depth map.

[0181] S704, based on the training reference sparse depth map, performs physical scale registration on the first training monocular depth map to obtain the second training monocular depth map.

[0182] The physical scale of the first training monocular depth map obtained by single-view depth estimation may not be consistent with the actual physical scale, while the physical scale of the training reference sparse depth map is consistent with the actual physical scale. Therefore, the scale ambiguity problem of monocular depth estimation can be solved by training the reference sparse depth map.

[0183] Optionally, the scale value and translation can be solved using formula (3). Where, s* t represents the target scale obtained by solving. * p represents the solved target offset; p represents the first depth image D. dense A pixel in D; sparse (p) indicates that pixel p is in the training reference sparse depth map D. sparse The depth value in D; dense (p; s, t) represents the result of adjusting the scale and offset of pixel p using s and t in the first depth image D. dense The depth value in the text.

[0184]

[0185] Thus, the target scale and target offset obtained through formula (3) can be used to adjust the first training monocular depth map, thereby obtaining the second training monocular depth map. Optionally, the second training monocular depth image D can be determined through the following formula (4). d ′ ense This ensures that the second training monocular depth map has accurate depth information and conforms to the actual physical scale.

[0186] D d ′ ense =s * ×D dense +t * Formula (4)

[0187] S705 fuses the training reference sparse depth map and the second training monocular depth map to obtain the training depth prior image.

[0188] Optionally, the training depth prior image can be represented as D′ dense ∪D sparse .

[0189] Specifically, for any pixel p in the second training monocular depth image, if the depth value of pixel p is the same in the training reference sparse depth image and the second training monocular depth image, it is denoted as D. p This means that the depth value of pixel p is accurate; in other words, the depth value of pixel p in the training depth prior image is equal to D. p For any pixel q in the second training monocular depth image, if the depth value of pixel q in the training reference sparse depth image is D1, and the depth value in the second training monocular depth image is D2, and D1 and D2 are different, then the depth value D in the training reference sparse depth image can be determined. q As the depth value of pixel q, i.e., D q=D1. In other words, in the training depth prior image, the depth value of pixel p is equal to the depth value D of p in the second training monocular depth image. p The depth value of pixel q is equal to the depth value D1 of q in the training reference sparse depth image.

[0190] This ensures that the depth values ​​of the training depth prior image conform to the actual physical scale and guarantees the accuracy of the depth value of each pixel, thereby improving the accuracy and realism of the depth map rendering of the target Gaussian rendering model.

[0191] Some embodiments of this application provide an image processing apparatus that, when operated on an electronic device, enables the electronic device to execute the image processing methods described in this application.

[0192] Some embodiments of this application provide an electronic device, which includes one or more processors and a memory; the memory is used to store computer program code, which includes computer instructions, and when one or more processors execute the computer instructions, the electronic device performs the above-described image processing method.

[0193] Some embodiments of this application provide a chip system applied to an electronic device. The chip system includes at least one processor and an interface for receiving instructions and transmitting them to the at least one processor. The at least one processor executes the instructions to cause the electronic device to perform the aforementioned image processing method. The chip system may be a modem processor or a system-on-chip (SOC) including a modem processor, and the aforementioned image processing method may be implemented by a modem processor.

[0194] Some embodiments of this application also provide a computer-readable storage medium including instructions that, when executed on an electronic device, cause the electronic device to perform the above-described image processing method. Specific implementation details are provided above and will not be repeated here.

[0195] Some embodiments of this application also provide a computer program product containing instructions that, when run on an electronic device, cause the electronic device to perform the image processing method described above. The computer program product includes computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the image processing method as described in the embodiments of this application. Specific implementation details can be found in the foregoing description and will not be repeated here.

[0196] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0197] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

[0198] In summary, the above description is merely an embodiment of the technical solution of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made based on the disclosure of this application should be included within the scope of protection of this application.

Claims

1. An image processing method, characterized in that, The method includes: Based on the first and second images of the target scene, determine the camera's internal and external parameters and scene point cloud information of the target scene, wherein the first and second images have different shooting angles. Based on the camera's intrinsic and extrinsic parameters and the scene point cloud information, determine the first attribute information of the scene Gaussian sphere; Based on the first attribute information, the reference shooting angle, and the reference optical center position, a target depth image of the scene Gaussian sphere is determined; the reference shooting angle is different from the shooting angles of the first image and the second image; the target depth image includes the depth value of the intersection point of the observation ray and the scene Gaussian sphere from the reference optical center position, the endpoint position of the observation ray is the reference optical center position, and the direction of the observation ray is within the field of view of the reference shooting angle. Based on the target depth image and color ground truth information, the first attribute information is adjusted to obtain the second attribute information of the scene Gaussian sphere; Based on the reference shooting angle and the second attribute information, a target image in the target scene is generated, wherein the shooting angle of the target image is the reference shooting angle.

2. The method according to claim 1, characterized in that, The step of determining the target depth image of the scene Gaussian sphere based on the first attribute information, the shooting angle, and the reference optical center position includes: The first attribute information, with reference to the shooting angle, the optical center position, and the depth prior image, determines the first depth image of the Gaussian sphere of the scene; Edge extraction processing is performed on the first depth image to determine the non-edge regions of the first depth image; The non-edge regions of the first depth image are subjected to depth regularization constraint processing to obtain the target depth image.

3. The method according to claim 2, characterized in that, The step of determining the first depth image of the scene's Gaussian sphere based on the first attribute information, referencing the shooting angle, the reference optical center position, and the depth prior image includes: The first attribute information and the reference shooting angle are input into the multilayer sensing network to obtain the first distance value output by the multilayer sensing network. The first distance value is the distance between the intersection point and the reference optical center position. Based on the reference optical center position and the first distance value, first position information is determined, which is the position information of the intersection point of the observation ray and the scene Gaussian sphere; Based on the first location information and the first distance value, a second distance value is determined; wherein, the second distance value is the distance between the projection point of the intersection point on the camera orientation and the reference optical center position; the camera orientation is determined based on the camera's intrinsic and extrinsic parameters; Based on the second distance value, determine the first depth value of the Gaussian sphere in the scene; Based on the first depth value, a first depth image of the Gaussian sphere of the scene is determined.

4. The method according to claim 2 or 3, characterized in that, The method further includes: Perform monocular depth estimation on the monocular image to determine the first monocular depth image corresponding to the monocular image, wherein the monocular image is either the first image or the second image; The first monocular depth image is physically scaled and registered based on a reference sparse depth image to obtain a second monocular depth image. The reference sparse depth map and the second monocular depth image are fused to obtain the depth prior image.

5. The method according to claim 4, characterized in that, The method further includes: Determine the monocular scene point cloud information of the monocular image; The reference sparse depth image is determined based on the monocular scene point cloud information of the monocular image.

6. The method according to any one of claims 1-5, characterized in that, The first attribute information includes first geometric attribute information and first color attribute information; The step of adjusting the first attribute information based on the target depth image and color ground truth information to obtain the second attribute information of the scene Gaussian sphere includes: Based on the target depth image, the first geometric attribute information is adjusted to obtain the second geometric attribute information; Based on the color truth information, the first color attribute information is adjusted to obtain the second color attribute information; Determine the second attribute information of the Gaussian sphere in the scene, the second attribute information including the second geometric attribute information and the second color attribute information.

7. The method according to claim 6, characterized in that, The first geometric attribute information includes one or more of the following: size information, orientation information, sphere center position information, and opacity; the first color attribute information includes color information.

8. The method according to any one of claims 1-7, characterized in that, The method further includes: The color truth information is determined based on the color information of the first image and / or the second image.

9. An electronic device comprising a memory, one or more processors, multiple application programs, and one or more programs; wherein, The one or more programs are stored in the memory; characterized in that, when the one or more processors execute the one or more programs, the electronic device causes the electronic device to implement the method as described in any one of claims 1-8.

10. A chip system, characterized in that, The chip system includes at least one processor, a memory, and an interface circuit. The memory, the interface circuit, and the at least one processor are interconnected via circuits. The at least one memory stores program instructions. When the program instructions are executed by the processor, the chip system performs the method of any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1-8.

Images