Image processing method, apparatus and medium product

Abstract

The application discloses an image processing method, device and equipment and a medium product. The method comprises the following steps: acquiring a panoramic image, and mapping the panoramic image to a three-dimensional model surface; acquiring a three-dimensional space position of each pixel point in the panoramic image, the three-dimensional space position being used for indicating a position of the pixel point in the three-dimensional model surface; encoding the three-dimensional space position of each pixel point in the panoramic image to obtain position encoding information; and calling a graph depth prediction model to predict depth information of the panoramic image according to the position encoding information. It can be seen that the application can realize end-to-end depth prediction of the panoramic image, and improves the efficiency and accuracy of panoramic image depth prediction.

Description

Technical Field

[0001] This application relates to the field of Internet technology, specifically to the field of computer technology, and in particular to an image processing method, device, and medium product. Background Technology

[0002] In recent years, depth estimation (prediction) techniques for panoramic images have become an important research direction in the field of computer vision. Traditional panoramic depth estimation methods require splitting the panoramic image into several single-view images, performing depth prediction on each single-view image separately, and then recursively fusing the depth prediction results of each single-view image to achieve depth estimation and obtain the depth information of the panoramic image. It is evident that traditional panoramic depth estimation methods cannot achieve end-to-end depth prediction for panoramic images. They require operations such as panoramic image splitting and fusing multiple depth prediction results, which consume a significant amount of time, resulting in a long prediction process and low efficiency. Furthermore, this method is prone to instability in panoramic depth prediction results due to potential differences in the quality and consistency of depth prediction results among the various single-view images, making it difficult to guarantee the accuracy of the final panoramic depth information. Summary of the Invention

[0003] This application provides an image processing method, apparatus, and storage product that can perform end-to-end depth prediction on panoramic images, improving the efficiency and accuracy of panoramic image depth prediction.

[0004] On one hand, embodiments of this application provide an image processing method, the method comprising:

[0005] Obtain a panoramic image and map the panoramic image onto the surface of a 3D model;

[0006] The three-dimensional spatial position of each pixel in the panoramic image is obtained, and the three-dimensional spatial position is used to indicate the position of the pixel on the surface of the three-dimensional model.

[0007] The three-dimensional spatial position of each pixel in the panoramic image is encoded to obtain position encoding information;

[0008] The depth prediction model is invoked to predict the depth information of the panoramic image based on the location encoding information.

[0009] On the other hand, embodiments of this application provide an image processing apparatus, the apparatus comprising:

[0010] An acquisition unit is used to acquire a panoramic image and map the panoramic image onto the surface of a three-dimensional model.

[0011] The acquisition unit is further configured to acquire the three-dimensional spatial position of each pixel in the panoramic image, wherein the three-dimensional spatial position is used to indicate the position of the pixel on the surface of the three-dimensional model.

[0012] The processing unit is used to encode the three-dimensional spatial position of each pixel in the panoramic image to obtain position encoding information;

[0013] The processing unit is also used to call the image depth prediction model to predict the depth information of the panoramic image based on the location encoding information.

[0014] In another aspect, embodiments of this application provide a computer device, the computer device including an input interface and an output interface, the computer device further including:

[0015] Processor and computer storage media;

[0016] The processor is adapted to implement one or more instructions, and the computer storage medium stores one or more instructions, which are adapted to be loaded by the processor and executed by the aforementioned image processing method.

[0017] In another aspect, embodiments of this application provide a computer storage medium storing one or more instructions adapted for loading and executing the aforementioned image processing method by a processor.

[0018] In another aspect, embodiments of this application provide a computer program product comprising one or more instructions; when one or more instructions in the computer program product are executed by a processor, they implement the image processing method mentioned above.

[0019] This application embodiment can acquire a panoramic image and map it onto the surface of a 3D model to accurately obtain the 3D spatial position of each pixel in the panoramic image. The 3D spatial position of each pixel in the panoramic image is then encoded to obtain position encoding information. This information is then used to call a depth prediction model to predict the depth information of the panoramic image. This allows the depth prediction model to correctly identify the perspective relationships between pixels and the spatial hierarchy of the panoramic image based on the 3D spatial positions of each pixel contained in the position encoding information. This avoids spatial distortion and geometric inconsistencies in the depth prediction process, thereby improving the accuracy of depth prediction and resulting in better visual effects and spatial consistency in the predicted depth information, thus enhancing the information quality of the depth information. Furthermore, by inputting the entire panoramic image into the image depth prediction model for depth prediction, this embodiment of the application can achieve end-to-end depth prediction of the panoramic image without performing panoramic image splitting and depth prediction result fusion during the prediction process. This not only improves the efficiency of panoramic image depth prediction, but also avoids the influence of the lack of consistency between the depth prediction results of various single-view images on the panoramic image depth prediction effect, thereby improving the stability and accuracy of the panoramic image depth prediction effect. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1a This is a schematic diagram illustrating the principle of an image processing method provided in an embodiment of this application;

[0022] Figure 1b This is a logical schematic diagram of applying an image processing method during the model training phase, provided in an embodiment of this application.

[0023] Figure 1c This is a logical diagram illustrating the application of an image processing method in the model inference stage, as provided in an embodiment of this application.

[0024] Figure 1d This is a schematic diagram illustrating the effect of an image processing method provided in an embodiment of this application;

[0025] Figure 1e This is a schematic diagram of a visual image provided in an embodiment of this application;

[0026] Figure 2This is a schematic flowchart of an image processing method provided in an embodiment of this application;

[0027] Figure 3a This is a schematic diagram of a rotational position encoding provided in an embodiment of this application;

[0028] Figure 3b This is a schematic diagram illustrating panoramic image depth prediction based on location coding information, provided in an embodiment of this application.

[0029] Figure 3c This is a schematic diagram illustrating another panoramic image depth prediction based on location coding information provided in an embodiment of this application;

[0030] Figure 3d This is a schematic diagram illustrating another method for panoramic image depth prediction based on location coding information provided in this application embodiment;

[0031] Figure 4 This is a flowchart illustrating another image processing method provided in an embodiment of this application;

[0032] Figure 5a This is a logical schematic diagram of generating simulated panoramic depth data based on a monocular perspective view, provided in an embodiment of this application.

[0033] Figure 5b This is a schematic diagram illustrating a method for predicting a target depth map based on location encoding information, provided in an embodiment of this application.

[0034] Figure 5c This is a schematic diagram of a depth prediction model based on depth loss value optimization provided in an embodiment of this application;

[0035] Figure 5d This is a schematic diagram of a depth prediction model based on normal loss value optimization provided in an embodiment of this application;

[0036] Figure 5e This is a schematic diagram of an optimized map depth prediction model based on depth loss value and normal loss value provided in an embodiment of this application;

[0037] Figure 6 This is a schematic diagram of the structure of an image processing device provided in an embodiment of this application;

[0038] Figure 7 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0039] To facilitate understanding of the specific implementation of the technical solutions in the embodiments of this application, the key technical terms involved in the embodiments of this application are introduced below:

[0040] I. Panoramic View

[0041] A panorama, also known as a panoramic image, is a wide-angle or ultra-wide-angle image (i.e., a viewing angle greater than a threshold). In other words, a panorama is an image with an extremely wide field of view. For a given scene, a panorama can capture a much wider range of scene content than a traditional image (ordinary image), typically covering a 360-degree horizontal field of view and even a 180-degree vertical field of view. Therefore, compared to traditional images, panoramas can comprehensively and completely record all scene content from a full 360-degree perspective. Through this 360-degree panoramic view, it can present details within the scene from all angles, providing users with a more immersive and lifelike experience. It plays a crucial role, especially in virtual tourism, game scenarios, and film production, not only enhancing visual expressiveness but also greatly improving user engagement and realism, making applications in these fields more vivid and interactive. As can be seen, panoramic images, as an image type that can capture and display a wide field of view, can be widely used in many fields such as virtual reality (VR), augmented reality (AR), mapping, video games, and model training.

[0042] II. Depth Estimation of Panoramic Images

[0043] Depth estimation of panoramic images, also known as depth prediction, is a technique for analyzing panoramic images to obtain their depth information. This technique serves as a key technology for converting panoramic images into realistic three-dimensional (3D) space. The depth information of a panoramic image is information indicating the depth value of each pixel. It can be an image or text, without limitation. When the depth information is an image, it is called a depth map, and the value of each pixel in the depth map represents its depth value. The depth value of any pixel refers to the distance between that pixel and the camera, which is a component used to present the panoramic view from various perspectives.

[0044] As can be seen, depth estimation of panoramic images can yield depth information, allowing for the accurate reconstruction of the 3D structure within the panoramic image. This enables further 3D scene reconstruction based on 3D representations (such as meshes and 3D Gaussian methods). In practical applications, panoramic depth estimation technology is significant across multiple fields, including virtual reality, augmented reality, mapping, autonomous driving, and 3D modeling. For instance, in virtual reality and augmented reality, accurate depth information is obtained through depth estimation, forming the basis for realistic interaction and environmental perception. Similarly, in mapping and navigation, accurate depth information helps construct more precise 3D geographic data. Furthermore, in film production and game development, accurate depth information enhances the realism and interactivity of virtual worlds.

[0045] As described above, panoramic images are characterized by their ability to display scene information from a 360-degree perspective, providing users with a more immersive experience. However, since a panoramic image is obtained by unfolding a three-dimensional model surface (such as a sphere), it possesses a unique three-dimensional model surface structure (such as a spherical structure). Given the characteristics of panoramic images, such as their three-dimensional model surface structure, omnidirectional viewpoint, and complex viewpoint distortion, depth estimation (depth prediction) of panoramic images has always been a challenging problem in the field of computer vision. How to effectively remove image distortion while ensuring depth prediction accuracy, so that the predicted depth information has good visual effects and spatial consistency, remains a significant technical challenge. Therefore, this application provides a more accurate, efficient, and reliable technical solution to address the problem of panoramic image depth prediction. This solution achieves more efficient and accurate panoramic image depth prediction and removes image distortion during the depth prediction process, resulting in predicted depth information with better visual effects and spatial consistency, thereby improving the quality of the depth information in panoramic images.

[0046] Specifically, the technical solutions provided in this application may include at least the following technical points:

[0047] (1) End-to-end method design: This application embodiment designs an end-to-end image depth prediction model (i.e., a model with panoramic image depth prediction capability), and uses this image depth prediction model to perform end-to-end prediction of panoramic images. The so-called end-to-end depth prediction means that: the entire panoramic image is directly input into the image depth prediction model through the input end of the image depth prediction model, and the image depth prediction model automatically predicts the depth value of each pixel in the panoramic image to generate the depth information of the panoramic image, and outputs the depth information of the panoramic image through the output end of the image depth prediction model. It can be seen that by performing end-to-end depth prediction on the panoramic image, there is no need to perform complex operations such as panoramic image splitting and depth prediction result fusion during the depth prediction process. This not only saves a lot of time consumed by these complex operations, thereby improving the efficiency of panoramic image depth prediction, but also avoids the influence of the lack of consistency between the depth prediction results of various single-view images on the panoramic image depth prediction effect, thereby improving the stability and accuracy of the panoramic image depth prediction effect.

[0048] (2) Model Design Based on Prior Knowledge of Panoramic Image Structure: Considering the unique 3D model surface structure (such as a spherical structure) of panoramic images, this application proposes a position encoding strategy based on 3D model surface perception and introduces it into the image depth prediction model. The position encoding strategy based on 3D model surface perception refers to encoding the position (i.e., 3D spatial position) of each pixel in the panoramic image on the 3D model surface. Therefore, by introducing the position encoding strategy based on the 3D model surface into the image depth prediction model, the model can combine the prior knowledge of the panoramic image structure to predict depth, thereby solving the problems of spatial distortion and geometric inconsistency that occur during the depth prediction process of panoramic images. This improves the accuracy of panoramic image depth prediction, resulting in better visual effects and spatial consistency in the predicted depth information, thus enhancing the information quality of the depth information.

[0049] By combining the two aforementioned technical aspects, this application proposes an image processing method to achieve end-to-end depth prediction of panoramic images, thereby improving the efficiency and accuracy of panoramic image depth prediction methods. See also... Figure 1aAs shown, the general principle of the image processing method proposed in this application is as follows: A panoramic image is acquired, and position encoding information is generated using a position encoding strategy based on 3D model surface perception. Specifically, the panoramic image can be mapped onto the surface of a 3D model to obtain the 3D spatial position of each pixel in the panoramic image, thereby encoding the 3D spatial position of each pixel to obtain position encoding information. Further, the position encoding information and the panoramic image can be input into an image depth prediction model, enabling the image depth prediction model to predict the depth of the panoramic image based on the position encoding information. This allows the image depth prediction model to correctly identify the perspective relationships between pixels and the spatial hierarchy of the panoramic image based on the 3D spatial positions of each pixel contained in the position encoding information during the depth prediction process. This reduces the impact of spatial distortion in the panoramic image on depth prediction, thereby improving the accuracy of panoramic image depth prediction and resulting in predicted depth information with better visual effects and spatial consistency, thus improving the information quality of the depth information.

[0050] In specific implementations, the image processing method proposed in this application can be executed by a computer device, which can be a terminal or a server; alternatively, the image processing method can be executed jointly by a terminal and a server, without limitation. The terminal can be a smartphone, computer (such as a tablet, laptop, or desktop computer), smart wearable device (such as a smartwatch or smart glasses), smart voice interaction device, smart home appliance (such as a smart TV), vehicle terminal, or aircraft, etc.; the server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms, etc.

[0051] It should be noted that the application scenarios of the above image processing methods are not limited in the embodiments of this application.

[0052] For example, the image processing method described above can be applied to the model training stage of a graph depth prediction model. In this case, the aforementioned panoramic image can be any sample panoramic image in the training dataset of the graph depth prediction model. That is, a computer device (e.g., a server) can obtain sample panoramic images from the training dataset of the graph depth prediction model and perform the image processing method described above on those images. Specifically, it can generate position encoding information for the sample panoramic image using a position encoding strategy based on 3D model surface perception, call the graph depth prediction model to predict the depth of the sample panoramic image based on this position encoding information, and then train the graph depth prediction model based on the predicted depth information. Figure 1b As shown.

[0053] For example, the image processing method described above can also be applied to the model inference stage of the image depth prediction model; in this case, the panoramic image mentioned above can be the target panoramic image input by the user. That is, the terminal can obtain the target panoramic image input by the user and send it to the server. The server then performs the image processing method described above on the target panoramic image, that is, it uses a position encoding strategy based on the surface perception of the 3D model to generate position encoding information of the target panoramic image, calls the image depth prediction model to perform depth prediction on the target panoramic image based on the position encoding information, obtains the depth information of the target panoramic image, and then returns the depth information to the terminal, such as... Figure 1c As shown. It should be noted that, in the scenario of applying the image processing method to the model inference stage of the image depth prediction model, as an optional implementation, the image processing method can also be executed by the terminal, or jointly by the terminal and the server; there is no limitation on this.

[0054] Practice has proven that the image processing method proposed in this application has the following advantages:

[0055] (1) Highly efficient panoramic depth prediction: An end-to-end model architecture is adopted to directly predict the depth of panoramic images, avoiding the complex process of splitting panoramic images into multiple single-view images for depth prediction in traditional methods. This not only improves the processing speed and efficiency of panoramic depth prediction, making it applicable to various scenarios, especially applications with high real-time requirements, such as virtual reality, augmented reality, and autonomous driving; but also avoids the influence of the lack of consistency between the depth prediction results of various single-view images on the panoramic depth prediction effect, thereby improving the stability and accuracy of the panoramic depth prediction effect.

[0056] (2) Accurate panoramic depth prediction: By introducing position coding based on surface perception of 3D model into the panoramic depth prediction model, the panoramic depth prediction model can effectively reduce spatial distortion in the panoramic image based on the position coding information when predicting the depth of the panoramic image. This solves the problems of geometric distortion and spatial inconsistency in panoramic depth prediction, thereby ensuring that the predicted depth information has high geometric consistency and improving the accuracy of panoramic depth prediction. This is especially important for depth estimation in the edge areas and complex environments of panoramic images.

[0057] In summary, the image processing method proposed in this application can significantly improve the efficiency, accuracy, and stability of panoramic image depth prediction. To more intuitively demonstrate the effect of this image processing method, Figure 1d A simple demo (image example) is shown, which, from top to bottom, includes the input panoramic image, the depth information (depth map) generated by the image processing method, and the normal map calculated based on the depth map. To intuitively assess its geometric accuracy, this embodiment also performs mesh reconstruction on the panoramic image based on the generated depth information. The external and internal visualization images of the reconstructed mesh can be found in [reference needed]. Figure 1e As shown. Based on Figures 1d-1e As can be seen, the image processing method proposed in this application not only improves the accuracy of depth prediction, but also effectively solves the shortcomings of traditional methods in multi-view fusion and spatial distortion, greatly improving the quality of depth prediction. Its technical advantages can be widely applied in fields such as virtual reality, augmented reality, map drawing, and film and television production, providing these industries with more reliable and efficient technical support.

[0058] It is worth emphasizing that in the embodiments of this application, if user information and other related data are involved, when any method embodiment proposed in the embodiments of this application is applied to a specific product or technology, such related data is collected with the user's permission or consent, and the collection, use and processing of such data comply with the relevant laws, regulations and standards of the relevant regions.

[0059] Based on the above description, the following section uses a computer device as the execution subject as an example, combined with... Figure 2 The flowchart shown illustrates the specific implementation process of the image processing method proposed in the embodiments of this application. Please refer to... Figure 2 As shown, the method can generally include the following steps S201-S204:

[0060] S201, acquire the panoramic image and map the panoramic image onto the surface of the 3D model.

[0061] In this embodiment, the panoramic image can be a target panoramic image obtained during the model inference stage (i.e., the panoramic image to be used for depth prediction), or it can be a sample panoramic image obtained during the model training stage (i.e., the panoramic image used for model training), without limitation. Furthermore, when the panoramic image is a sample panoramic image, it can be a real panoramic image from publicly available panoramic depth data on the Internet; or, considering the scarcity of high-quality panoramic depth data, which poses a challenge to training a high-quality image depth prediction model, while high-quality data is relatively abundant in monocular perspective depth estimation, this embodiment can also construct a sample panoramic image based on a monocular perspective. That is, the sample panoramic image can be a panoramic image generated based on a monocular perspective, or it can be a panoramic image generated based on a monocular perspective in this embodiment, without limitation. A monocular perspective, also known as a single-view perspective, refers to an image obtained by shooting or rendering from a certain field of view using a single camera. It conforms to the geometric laws of perspective projection, causing the size and shape of objects in the image to appear larger when closer and smaller when farther from the camera.

[0062] The surface of a 3D model refers to the surface of a 3D model, which is a stereoscopic model constructed in a digital 3D space. If a human eye (or camera) is placed at the center of the 3D model, the human eye's (or camera's) field of view within the 3D model can include 360 ​​degrees horizontally and 180 degrees vertically. This field of view can simulate the omnidirectional view corresponding to a panoramic image. Therefore, the generation and display of panoramic images both depend on the 3D model, and the depth value of each pixel within it is closely related to the position of that pixel on the 3D model surface. Based on this, a panoramic image can be mapped onto the 3D model surface to determine the 3D spatial position of each pixel in the panoramic image (i.e., its position on the 3D model surface). This allows for depth prediction of the panoramic image based on these 3D spatial positions, improving the accuracy of depth prediction.

[0063] It should be noted that the specific selection of the three-dimensional model in this application embodiment is not limited. For example, the three-dimensional model can be a sphere, or a cube, cylinder, ellipsoid, or other three-dimensional model. Considering that compared to other three-dimensional models such as cubes, cylinders, and ellipsoids, a sphere can more naturally represent a 360-degree omnidirectional view, and is more suitable for simulating the human eye's (or camera's) perspective experience in three-dimensional space, this application embodiment can preferably use a sphere as the three-dimensional model. In this case, the surface of the three-dimensional model can be called a spherical surface, and the panoramic image can have a spherical structure. In addition, this application embodiment does not limit the specific method of projecting the panoramic image onto the surface of the three-dimensional model. For example, if the panoramic image is obtained by projecting the image on the surface of the three-dimensional model (such as a sphere) onto a two-dimensional plane based on equirectangular projection, then inverse equirectangular projection can be used to project the panoramic image in reverse onto the surface of the three-dimensional model (such as a sphere).

[0064] S202, obtain the three-dimensional spatial position of each pixel in the panoramic image. The three-dimensional spatial position is used to indicate the position of the pixel on the surface of the three-dimensional model.

[0065] Specifically, computer equipment can select a point (such as the center point) from the 3D model as the origin of coordinates and establish a 3D coordinate system based on this origin. Then, based on the positional relationship between the location of each pixel in the panoramic image on the surface of the 3D model and the origin of coordinates, the 3D spatial position of the corresponding pixel can be determined. Understandably, when the 3D model is a sphere, the surface of the 3D model is a sphere, and the 3D coordinate system can be a spherical coordinate system. The 3D spatial position of any pixel can be spherical coordinates, which specifically may include pitch and yaw angles. When the 3D model is a cube, cylinder, ellipsoid, or other shaped solid model, the surface of the 3D model is a cube face, cylinder face, or ellipsoid face. The 3D coordinate system can be a spatial rectangular coordinate system (i.e., a coordinate system composed of three mutually perpendicular coordinate axes (i.e., the X-axis, Y-axis, and Z-axis)). The 3D spatial position of any pixel can be 3D coordinates, which specifically may include X, Y, and Z values.

[0066] S203, the three-dimensional spatial position of each pixel in the panoramic image is encoded to obtain position encoding information.

[0067] In one specific implementation, when the surface of the 3D model is spherical, the 3D spatial position of any pixel includes pitch and yaw angles. In this case, step S203 can be implemented as follows: using the pitch and yaw angles of each pixel in the panoramic image, an initial position vector for each pixel is constructed. The initial position vector of any pixel is obtained by concatenating the pitch and yaw angles of the corresponding pixel. Further, a Rotary Positional Encoding (RoPE) algorithm can be used to encode the initial position vector of each pixel in the panoramic image to obtain position encoding information, such as... Figure 3a As shown in the figure. Among them, rotational position encoding is a position encoding method based on rotation mechanism, which can ensure that the encoding results of each pixel have positional consistency during the rotation of the sphere, thereby ensuring spatial consistency among each pixel.

[0068] As can be seen, this application proposes a 3D Spherical Rotary Positional Encoding strategy based on spherical perception when the surface of the 3D model is spherical, to better adapt to the spherical structure of the panoramic image. This strategy encodes the 3D spatial position of each pixel in the panoramic image. Its core is based on the spherical structure of the panoramic image, generating positional encoding information based on the 3D spatial position of each pixel. Specifically, to better utilize the prior structural knowledge of the panoramic image, this strategy maps the panoramic image onto the surface of the 3D model to derive the 3D spatial position of each pixel in the panoramic image, such as pitch and yaw angles. These two angle parameters can accurately represent the position of pixels in the panoramic image, so rotational positional encoding can be performed on these angle parameters. This rotational positional encoding can effectively convert the spatial information of each pixel into a format that the image depth prediction model can process, and ensure that the encoded positional encoding information has positional consistency. This helps to reduce the impact of spatial distortion and warping of the panoramic image on the depth prediction when performing depth prediction based on this positional encoding information.

[0069] Because panoramic images exhibit spatial distortion and warp, the depth values ​​of pixels in edge regions are easily affected by these distortions, leading to significant instability and inaccuracy. This impacts the geometric consistency of the predicted depth information. Therefore, reducing the influence of spatial distortion and warp on panoramic depth prediction through positional encoding can improve the stability of depth prediction (especially the stability of depth value prediction for pixels in edge regions), thereby effectively enhancing the geometric consistency and accuracy of the predicted depth information. It is evident that the advantage of spherical-aware rotational positional encoding lies in its full consideration of the spherical characteristics of panoramic images and its use of this spatial positional information to reduce errors caused by spherical projection distortion. Unlike traditional 2D image depth estimation methods, spherical-aware rotational positional encoding significantly improves the geometric consistency of panoramic depth estimation through precise encoding of the spherical space, particularly in the edge regions of panoramic images, where large errors often occur due to the limitations of traditional methods. In other words, the introduction of rotational position coding based on spherical perception effectively avoids spatial distortion caused by factors such as viewpoint differences and projection distortion, making the depth prediction results more stable. In the depth prediction of panoramic images, especially for edge regions, traditional methods may exhibit significant geometric distortion, while rotational position coding based on spherical perception can accurately process the depth information of these regions, maintaining the geometric consistency and high accuracy of the entire image.

[0070] In another specific implementation, regardless of whether the surface of the 3D model is spherical, step S203 can be implemented as follows: A first-order Taylor expansion position encoding method is used to encode the 3D spatial position of each pixel in the panoramic image, obtaining the position code of each pixel. This position code is then used to construct the position encoding information of the corresponding panoramic image. The core idea of ​​the first-order Taylor expansion position encoding method is: assuming a function f related to the 3D spatial position, a first-order Taylor expansion is performed on it at a reference point, and the expanded expression is used as the position code for that 3D spatial position. The expression obtained through the first-order Taylor expansion includes a constant term (the value of function f at the reference point) and a first-order partial derivative term (the result obtained by multiplying the rate of change of function f at the reference point by the offset of the 3D spatial position relative to the reference point).

[0071] In this context, the positional encoding of each pixel generated using the first-order Taylor expansion method can include: the value of a function related to the 3D spatial position of the corresponding pixel at a reference point, the rate of change of this function at the reference point, and the positional offset of the 3D spatial position relative to the reference point. Therefore, the positional encoding information constructed from the positional encoding of each pixel can contain information such as the positional offset of each pixel. This positional encoding information can be used to capture local positional changes of pixels, thereby ensuring the positional continuity of each pixel. This helps to ensure the edge continuity of the panoramic image when performing depth prediction based on this positional encoding information, thus improving the edge consistency of the predicted depth information.

[0072] S204, invoke the image depth prediction model to predict the depth information of the panoramic image based on the location encoding information.

[0073] In one specific implementation, the graph depth prediction model may include a feature extraction network and a depth predictor, both of which can be composed of one or more network layers. In this case, step S204 may be implemented as follows: The feature extraction network is invoked to extract features from the panoramic image based on location encoding information to obtain panoramic image features; the depth predictor is invoked to predict the depth of the panoramic image based on the panoramic image features (i.e., predict the depth value of each pixel in the panoramic image) to obtain the depth information of the panoramic image, such as... Figure 3b As shown, this implementation method can perform feature extraction on the panoramic image based on the location encoding information to obtain panoramic image features containing pixel-level geometric information. These panoramic image features can then be used to predict depth information, thus improving the efficiency and accuracy of panoramic image depth prediction.

[0074] In another specific implementation, the image depth prediction model may include a first feature extraction network, a second feature extraction network, and a depth predictor, each of which can be composed of one or more network layers. In this case, step S204 can be implemented as follows: The first feature extraction network is invoked to extract features from the panoramic image, obtaining initial image features; the second feature extraction network is invoked to extract features from the initial image features based on positional encoding information, obtaining target image features of the panoramic image; the depth predictor is invoked to predict the depth information of the panoramic image based on the target image features, such as... Figure 3c As shown.

[0075] As can be seen, this implementation method can perform feature extraction on the panoramic image twice, introducing positional encoding information in the second feature extraction process to obtain target image features containing pixel-level geometric information. This target image feature is then used to predict depth information, thereby improving the accuracy of panoramic image depth prediction. Furthermore, this implementation method focuses on extracting high-level initial image features such as image semantics in the first feature extraction, and on geometrically correcting the initial image features using positional encoding information in the second feature extraction. Compared to directly performing feature extraction on the panoramic image based on positional encoding information in a single step, this avoids the feature extraction network simultaneously learning the geometric features involved in both image features and positional encoding information. Each feature extraction focuses on only one processing step, and the second feature extraction better understands the image features of the panoramic image. This effectively improves the accuracy and stability of each feature extraction, thereby enhancing the accuracy and stability of panoramic image depth prediction.

[0076] It should be noted that the specific selection of any of the feature extraction networks mentioned above is not limited in the embodiments of this application. For example, any of the feature extraction networks mentioned above can be a convolutional neural network model, or a model built based on the ViT (VisionTransformer) model, etc., and the embodiments of this application do not limit this. Among them, the ViT model is a visual feature extraction network based on the Transformer (attention mechanism) structure; it can be seen that the embodiments of this application can utilize the powerful extensibility of the ViT model and choose it as the basic model structure to construct the image depth prediction model. For example, taking the 3D model surface structure of the panoramic image as a spherical structure as an example, the ViT model can be used as the first feature extraction network mentioned above, and position encoding can be introduced into the ViT model based on the unique spherical structure of the panoramic image to design a spherical ViT (Spheric Vision Transformer, SphereViT) model more suitable for processing panoramic image tasks as the second feature extraction network. This spherical ViT model can effectively capture different viewpoints and spatial features in the panoramic image depth prediction process, thereby improving the quality and geometric accuracy of panoramic image depth prediction. As can be seen, since traditional image processing methods cannot fully utilize the spherical structure of panoramic images, this application proposes the SphereViT model as a specially designed architecture for processing panoramic images. By introducing rotational position encoding into the SphereViT model, the SphereViT model can accurately process the three-dimensional spatial position information of each pixel, avoiding the spatial distortion and geometric inconsistency problems common in planar image processing methods.

[0077] Optionally, when the graph depth prediction model includes a first feature extraction network, a second feature extraction network, and a depth predictor, the graph depth prediction model may further include an image encoder. This image encoder encodes the panoramic image to generate an image vector adapted to the input size of the first feature extraction network. This image vector is then input into the first feature extraction network, which extracts features from the image vector to obtain the initial image features of the panoramic image. This improves the success rate of initial image feature extraction. In this case, a flowchart illustrating the process of a computer device using the graph depth prediction model to predict the depth information of a panoramic image based on location encoding information can be found in [reference needed]. Figure 3d As shown.

[0078] This application embodiment can acquire a panoramic image and map it onto the surface of a 3D model to accurately obtain the 3D spatial position of each pixel in the panoramic image. The 3D spatial position of each pixel in the panoramic image is then encoded to obtain position encoding information. This information is then used to call a depth prediction model to predict the depth information of the panoramic image. This allows the depth prediction model to correctly identify the perspective relationships between pixels and the spatial hierarchy of the panoramic image based on the 3D spatial positions of each pixel contained in the position encoding information. This avoids spatial distortion and geometric inconsistencies in the depth prediction process, thereby improving the accuracy of depth prediction and resulting in better visual effects and spatial consistency in the predicted depth information, thus enhancing the information quality of the depth information. Furthermore, by inputting the entire panoramic image into the image depth prediction model for depth prediction, this embodiment of the application can achieve end-to-end depth prediction of the panoramic image without performing panoramic image splitting and depth prediction result fusion during the prediction process. This not only improves the efficiency of panoramic image depth prediction, but also avoids the influence of the lack of consistency between the depth prediction results of various single-view images on the panoramic image depth prediction effect, thereby improving the stability and accuracy of the panoramic image depth prediction effect.

[0079] Based on the above description, this application proposes another image processing method to overcome the problems of low computational efficiency, poor geometric consistency, and limitations of training datasets in traditional methods. This image processing method enables efficient, accurate, and stable panoramic image depth prediction, greatly improving the quality and accuracy of panoramic image depth prediction. It has broad application prospects, particularly in fields such as virtual reality, augmented reality, autonomous driving, and 3D modeling.

[0080] The following example will still use a computer device to execute this image processing method, combined with... Figure 4 The flowchart shown illustrates the specific implementation process of the image processing method proposed in the embodiments of this application. Please refer to... Figure 4 As shown, the image processing method may include the following steps S401-S406:

[0081] S401, Obtain the panoramic image, which is any sample panoramic image obtained from the training dataset of the graph depth prediction model.

[0082] The training dataset for the image depth prediction model can be pre-configured for the model. In one implementation, real panoramic depth data can be obtained from the internet and used to configure the training dataset for the image depth prediction model. This real panoramic depth data may include publicly available sample panoramic images and corresponding sample depth maps from the internet. In another implementation, considering the scarcity of publicly available real panoramic depth data on the internet, which poses a challenge to training a high-quality image depth prediction model, while high-quality data is relatively abundant in monocular perspective depth estimation, this application proposes a scheme to construct simulated panoramic depth data based on monocular perspective and use this simulated panoramic depth data to configure the training dataset for the image depth prediction model. This effectively expands the training dataset for the image depth prediction model. Furthermore, to ensure the diversity and high quality of the generated training dataset, a large number of monocular perspective views from different scenes were selected as data sources to construct the simulated panoramic depth data. Since these monocular perspective views include various environmental conditions, such as indoor and outdoor scenes, different lighting conditions, different object types, and complex spatial structures, the simulated panoramic depth data constructed from these rich scene data not only covers different spatial structures but also has strong generalization ability. This allows the training dataset to cover different scenes, environmental conditions, and object types, thereby enhancing the adaptability of the image depth prediction model in different application scenarios and improving the accuracy and generalization of the image depth prediction model.

[0083] Therefore, the training dataset for the image depth prediction model can include at least one of the following: real panoramic image depth data and simulated panoramic image depth data. Real panoramic image depth data includes publicly available sample panoramic images and their corresponding depth maps from the internet; simulated panoramic image depth data includes sample panoramic images generated based on a monocular perspective view and their corresponding depth maps. In this case, when the computer device executes step S401, it can traverse each panoramic image depth data in the training dataset and obtain a panoramic image (i.e., a sample panoramic image) from the currently traversed panoramic image depth data. It is understood that if the currently traversed panoramic image depth data is real panoramic image depth data, then the obtained panoramic image is a publicly available sample panoramic image from the internet; if the currently traversed panoramic image depth data is simulated panoramic image depth data, then the obtained panoramic image is a sample panoramic image generated based on a monocular perspective view.

[0084] For example, the generation method of depth data for simulated panoramic images can be as follows: Figure 5a As shown, the process generally includes the following steps: obtaining a monocular perspective view and its corresponding monocular depth map, which includes the depth values ​​of each pixel in the monocular perspective view; projecting the monocular perspective view onto the surface of a 3D model (such as a sphere) to obtain a perspective projection image. Since only a portion of the perspective projection image contains image data from the monocular perspective view, while other areas are empty, after obtaining the perspective projection image, an image generation model can be called to generate a sample panoramic image based on the perspective projection image, ensuring that each pixel in the sample panoramic image has a value, thus improving the completeness of the sample panoramic image. Additionally, the monocular depth map can be projected onto the surface of a 3D model (such as a sphere) to obtain a sample depth map, thereby using the sample depth map and the generated sample panoramic image to construct simulated panoramic depth data. Therefore, this embodiment can generate simulated panoramic depth data based on a monocular perspective view and its corresponding monocular depth map, thereby expanding the scale of the training dataset and ensuring that the generated data has sufficient diversity and representativeness.

[0085] One specific method for projecting a monocular perspective view (or monocular depth map) onto the surface of a 3D model is as follows: Obtain the field of view of the monocular perspective view, and project the monocular perspective view (or monocular depth map) onto the surface of the 3D model based on this field of view. Specifically, this can be based on the field of view and the projection relationship (Perspective to Equiangular Projection, P2E Projection) between perspective coordinates (image coordinates of pixels) and spherical coordinates. The core of the P2E Projection method lies in accurately deriving the geometric relationship between perspective coordinates and spherical coordinates. This method effectively maps the depth information of the monocular perspective view to the spherical coordinate system of the panoramic image. This mapping not only preserves the depth information of each pixel in the perspective image but also eliminates the geometric distortion caused by traditional projection methods. As can be seen, this projection method allows for the projection of a monocular perspective view (or monocular depth map) onto the surface of a 3D model based on the field of view corresponding to the monocular perspective view. This confines the monocular perspective view (or monocular depth map) to the region corresponding to the specific field of view, preventing it from appearing in other areas of the 3D model surface (i.e., other field of view). This ensures spatial consistency of the projection results, thereby improving the accuracy of the depth data in the simulated panoramic image constructed based on the projection results. It is understood that in other implementations, the computer device may also project the monocular perspective view (or monocular depth map) directly onto the surface of the 3D model without referencing the field of view.

[0086] It should be noted that the image generation model mentioned above refers to any model capable of generating panoramic images, such as a convolutional neural network model or a diffusion model, etc., and this application embodiment does not limit this. Specifically, the diffusion model can be the DIT model (Diffusion Image Transformer), which is an image diffusion generation model based on the Transformer architecture. It can generate sample panoramic images by performing out-painting processing on the perspective projection image through the diffusion process. The diffusion process involves progressively denoising the input noisy image (i.e., performing denoising operations sequentially through multiple time steps, with each time step corresponding to one denoising operation; the so-called denoising operation refers to the removal of noise). The extrapolation of the panoramic image is the process of generating image data in the spatial region based on the existing image data in the perspective projection image. It can be seen that the DIT model can take advantage of the diffusion model to gradually recover and supplement the missing image data in the perspective projection image from the existing image data in the perspective projection image through the diffusion process, thereby extrapolating the projected partial image into a complete sample panoramic image. This method can generate panoramic images with rich details and effectively avoid blank or discontinuous areas caused by data scarcity in traditional methods.

[0087] Based on this, when the image generation model is a diffusion model, the specific implementation of generating a sample panoramic image based on the perspective projection image using the image generation model can include: obtaining the initial noise map of the diffusion model, fusing the initial noise map and the perspective projection image to obtain the target noise map of the diffusion model. Specifically, the initial noise map and the perspective projection image can be concatenated along the channel direction to obtain the target noise map of the diffusion model; where the channel direction refers to the direction formed by arranging multiple channels, and a channel is a data structure used to store data. The specific way to concatenate the initial noise map and the perspective projection image along the channel direction can be: placing the initial noise map and the perspective projection image in different channels along the direction formed by arranging multiple channels. It can be seen that the target noise map obtained through this implementation is a multi-channel noise map; where the initial noise map is located in the first channel of the target noise map, and the perspective projection image is located in the second channel of the target noise map, the first channel and the second channel are two different channels. Furthermore, a diffusion model can be invoked to progressively denoise the target noise map to generate a sample panoramic image. Specifically, the diffusion model can be directly invoked to progressively denoise the target noise map to obtain a sample panoramic image, or a panoramic image description text can be obtained, and the diffusion model can be invoked to progressively denoise the target noise map based on the panoramic image description text to generate a sample panoramic image that matches the panoramic image description text, thereby improving the accuracy of the sample panoramic image.

[0088] As described above, thanks to the scalability advantages of the DIT architecture, high-quality sample panoramic images can be generated by using it as the base model, thereby enabling the expansion and generation of panoramic image depth data. Optionally, during the progressive denoising process, an adaptive fusion strategy can be adopted (a strategy that fuses the left and right feature values ​​of the feature map generated at each time step of progressive denoising), continuously optimizing the quality and spatial consistency of the feature map during progressive denoising, thus ensuring the coherence and accuracy of the generated sample panoramic images.

[0089] S402 maps the panoramic image onto the surface of the 3D model and obtains the 3D spatial position of each pixel in the panoramic image.

[0090] S403 encodes the three-dimensional spatial position of each pixel in the panoramic image to obtain position encoding information.

[0091] In this embodiment, the surface of the three-dimensional model can be a sphere, and the three-dimensional spatial position of any pixel includes pitch and yaw angles. Based on this, when executing step S403, the computer device can use the pitch and yaw angles of each pixel in the panoramic image to construct an initial position vector for each pixel. This initial position vector can be a 2D vector, and its vector form can be (pitch angle, yaw angle). Furthermore, a rotational position encoding algorithm can be used to encode the initial position vector of each pixel in the panoramic image to obtain position encoding information.

[0092] In one implementation, a rotational position encoding algorithm can be used to directly encode the initial position vectors of each pixel in the panoramic image, obtaining a position encoding vector for each pixel, which can be a K-dimensional vector. The position encoding vectors of each pixel in the panoramic image are then combined to obtain the position encoding result of the panoramic image, which can be a P×Q×K dimensional vector, where P×Q is the size of the panoramic image. If the position encoding result matches the data size supported by the image depth prediction model during depth prediction, it can be directly used as position encoding information. If the position encoding result does not match the data size supported by the image depth prediction model during depth prediction, the position encoding result can be dimensionality-reduced (or dimensionality-increased) according to the data size to obtain position encoding information, which can be an N×M×K dimensional vector, where N×M belongs to the data size supported by the image depth prediction model during depth prediction.

[0093] For example, suppose a panoramic image contains 512×1024 pixels, and the image depth prediction model supports processing data of 40×80×512 dimensions during depth prediction. Then, P=512, Q=1024, K=512, N=40, and M=80. The computer device then uses a rotational position encoding algorithm to directly encode the initial position vector of each pixel in the panoramic image to obtain a 512×1024×512 dimensional vector as the position encoding result. This position encoding result is then dimensionality-reduced to obtain a 40×80×512 dimensional vector as the position encoding information.

[0094] In another implementation, the panoramic image can be compressed to obtain a new N×M dimensional panoramic image. This new panoramic image includes N×M new pixels, where each new pixel corresponds to at least two pixels in the panoramic image. Each new pixel is obtained by compressing the corresponding pixels. Based on the 3D spatial position of each pixel in the panoramic image, the 3D spatial positions of each new pixel in the new panoramic image are determined. The 3D spatial position of any new pixel is determined based on the 3D spatial positions of the pixels corresponding to it in the panoramic image. Furthermore, a rotational position encoding algorithm can be used to encode the 3D spatial position of each new pixel in the N×M dimensional new panoramic image, obtaining position encoding information, which can be an N×M×K dimensional vector.

[0095] For example, suppose the panoramic image still contains 512×1024 pixels, and the image depth prediction model supports processing data sizes of 40×80×512 during depth prediction. Then, P=512, Q=1024, K=512, N=40, and M=80. The computer can first compress the panoramic image to obtain a new 40×80 panoramic image, containing 40×80 new pixels. The computer can then obtain the three-dimensional spatial position of each new pixel in the 40×80-dimensional panoramic image, including pitch and yaw angles. Using a rotational position encoding algorithm, the three-dimensional spatial position of each new pixel is encoded to obtain a position encoding vector for each pixel. Each position encoding vector is a K-dimensional vector. Combining the position encoding vectors of N×M new pixels yields N×M×K-dimensional position encoding information.

[0096] S404, invoke the map depth prediction model to predict the depth information of the panoramic image based on the location encoding information. The depth information of the panoramic image includes the target depth map.

[0097] In this embodiment, the graph depth prediction model may include an image encoder, a first feature extraction network, a second feature extraction network, and a depth predictor. In this case, the graph depth prediction model can be an integrated model, i.e., the image encoder, the first feature extraction network, the second feature extraction network, and the depth predictor are integrated into one model; or, the graph depth prediction model can be a distributed model, i.e., the image encoder, the first feature extraction network, the second feature extraction network, and the depth predictor can be independent and different networks (models). The input size of the first feature extraction network and the input size of the second feature extraction network can be the same, for example, both N×M×K; or, the input size of the first feature extraction network and the input size of the second feature extraction network can be different, without limitation. It is worth emphasizing that when the first feature extraction network is a ViT model and the second feature extraction network is a spherical ViT model, the input and output of the ViT model have the characteristic of size consistency. The input and output sizes of the first feature extraction network are the same. Since the spherical ViT model is built on the ViT model, it also has the characteristics of the ViT model. Therefore, the input and output sizes of the second feature extraction network can also be the same, and the input size of the second feature extraction network is the same as the input size of the first feature extraction network.

[0098] In the specific implementation of step S404, the computer device can call an image encoder to encode the panoramic image to obtain an image vector of the panoramic image. It is understood that when the size of the panoramic image is larger than the input size (N×M×K) of the first feature extraction network, the image encoder can obtain an image vector by downsampling and encoding the panoramic image. Specifically, the target panoramic image can be divided into N×M image blocks, and each image block can be encoded to obtain a block vector for each image block. Each block vector can be a K-dimensional vector. Combining the block vectors of the N×M image blocks yields an N×M×K dimensional image vector. Further, the first feature extraction network can be called to extract features from the image vector to obtain the initial image features of the panoramic image, and the second feature extraction network can be called to extract features from the initial image features based on the positional encoding information to obtain the target image features of the panoramic image. Specifically, both the initial image features and the location encoding information can be N×M×K vectors. Therefore, when calling the second feature extraction network to extract features from the initial image features based on the location encoding information to obtain the target image features of the panoramic image, the location encoding information and the initial image features can be fused (e.g., vector concatenation or mean operation) to obtain fused image features. The second feature extraction network can then be called to extract features from the fused image features to obtain the target image features of the panoramic image. Optionally, the second feature extraction network can also be called to directly extract features from the initial image features based on the location encoding information to obtain the target image features; this is not limited. Furthermore, a depth predictor can be called to predict the depth information of the panoramic image based on the target image features. Specifically, the depth predictor can be called to upsample the target image features to predict the depth of the panoramic image and obtain the target depth map, such as... Figure 5b As shown.

[0099] S405: Obtain the sample depth map corresponding to the panoramic image from the training dataset, and perform loss calculation based on the sample depth map and target depth map corresponding to the panoramic image to obtain the model loss value.

[0100] In one specific implementation, the method for obtaining the model loss value by performing loss calculation based on the sample depth map and target depth map corresponding to the panoramic image can be as follows: Perform loss calculation based on the sample depth map and target depth map corresponding to the panoramic image to obtain a depth loss value, and directly use the depth loss value as the model loss value. It can be seen that this implementation, by using the depth loss value as the model loss value, can directly optimize the model parameters of the map depth prediction model based on the depth loss value. This ensures that the map depth prediction model learns the correct depth scale during the model optimization process, thereby improving the accuracy of panoramic image depth prediction by the map depth prediction model.

[0101] When the panoramic image is a publicly available sample panoramic image on the internet, the specific implementation method for obtaining the depth loss value by performing loss calculation based on the sample depth map and target depth map corresponding to the panoramic image can be: directly calling a preset loss function to perform loss calculation based on the sample depth map and target depth map corresponding to the panoramic image to obtain the depth loss value. When the panoramic image is a sample panoramic image generated based on a monocular perspective view, the sample depth map corresponding to the panoramic image includes a depth region, which includes: each depth value in the monocular depth map corresponding to the monocular perspective view; in this case, the specific implementation method for obtaining the depth loss value by performing loss calculation based on the sample depth map and target depth map corresponding to the panoramic image can be: extracting the depth region from the sample depth map corresponding to the panoramic image to obtain a first depth sub-image; and extracting the image region corresponding to the depth region from the target depth map (the position of the image region in the target depth map is the same as the position of the depth region in the sample depth map) to obtain a second depth sub-image; performing loss calculation based on the first depth sub-image and the second depth sub-image to obtain the depth loss value, specifically by calling a preset loss function to perform loss calculation based on the first depth sub-image and the second depth sub-image to obtain the depth loss value, such as... Figure 5c As shown. The preset loss function may include, but is not limited to: L1 loss function (minimum absolute deviation function), L2 loss function (least square error function), etc.

[0102] In another specific implementation, any panoramic depth data in the training dataset may further include: a sample normal map corresponding to the sample panoramic image; wherein, the sample normal map in the real panoramic depth data in the training dataset can be obtained from the Internet, and the generation method of the sample normal map in the simulated panoramic depth data in the training dataset may include: obtaining the monocular normal map corresponding to the monocular perspective view, the monocular normal map including the normal vector of each pixel in the monocular perspective view, the normal vector of any pixel is a vector perpendicular to the plane where the pixel is located, which can be used to indicate the direction of the pixel; projecting the monocular normal map onto the surface of the three-dimensional model to obtain the sample normal map, for example, the monocular normal map can be projected onto the surface of the three-dimensional model based on the field of view of the monocular perspective view to improve the accuracy of the sample normal map.

[0103] Based on this, the specific method for obtaining the model loss value by performing loss calculation based on the sample depth map and target depth map corresponding to the panoramic image can be as follows: obtain the sample normal map corresponding to the panoramic image from the training dataset, generate the target normal map based on the target depth map, perform loss calculation based on the sample normal map and target normal map corresponding to the panoramic image to obtain the normal loss value, and directly use the normal loss value as the model loss value. It can be seen that this implementation method, by using the normal loss value as the model loss value, can directly optimize the model parameters of the image depth prediction model based on the normal loss value. Since the normal map is the result of the local differentiation (gradient) of the depth map, it can directly reflect the geometric surface inversion of the panoramic image and is more sensitive to small surface changes of each pixel in the panoramic image. Therefore, compared with optimizing the image depth prediction model based on the depth loss value, optimizing the image depth prediction model based on the normal loss value can guide the image depth prediction model to capture details more accurately during the model optimization process, thereby improving the panoramic image depth prediction capability, accuracy, and geometric consistency of the panoramic image depth prediction.

[0104] When the panoramic image is a publicly available sample panoramic image on the Internet, the specific implementation method for performing loss calculation based on the sample normal map and target normal map corresponding to the panoramic image to obtain the normal loss value can be: directly calling a preset loss function (such as L1 loss function, L2 loss function, etc.) to perform loss calculation based on the sample normal map and target normal map corresponding to the panoramic image to obtain the normal loss value. When the panoramic image is a sample panoramic image generated based on a monocular perspective view, the sample normal map corresponding to the panoramic image includes a normal region, which includes each normal vector in the monocular normal map corresponding to the monocular perspective view. In this case, the specific implementation of loss calculation based on the sample normal map and the target normal map corresponding to the panoramic image to obtain the normal loss value can be as follows: Extract the normal region from the sample normal map corresponding to the panoramic image to obtain a first normal sub-map; and extract the image region corresponding to the normal region from the target normal map (the position of this image region in the target normal map is the same as the position of the normal region in the sample normal map) to obtain a second normal sub-map; perform loss calculation based on the first normal sub-map and the second normal sub-map to obtain the normal loss value. Specifically, a preset loss function can be called to perform loss calculation based on the first normal sub-map and the second normal sub-map to obtain the normal loss value, such as... Figure 5d As shown.

[0105] In another specific implementation, the depth data of any panoramic image in the training dataset may further include: the sample normal map corresponding to the sample panoramic image. In this case, the specific way to obtain the model loss value by performing loss calculation based on the sample depth map and the target depth map corresponding to the panoramic image can also be: performing loss calculation based on the sample depth map and the target depth map corresponding to the panoramic image to obtain the depth loss value; obtaining the sample normal map corresponding to the panoramic image from the training dataset, and generating the target normal map based on the target depth map; performing loss calculation based on the sample normal map and the target normal map corresponding to the panoramic image to obtain the normal loss value; integrating the depth loss value and the normal loss value (such as mean calculation, weighted average, or weighted summation, etc.) to obtain the model loss value. For example, one method for calculating the model loss value in this case can be found in [link to relevant documentation]. Figure 5e As shown, this method can optimize the model parameters of the map depth prediction model by combining depth loss and normal loss values. This allows the map depth prediction model to learn the correct depth scale and capture details correctly during the optimization process, thereby improving the panoramic map depth prediction capability of the map depth prediction model in multiple dimensions.

[0106] S406, Optimize the model parameters of the map depth prediction model based on the model loss value.

[0107] In practical implementation, the model parameters of the graph depth prediction model can be optimized in the direction of reducing the model loss value. "In the direction of reducing the model loss value" means optimizing the model in a way that minimizes the model loss value. Optimizing the model in this direction ensures that the model loss value generated after each optimization is less than the model loss value generated before optimization. For example, if the calculated model loss value is 0.85, then after optimizing the model parameters in the direction of reducing the model loss value, the model loss value generated by the optimized graph depth prediction model should be less than 0.85.

[0108] As mentioned above, a graph depth prediction model can include multiple networks such as a first feature extraction network (e.g., the ViT model) and a second feature extraction network (e.g., the SphereViT model). Therefore, the model parameters of the graph depth prediction model can include the model parameters of these networks. When optimizing the model parameters of the graph depth prediction model, these network parameters can be optimized using a uniform learning rate, i.e., the same learning rate can be used to optimize the model parameters of the first and second feature extraction networks. Alternatively, different learning rates can be used to optimize the model parameters of different networks, such as using a first learning rate to optimize the model parameters of the first feature extraction network and a second learning rate to optimize the model parameters of the second feature extraction network. Since the second feature extraction network introduces positional encoding information, which plays a crucial role in panoramic image depth prediction, the network parameters of the second feature extraction network can be optimized to improve the accuracy of panoramic image depth prediction. Based on this, the second learning rate can be set to be greater than the first learning rate; for example, the second learning rate can be set to be ten times the first learning rate, so that in one optimization process, the optimization range of the second feature extraction network is greater than that of the first feature extraction network.

[0109] Understandably, steps S401-S406 described above represent one training process of the graph depth prediction model. In practical applications, steps S401-S406 can be executed iteratively multiple times until the graph depth prediction model converges (e.g., the number of training iterations reaches a threshold, the model loss value no longer changes, or the change magnitude is less than a threshold). At this point, the training process for the graph depth prediction model can be terminated, resulting in a graph depth prediction model with superior performance. After obtaining a graph depth prediction model with superior performance, a model inference phase can be performed. In this inference phase, any panoramic image to be predicted for depth can be acquired, and the three-dimensional spatial position of each pixel in the panoramic image can be encoded to obtain positional encoding information. The graph depth prediction model can then use this positional encoding information to predict the depth information of the corresponding panoramic image, thereby improving the accuracy and geometric consistency of the depth information.

[0110] In summary, the image processing method proposed in this application embodiment has at least the following technical points:

[0111] (1) End-to-end method design: This application embodiment designs an end-to-end image depth prediction model and uses this image depth prediction model to achieve end-to-end prediction of panoramic images. It can be seen that the image processing method proposed in this application embodiment is an end-to-end panoramic image depth prediction method, which can avoid the complex process of splitting the panoramic image into multiple single-view perspective views for depth estimation in traditional methods, thereby improving the efficiency and accuracy of panoramic image depth prediction.

[0112] (2) Spherical Network Design Based on Panoramic Image Structural Priors: Considering the unique spherical structure of panoramic images, this application proposes a spherical-aware rotational position encoding and incorporates it into the Vision Transformer (ViT) model. To better utilize the structural prior knowledge of panoramic images, the panoramic image can be mapped onto a sphere to derive the three-dimensional spatial position of each pixel in the panoramic image, such as pitch and yaw angles. These three-dimensional spatial positions are then rotated and encoded, enabling the image depth prediction model to predict the panoramic image depth based on the rotational position encoding results. This effectively reduces spatial distortion and warping in the panoramic image, enhances the geometric consistency of the image, and makes the model more accurate and stable when processing panoramic images.

[0113] (3) Design of a SphericalViT-based Depth Estimation Framework: This application adopts the DIT model as the basic framework and combines it with the SphereViT architecture to design a depth estimation framework specifically for handling panoramic image depth estimation tasks. In this framework, the DIT model can generate sample panoramic images based on monocular perspective views using the diffusion process, and can construct sample depth maps based on the monocular depth maps corresponding to the monocular perspective views, so as to effectively avoid model estimation errors caused by scarce panoramic image depth data; while the SphereViT model can efficiently capture complex spatial features in panoramic images by utilizing the attention mechanism of Transformer, and model these features in an adaptive manner to obtain accurate image features, thereby using these image features to perform accurate panoramic image depth prediction. As can be seen, the core advantage of the depth estimation framework proposed in this application is that it can directly estimate the depth of panoramic images without splitting them into multiple single-view perspective views, thereby improving processing efficiency. Furthermore, by combining the global feature modeling capability of the ViT model and the stepwise recovery mechanism of DIT, the depth estimation framework proposed in this application can better handle the high-dimensional spatial features of panoramic images, effectively avoiding problems caused by the special structure of spherical images while ensuring high accuracy, thereby improving the accuracy, stability, and detail preservation capabilities of depth estimation.

[0114] (4) Model Training Data Scale-up: To address the scarcity of high-quality panoramic depth datasets, this application proposes a scheme for constructing panoramic depth data based on monocular perspective views and monocular depth maps. This enriches the training dataset for the image depth prediction model, thereby enhancing the accuracy and generalization of the model. Specifically, given a monocular perspective view and its corresponding field of view (FOV), the monocular perspective view can be projected onto a sphere based on the FOV and the projection relationship between perspective coordinates and spherical coordinates to obtain a perspective projection map. By calling the DIT model and performing progressive denoising, the perspective projection map is extrapolated to the panoramic image to fill the blank areas in the perspective projection map, thus generating a complete sample panoramic image. The monocular depth map corresponding to the monocular perspective view can be projected onto a sphere to obtain a sample depth map. Finally, the sample panoramic image and sample depth map are used to construct simulated panoramic depth data. This strategy effectively extends the depth map of a monocular perspective view to the spherical representation of a panoramic view, thereby significantly increasing the available training data and solving the problem of scarce high-quality panoramic depth data. This can improve the generalization ability of the depth prediction model, thus greatly improving the accuracy and reliability of panoramic depth prediction.

[0115] As can be seen, by combining the above-mentioned key technologies, the image processing method proposed in this application not only improves computational efficiency compared to traditional methods, but also enhances the quality of depth estimation and the generalization ability of the model without relying on large datasets. Furthermore, by combining spherical network design and adaptive data construction methods, the efficiency, accuracy, and stability of panoramic image depth prediction can be significantly improved.

[0116] Specifically, the image processing method proposed in this application embodiment can have the following significant beneficial effects:

[0117] (1) Efficient Depth Estimation: By introducing the SphereViT architecture and combining rotational position encoding and the DIT model, the complex process of splitting the panoramic image into multiple single-view perspectives in traditional methods is successfully avoided, achieving end-to-end depth estimation of the complete panoramic image. By directly predicting the depth of the panoramic image, not only can the processing flow be simplified, but the processing speed can also be improved, thereby significantly improving computational efficiency. This makes the method proposed in this application particularly suitable for real-time application scenarios (i.e., application scenarios with high real-time requirements), such as virtual reality, augmented reality, and autonomous driving.

[0118] (2) Improved Geometric Consistency and Accuracy: By innovatively adopting a rotational position encoding method based on the spherical structure of panoramic images, spatial distortion in panoramic images can be effectively reduced, thereby solving the problems of geometric distortion and spatial inconsistency in panoramic depth estimation, especially in the geometric distortion and spatial inconsistency existing in the image edge region. This ensures that the depth estimation results have high geometric consistency, which is particularly important for depth estimation in the edge region and complex environment of panoramic images. Furthermore, the rotational position encoding method based on the spherical structure of panoramic images can accurately encode the three-dimensional spatial position information of each pixel based on the pitch and yaw angles of each pixel, reducing the distortion caused by the projection of the spherical image and ensuring the geometric consistency and accuracy of the depth estimation results.

[0119] (3) Improve the quality of the training dataset and the generalization ability of the model: By generating panoramic depth data from the depth map of a monocular perspective view, the scale of the training dataset is effectively expanded. With the help of a large amount of high-quality panoramic depth data (including real panoramic depth data from different scenes and environments and simulated panoramic depth data generated by monocular perspective views), for example, with 136.3K high-quality panoramic depth data, of which 63.1K is real panoramic depth data used directly and 73.2K is simulated panoramic depth data constructed based on monocular perspective views, the generalization ability of the image depth prediction model in diverse scenarios is improved. This ensures that the image depth prediction model can adapt to complex and ever-changing real-world application environments, thereby improving its accuracy in panoramic depth estimation in various complex scenarios.

[0120] (4) Significantly reduced computational resource requirements: It avoids the repetitive computation of processing multiple single-view images in traditional methods, reducing the demand for computational resources, especially when processing high-resolution panoramic images. Compared with traditional depth estimation methods, the computation time of the entire depth estimation process is greatly shortened, making the image processing method proposed in this application a great advantage in applications with high real-time requirements.

[0121] (5) Enhanced model applicability and flexibility: By combining innovative technologies such as RoPE, SphereViT, and DIT, the image processing method proposed in this application can more accurately estimate the depth of panoramic images and achieve consistent and excellent performance in multiple application scenarios. Whether in complex indoor or outdoor environments or dynamically changing virtual reality scenes, this solution can provide high-quality and stable depth estimation results, and is widely applicable to fields such as virtual reality, augmented reality, mapping, and autonomous driving.

[0122] In summary, the embodiments of this application provide an efficient, accurate and stable image processing method, which has significant advantages in improving computational efficiency, depth estimation accuracy, geometric consistency and model generalization ability. It provides reliable technical support for related industries and promotes the application of depth estimation technology in multiple fields.

[0123] Based on the descriptions of the above method embodiments, this application also discloses an image processing apparatus; the image processing apparatus may be a computer program (including one or more instructions) running on a computer device, and the image processing apparatus may execute each step in any of the above method flows. Please refer to... Figure 6 The image processing device can operate the following units:

[0124] Acquisition unit 601 is used to acquire a panoramic image and map the panoramic image onto the surface of a three-dimensional model;

[0125] The acquisition unit 601 is further configured to acquire the three-dimensional spatial position of each pixel in the panoramic image, wherein the three-dimensional spatial position is used to indicate the position of the pixel on the surface of the three-dimensional model.

[0126] Processing unit 602 is used to encode the three-dimensional spatial position of each pixel in the panoramic image to obtain position encoding information;

[0127] The processing unit 602 is also used to call the image depth prediction model to predict the depth information of the panoramic image based on the location encoding information.

[0128] In one embodiment, the surface of the three-dimensional model is spherical, and the three-dimensional spatial position of any pixel includes pitch and yaw angles; correspondingly, when encoding the three-dimensional spatial position of each pixel in the panoramic image to obtain position encoding information, the processing unit 602 can be used for:

[0129] The initial position vector of each pixel is constructed using the pitch angle and yaw angle of each pixel in the panoramic image. The initial position vector of any pixel is obtained by stitching together the pitch angle and yaw angle of the corresponding pixel.

[0130] A rotational position encoding algorithm is used to encode the initial position vector of each pixel in the panoramic image to obtain position encoding information.

[0131] In another embodiment, the graph depth prediction model includes a first feature extraction network, a second feature extraction network, and a depth predictor; when the graph depth prediction model is invoked to predict the depth information of the panoramic image based on the location encoding information, the processing unit 602 can be used for:

[0132] The first feature extraction network is invoked to extract features from the panoramic image, thereby obtaining the initial image features of the panoramic image;

[0133] The second feature extraction network is invoked to extract features from the initial image features based on the location encoding information, thereby obtaining the target image features of the panoramic image.

[0134] The depth predictor is invoked to predict the depth information of the panoramic image based on the target image features.

[0135] In another embodiment, the image depth prediction model further includes an image encoder. When the first feature extraction network is invoked to extract features from the panoramic image to obtain the initial image features of the panoramic image, the processing unit 602 can be used to:

[0136] The image encoder is invoked to encode the panoramic image, thereby obtaining the image vector of the panoramic image;

[0137] The first feature extraction network is invoked to extract features from the image vector to obtain the initial image features of the panoramic image.

[0138] In another implementation, when the second feature extraction network is invoked to extract features from the initial image features based on the location encoding information to obtain the target image features of the panoramic image, the processing unit 602 can be used to:

[0139] The location encoding information and the initial image features are fused to obtain fused image features;

[0140] The second feature extraction network is invoked to extract features from the fused image to obtain the target image features of the panoramic image.

[0141] In another embodiment, the image depth prediction model is configured with a training dataset, which includes at least one of the following: real panoramic image depth data and simulated panoramic image depth data;

[0142] The real panoramic image depth data includes: publicly available sample panoramic images and corresponding sample depth maps on the Internet; the simulated panoramic image depth data includes: sample panoramic images generated based on monocular perspective views and corresponding sample depth maps.

[0143] The panoramic image is any sample panoramic image obtained from the training dataset, and the depth information of the panoramic image includes the target depth map. The method further includes:

[0144] Obtain the sample depth map corresponding to the panoramic image from the training dataset, perform loss calculation based on the sample depth map corresponding to the panoramic image and the target depth map to obtain the model loss value, and optimize the model parameters of the image depth prediction model based on the model loss value.

[0145] Processing unit 602 can also be used for:

[0146] Obtain a monocular perspective view and a monocular depth map corresponding to the monocular perspective view, wherein the monocular depth map includes the depth value of each pixel in the monocular perspective view;

[0147] The monocular perspective view is projected onto the surface of the three-dimensional model to obtain a perspective projection image, and an image generation model is called to generate a sample panoramic image based on the perspective projection image;

[0148] The monocular depth map is projected onto the surface of the 3D model to obtain a sample depth map. The sample depth map and the generated sample panoramic map are used to construct the depth data of the simulated panoramic map.

[0149] In another embodiment, the image generation model is a diffusion model. When the image generation model is invoked to generate a sample panoramic image based on the perspective projection image, the processing unit 602 can be used for:

[0150] Obtain the initial noise map of the diffusion model, and fuse the initial noise map and the perspective projection map to obtain the target noise map of the diffusion model;

[0151] The diffusion model is invoked to progressively denoise the target noise map in order to generate a sample panoramic image.

[0152] In another embodiment, the depth data of any panoramic image in the training dataset further includes: a sample normal map corresponding to the sample panoramic image; when performing loss calculation based on the sample depth map corresponding to the panoramic image and the target depth map to obtain the model loss value, the processing unit 602 can be used to:

[0153] Loss calculation is performed based on the sample depth map corresponding to the panoramic image and the target depth map to obtain the depth loss value;

[0154] Obtain the sample normal map corresponding to the panoramic image from the training dataset, generate the target normal map based on the target depth map, and perform loss calculation based on the sample normal map corresponding to the panoramic image and the target normal map to obtain the normal loss value.

[0155] The depth loss value and the normal loss value are integrated to obtain the model loss value.

[0156] In another embodiment, the processing unit 602 may be used for:

[0157] Obtain the monocular normal map corresponding to the monocular perspective view, wherein the monocular normal map includes the normal vector of each pixel in the monocular perspective view;

[0158] The monocular normal map is projected onto the surface of the three-dimensional model to obtain a sample normal map.

[0159] In another embodiment, when the panoramic image is a sample panoramic image generated based on a monocular perspective view, the sample depth map corresponding to the panoramic image includes a depth region, and the depth region includes: each depth value in the monocular depth map corresponding to the monocular perspective view;

[0160] When performing loss calculations based on the sample depth map corresponding to the panoramic image and the target depth map to obtain the depth loss value, the processing unit 602 can be used for:

[0161] The depth region is extracted from the sample depth map corresponding to the panoramic image to obtain a first depth sub-image; and the image region corresponding to the depth region is extracted from the target depth map to obtain a second depth sub-image.

[0162] The depth loss value is obtained by calculating the loss value based on the first depth sub-map and the second depth sub-map.

[0163] In another embodiment, when the panoramic image is a sample panoramic image generated based on a monocular perspective view, the sample normal map corresponding to the panoramic image includes a normal region, and the normal region includes: each normal vector in the monocular normal map corresponding to the monocular perspective view;

[0164] When performing loss calculations based on the sample normal map and the target normal map corresponding to the panoramic image to obtain the normal loss value, the processing unit 602 can be used for:

[0165] The normal region is extracted from the sample normal map corresponding to the panoramic image to obtain a first normal sub-map; and the image region corresponding to the normal region is extracted from the target normal map to obtain a second normal sub-map.

[0166] The loss value is calculated based on the first normal submap and the second normal submap to obtain the normal loss value.

[0167] According to another embodiment of this application, Figure 6The various units in the image processing apparatus shown can be individually or entirely merged into one or more other units, or some of the units can be further divided into multiple functionally smaller units. This achieves the same operation without affecting the technical effects of the embodiments of this application. The above-mentioned units are divided based on logical functions. In practical applications, the function of one unit can also be implemented by multiple units, or the function of multiple units can be implemented by one unit. In other embodiments of this application, the image processing apparatus may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented collaboratively by multiple units.

[0168] According to another embodiment of this application, a computer program (including one or more instructions) capable of performing the steps involved in any of the above methods can be run on a general-purpose computing device, such as a computer, which includes processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM), to construct a system such as... Figure 6 The image processing apparatus shown herein, and the method for implementing the embodiments of this application, are described. The computer program may be recorded on, for example, a computer-readable storage medium, loaded onto the aforementioned computing device via the computer-readable storage medium, and run therein.

[0169] It is worth noting that, in the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program with a predetermined function, which works together with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can contain a portion of the overall module or unit's functionality.

[0170] This application embodiment can acquire a panoramic image and map it onto the surface of a 3D model to accurately obtain the 3D spatial position of each pixel in the panoramic image. The 3D spatial position of each pixel in the panoramic image is then encoded to obtain position encoding information. This information is then used to call a depth prediction model to predict the depth information of the panoramic image. This allows the depth prediction model to correctly identify the perspective relationships between pixels and the spatial hierarchy of the panoramic image based on the 3D spatial positions of each pixel contained in the position encoding information. This avoids spatial distortion and geometric inconsistencies in the depth prediction process, thereby improving the accuracy of depth prediction and resulting in better visual effects and spatial consistency in the predicted depth information, thus enhancing the information quality of the depth information. Furthermore, by inputting the entire panoramic image into the image depth prediction model for depth prediction, this embodiment of the application can achieve end-to-end depth prediction of the panoramic image without performing panoramic image splitting and depth prediction result fusion during the prediction process. This not only improves the efficiency of panoramic image depth prediction, but also avoids the influence of the lack of consistency between the depth prediction results of various single-view images on the panoramic image depth prediction effect, thereby improving the stability and accuracy of the panoramic image depth prediction effect.

[0171] Based on the description of the above method and apparatus embodiments, this application also provides a computer device. Please refer to... Figure 7 The computer device includes at least a processor 701, an input interface 702, an output interface 703, and a computer storage medium 704. The processor 701, input interface 702, output interface 703, and computer storage medium 704 within the computer device can be connected via a bus or other means. The computer storage medium 704 can be stored in the computer device's memory. The computer storage medium 704 is used to store a computer program, which includes one or more instructions. The processor 701 is used to execute one or more instructions from the computer program stored in the computer storage medium 704. The processor 701 (or CPU (Central Processing Unit)) is the computing and control core of the computer device, adapted to implement one or more instructions, specifically adapted to load and execute one or more instructions to achieve a corresponding method flow or function.

[0172] In one embodiment, the processor 701 described in this application embodiment can be used to perform a series of image processing operations, specifically including: acquiring a panoramic image and mapping the panoramic image onto a three-dimensional model surface; acquiring the three-dimensional spatial position of each pixel in the panoramic image, the three-dimensional spatial position being used to indicate the position of the pixel on the three-dimensional model surface; encoding the three-dimensional spatial position of each pixel in the panoramic image to obtain position encoding information; calling an image depth prediction model to predict the depth information of the panoramic image based on the position encoding information, etc.

[0173] This application embodiment also provides a computer storage medium (Memory), which is a memory device in a computer device used to store computer programs and data. It is understood that the computer storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer storage medium provides storage space that stores the operating system of the computer device. Furthermore, the storage space also stores a computer program, which includes one or more instructions suitable for loading and execution by the processor 701. These instructions can be one or more program codes. It should be noted that the computer storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device; optionally, it can also be at least one computer storage medium located remotely from the aforementioned processor.

[0174] In one embodiment, a processor may load and execute one or more instructions stored in a computer storage medium to implement the corresponding steps in the above method embodiments; specifically, one or more instructions in the computer storage medium may be loaded and executed by the processor in the following steps:

[0175] Obtain a panoramic image and map the panoramic image onto the surface of a 3D model;

[0176] The three-dimensional spatial position of each pixel in the panoramic image is obtained, and the three-dimensional spatial position is used to indicate the position of the pixel on the surface of the three-dimensional model.

[0177] The three-dimensional spatial position of each pixel in the panoramic image is encoded to obtain position encoding information;

[0178] The depth prediction model is invoked to predict the depth information of the panoramic image based on the location encoding information.

[0179] In one implementation, the surface of the three-dimensional model is spherical, and the three-dimensional spatial position of any pixel includes pitch and yaw angles; correspondingly, when encoding the three-dimensional spatial position of each pixel in the panoramic image to obtain position encoding information, the one or more instructions can be loaded and executed by the processor:

[0180] The initial position vector of each pixel is constructed using the pitch angle and yaw angle of each pixel in the panoramic image. The initial position vector of any pixel is obtained by stitching together the pitch angle and yaw angle of the corresponding pixel.

[0181] A rotational position encoding algorithm is used to encode the initial position vector of each pixel in the panoramic image to obtain position encoding information.

[0182] In another implementation, the graph depth prediction model includes a first feature extraction network, a second feature extraction network, and a depth predictor; when the graph depth prediction model is invoked to predict the depth information of the panoramic image based on the location encoding information, one or more instructions can be loaded and executed by the processor:

[0183] The first feature extraction network is invoked to extract features from the panoramic image, thereby obtaining the initial image features of the panoramic image;

[0184] The second feature extraction network is invoked to extract features from the initial image features based on the location encoding information, thereby obtaining the target image features of the panoramic image.

[0185] The depth predictor is invoked to predict the depth information of the panoramic image based on the target image features.

[0186] In another embodiment, the image depth prediction model further includes an image encoder. When the first feature extraction network is invoked to extract features from the panoramic image and obtain the initial image features of the panoramic image, one or more instructions can be loaded and executed by the processor:

[0187] The image encoder is invoked to encode the panoramic image, thereby obtaining the image vector of the panoramic image;

[0188] The first feature extraction network is invoked to extract features from the image vector to obtain the initial image features of the panoramic image.

[0189] In another implementation, when the second feature extraction network is invoked to extract features from the initial image features based on the location encoding information to obtain the target image features of the panoramic image, the one or more instructions can be loaded and executed by the processor:

[0190] The location encoding information and the initial image features are fused to obtain fused image features;

[0191] The second feature extraction network is invoked to extract features from the fused image to obtain the target image features of the panoramic image.

[0192] In another embodiment, the image depth prediction model is configured with a training dataset, which includes at least one of the following: real panoramic image depth data and simulated panoramic image depth data;

[0193] The real panoramic image depth data includes: publicly available sample panoramic images and corresponding sample depth maps on the Internet; the simulated panoramic image depth data includes: sample panoramic images generated based on monocular perspective views and corresponding sample depth maps.

[0194] The panoramic image is any sample panoramic image obtained from the training dataset, and the depth information of the panoramic image includes the target depth map. The method further includes:

[0195] Obtain the sample depth map corresponding to the panoramic image from the training dataset, perform loss calculation based on the sample depth map corresponding to the panoramic image and the target depth map to obtain the model loss value, and optimize the model parameters of the image depth prediction model based on the model loss value.

[0196] The one or more instructions can be loaded and executed by the processor:

[0197] Obtain a monocular perspective view and a monocular depth map corresponding to the monocular perspective view, wherein the monocular depth map includes the depth value of each pixel in the monocular perspective view;

[0198] The monocular perspective view is projected onto the surface of the three-dimensional model to obtain a perspective projection image, and an image generation model is called to generate a sample panoramic image based on the perspective projection image;

[0199] The monocular depth map is projected onto the surface of the 3D model to obtain a sample depth map. The sample depth map and the generated sample panoramic map are used to construct the depth data of the simulated panoramic map.

[0200] In another implementation, the image generation model is a diffusion model. When the image generation model is invoked to generate a sample panoramic image based on the perspective projection image, the one or more instructions can be loaded and executed by the processor:

[0201] Obtain the initial noise map of the diffusion model, and fuse the initial noise map and the perspective projection map to obtain the target noise map of the diffusion model;

[0202] The diffusion model is invoked to progressively denoise the target noise map in order to generate a sample panoramic image.

[0203] In another implementation, the depth data of any panoramic image in the training dataset further includes: a sample normal map corresponding to the sample panoramic image; when performing loss calculation based on the sample depth map corresponding to the panoramic image and the target depth map to obtain the model loss value, the one or more instructions can be loaded and executed by the processor:

[0204] Loss calculation is performed based on the sample depth map corresponding to the panoramic image and the target depth map to obtain the depth loss value;

[0205] Obtain the sample normal map corresponding to the panoramic image from the training dataset, generate the target normal map based on the target depth map, and perform loss calculation based on the sample normal map corresponding to the panoramic image and the target normal map to obtain the normal loss value.

[0206] The depth loss value and the normal loss value are integrated to obtain the model loss value.

[0207] In another implementation, the one or more instructions may be loaded and executed by the processor:

[0208] Obtain the monocular normal map corresponding to the monocular perspective view, wherein the monocular normal map includes the normal vector of each pixel in the monocular perspective view;

[0209] The monocular normal map is projected onto the surface of the three-dimensional model to obtain a sample normal map.

[0210] In another embodiment, when the panoramic image is a sample panoramic image generated based on a monocular perspective view, the sample depth map corresponding to the panoramic image includes a depth region, and the depth region includes: each depth value in the monocular depth map corresponding to the monocular perspective view;

[0211] When performing loss calculations based on the sample depth map corresponding to the panoramic image and the target depth map to obtain the depth loss value, one or more instructions can be loaded and executed by the processor:

[0212] The depth region is extracted from the sample depth map corresponding to the panoramic image to obtain a first depth sub-image; and the image region corresponding to the depth region is extracted from the target depth map to obtain a second depth sub-image.

[0213] The depth loss value is obtained by calculating the loss value based on the first depth sub-map and the second depth sub-map.

[0214] In another embodiment, when the panoramic image is a sample panoramic image generated based on a monocular perspective view, the sample normal map corresponding to the panoramic image includes a normal region, and the normal region includes: each normal vector in the monocular normal map corresponding to the monocular perspective view;

[0215] When performing loss calculations based on the sample normal map and the target normal map corresponding to the panoramic image to obtain the normal loss value, one or more instructions can be loaded and executed by the processor:

[0216] The normal region is extracted from the sample normal map corresponding to the panoramic image to obtain a first normal sub-map; and the image region corresponding to the normal region is extracted from the target normal map to obtain a second normal sub-map.

[0217] The loss value is calculated based on the first normal submap and the second normal submap to obtain the normal loss value.

[0218] This application embodiment can acquire a panoramic image and map it onto the surface of a 3D model to accurately obtain the 3D spatial position of each pixel in the panoramic image. The 3D spatial position of each pixel in the panoramic image is then encoded to obtain position encoding information. This information is then used to call a depth prediction model to predict the depth information of the panoramic image. This allows the depth prediction model to correctly identify the perspective relationships between pixels and the spatial hierarchy of the panoramic image based on the 3D spatial positions of each pixel contained in the position encoding information. This avoids spatial distortion and geometric inconsistencies in the depth prediction process, thereby improving the accuracy of depth prediction and resulting in better visual effects and spatial consistency in the predicted depth information, thus enhancing the information quality of the depth information. Furthermore, by inputting the entire panoramic image into the image depth prediction model for depth prediction, this embodiment of the application can achieve end-to-end depth prediction of the panoramic image without performing panoramic image splitting and depth prediction result fusion during the prediction process. This not only improves the efficiency of panoramic image depth prediction, but also avoids the influence of the lack of consistency between the depth prediction results of various single-view images on the panoramic image depth prediction effect, thereby improving the stability and accuracy of the panoramic image depth prediction effect.

[0219] It should be noted that, according to one aspect of this application, a computer program product or computer program is also provided, comprising one or more instructions stored in a computer storage medium. A processor of a computer device reads one or more instructions from the computer storage medium and executes the one or more instructions, causing the computer device to perform the methods provided in various optional embodiments of the above-described methods. It should be understood that the above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, equivalent variations made according to the claims of this application are still within the scope of this application.

Claims

1. An image processing method, characterized in that, include: Obtain a panoramic image and map the panoramic image onto the surface of a 3D model; The three-dimensional spatial position of each pixel in the panoramic image is obtained, and the three-dimensional spatial position is used to indicate the position of the pixel on the surface of the three-dimensional model. The surface of the three-dimensional model is a sphere, and the three-dimensional spatial position of any pixel includes the pitch angle and the yaw angle. The initial position vector of each pixel is constructed using the pitch angle and yaw angle of each pixel in the panoramic image. The initial position vector of any pixel is obtained by stitching together the pitch angle and yaw angle of the corresponding pixel. A rotational position encoding algorithm is used to encode the initial position vector of each pixel in the panoramic image to obtain position encoding information; The depth prediction model is invoked to predict the depth information of the panoramic image based on the location encoding information.

2. The method as described in claim 1, characterized in that, The graph depth prediction model includes a first feature extraction network, a second feature extraction network, and a depth predictor; the step of calling the graph depth prediction model to predict the depth information of the panoramic image based on the location encoding information includes: The first feature extraction network is invoked to extract features from the panoramic image, thereby obtaining the initial image features of the panoramic image; The second feature extraction network is invoked to extract features from the initial image features based on the location encoding information, thereby obtaining the target image features of the panoramic image. The depth predictor is invoked to predict the depth information of the panoramic image based on the target image features.

3. The method as described in claim 2, characterized in that, The image depth prediction model further includes an image encoder. The step of calling the first feature extraction network to extract features from the panoramic image to obtain initial image features of the panoramic image includes: The image encoder is invoked to encode the panoramic image, thereby obtaining the image vector of the panoramic image; The first feature extraction network is invoked to extract features from the image vector to obtain the initial image features of the panoramic image.

4. The method as described in claim 2, characterized in that, The step of calling the second feature extraction network to extract features from the initial image features based on the location encoding information to obtain the target image features of the panoramic image includes: The location encoding information and the initial image features are fused to obtain fused image features; The second feature extraction network is invoked to extract features from the fused image to obtain the target image features of the panoramic image.

5. The method as described in claim 1, characterized in that, The image depth prediction model is configured with a training dataset, which includes at least one of the following: real panoramic image depth data and simulated panoramic image depth data; The real panoramic image depth data includes: publicly available sample panoramic images and corresponding sample depth maps on the Internet; the simulated panoramic image depth data includes: sample panoramic images generated based on monocular perspective views and corresponding sample depth maps. The panoramic image is any sample panoramic image obtained from the training dataset, and the depth information of the panoramic image includes the target depth map. The method further includes: Obtain the sample depth map corresponding to the panoramic image from the training dataset, perform loss calculation based on the sample depth map corresponding to the panoramic image and the target depth map to obtain the model loss value, and optimize the model parameters of the image depth prediction model based on the model loss value.

6. The method as described in claim 5, characterized in that, The method for generating the depth data of the simulated panoramic image includes: Obtain a monocular perspective view and a monocular depth map corresponding to the monocular perspective view, wherein the monocular depth map includes the depth value of each pixel in the monocular perspective view; The monocular perspective view is projected onto the surface of the three-dimensional model to obtain a perspective projection image, and an image generation model is called to generate a sample panoramic image based on the perspective projection image; The monocular depth map is projected onto the surface of the 3D model to obtain a sample depth map. The sample depth map and the generated sample panoramic map are used to construct the depth data of the simulated panoramic map.

7. The method as described in claim 6, characterized in that, The image generation model is a diffusion model, and the step of calling the image generation model to generate a sample panoramic image based on the perspective projection image includes: Obtain the initial noise map of the diffusion model, and fuse the initial noise map and the perspective projection map to obtain the target noise map of the diffusion model; The diffusion model is invoked to progressively denoise the target noise map in order to generate a sample panoramic image.

8. The method as described in claim 5, characterized in that, The depth data of any panoramic image in the training dataset also includes: the sample normal map corresponding to the sample panoramic image; The loss calculation based on the sample depth map corresponding to the panoramic image and the target depth map to obtain the model loss value includes: Loss calculation is performed based on the sample depth map corresponding to the panoramic image and the target depth map to obtain the depth loss value; Obtain the sample normal map corresponding to the panoramic image from the training dataset, generate the target normal map based on the target depth map, and perform loss calculation based on the sample normal map corresponding to the panoramic image and the target normal map to obtain the normal loss value. The depth loss value and the normal loss value are integrated to obtain the model loss value.

9. The method as described in claim 8, characterized in that, The generation method of the sample normal map in the depth data of the simulated panoramic image includes: Obtain the monocular normal map corresponding to the monocular perspective view, wherein the monocular normal map includes the normal vector of each pixel in the monocular perspective view; The monocular normal map is projected onto the surface of the three-dimensional model to obtain a sample normal map.

10. The method as described in claim 8 or 9, characterized in that, When the panoramic image is a sample panoramic image generated based on a monocular perspective view, the sample depth map corresponding to the panoramic image includes a depth region, and the depth region includes: each depth value in the monocular depth map corresponding to the monocular perspective view. The step of performing loss calculations based on the sample depth map corresponding to the panoramic image and the target depth map to obtain a depth loss value includes: The depth region is extracted from the sample depth map corresponding to the panoramic image to obtain a first depth sub-image; and the image region corresponding to the depth region is extracted from the target depth map to obtain a second depth sub-image. The depth loss value is obtained by calculating the loss value based on the first depth sub-map and the second depth sub-map.

11. The method as described in claim 8 or 9, characterized in that, When the panoramic image is a sample panoramic image generated based on a monocular perspective view, the sample normal map corresponding to the panoramic image includes a normal region, and the normal region includes: each normal vector in the monocular normal map corresponding to the monocular perspective view. The loss calculation based on the sample normal map corresponding to the panoramic image and the target normal map to obtain the normal loss value includes: The normal region is extracted from the sample normal map corresponding to the panoramic image to obtain a first normal sub-map; and the image region corresponding to the normal region is extracted from the target normal map to obtain a second normal sub-map. The loss value is calculated based on the first normal submap and the second normal submap to obtain the normal loss value.

12. An image processing apparatus, characterized in that, include: An acquisition unit is used to acquire a panoramic image and map the panoramic image onto the surface of a three-dimensional model. The acquisition unit is further configured to acquire the three-dimensional spatial position of each pixel in the panoramic image, wherein the three-dimensional spatial position is used to indicate the position of the pixel on the surface of the three-dimensional model. The surface of the three-dimensional model is a sphere, and the three-dimensional spatial position of any pixel includes the pitch angle and the yaw angle. The processing unit is used to construct an initial position vector for each pixel in the panoramic image using the pitch angle and yaw angle of each pixel. The initial position vector of any pixel is obtained by stitching together the pitch angle and yaw angle of the corresponding pixel. The unit uses a rotation position encoding algorithm to encode the initial position vector of each pixel in the panoramic image to obtain position encoding information. The processing unit is also used to call the image depth prediction model to predict the depth information of the panoramic image based on the location encoding information.

13. A computer device, comprising an input interface and an output interface, characterized in that, Also includes: Processor and computer storage media; The processor is adapted to implement one or more instructions, the computer storage medium stores one or more instructions, and the one or more instructions are adapted to be loaded by the processor and executed as described in any one of claims 1-11.

14. A computer storage medium, characterized in that, The computer storage medium stores one or more instructions, which are adapted to be loaded by a processor and executed as described in any one of claims 1-11.

15. A computer program product, characterized in that, The computer program product includes one or more instructions; when one or more instructions in the computer program product are executed by a processor, they implement the image processing method as described in any one of claims 1-11.

Images