A cross-device rendering method, device, equipment, and medium

Abstract

The present disclosure provides a cross-device rendering method, apparatus, device, and medium, aiming to solve the problem of poor generalization ability of the three-dimensional Gaussian sputtering model in the related art. The method includes: obtaining original images from different camera devices; converting the original images into intermediate images according to the mapping relationship between the original camera model corresponding to the original images and the intermediate camera model, where the intermediate images carry corresponding intermediate poses; constructing a three-dimensional Gaussian sputtering model according to the intermediate images and the corresponding intermediate poses; rendering a target intermediate image by using the three-dimensional Gaussian sputtering model according to the rendering direction of the target intermediate image, and inverse mapping the target intermediate image to the inverse mapping area corresponding to the target original image to obtain the target original image.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular, to a cross-device rendering method, apparatus, device, and medium. Background Art

[0002] Photorealistic and real-time 3D reconstruction and rendering are the focus of academic research and industry, and are applied to virtual reality (VR), augmented reality (AR), and various robotic tasks. 3D-GS (3D Gaussian Splatting) and its variants have made significant progress in these fields. 3D-GS effectively models a scene by optimizing a set of 3D Gaussians, demonstrating state-of-the-art rendering quality and speed.

[0003] However, related technologies are usually designed for pinhole camera models and are difficult to process images from different camera models, resulting in poor generalization ability. Moreover, when extended to new camera models, it is necessary to re-derive new explicit gradient flows specific to the camera model, and as the complexity of the camera model increases, the complexity of model construction also increases significantly, making the 3D-GS model construction process more cumbersome, thus severely limiting its generalization ability. Summary of the Invention

[0004] To overcome the problems in related technologies, the present disclosure provides a cross-device rendering method, apparatus, device, and medium. The technical solutions of the present disclosure are as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, a cross-device rendering method is provided, including:

[0006] Obtain original images from different camera devices;

[0007] Convert the original images into intermediate images according to the mapping relationship between the original camera model corresponding to the original images and an intermediate camera model, where the intermediate images carry corresponding intermediate poses;

[0008] Construct a 3D Gaussian splatting model according to the intermediate images and the corresponding intermediate poses;

[0009] Render a target intermediate image using the 3D Gaussian splatting model according to the rendering direction of the target intermediate image, and inverse-map the target intermediate image to the inverse-mapping region corresponding to the target original image to obtain the target original image.

[0010] Optionally, it further includes:

[0011] Initialize learnable camera embeddings by the number of original camera models; the number of original camera models is the same as the number of categories of the camera devices;

[0012] Embed the learnable camera and splice it with the target original image along the image channel direction to obtain a spliced image;

[0013] Input the spliced image into an optimization network to output a pixel-level target transformation and a restored image, where the optimization network is used to optimize the target original image;

[0014] Obtain an optimized target original image through the target transformation, the restored image, and the target original image.

[0015] Optionally, it further includes:

[0016] Filter out redundant 3D Gaussians in the 3D Gaussian sputtering model through the inverse mapping region and the rendering direction to obtain a filtered 3D Gaussian sputtering model, where the redundant 3D Gaussians are those that do not participate in generating the target original image;

[0017] Render a target intermediate image through the rendering direction using the 3D Gaussian sputtering model, including:

[0018] Based on the filtered 3D Gaussian sputtering model, allocate valid 3D Gaussians to corresponding rendering directions to obtain 3D Gaussians corresponding to each rendering direction, where the valid 3D Gaussians are those that participate in rendering the intermediate image of the rendering direction;

[0019] Render a target intermediate image through the 3D Gaussians corresponding to each rendering direction using the 3D Gaussian sputtering model.

[0020] Optionally, convert the original image into an intermediate image according to the mapping relationship between the original camera model corresponding to the original image and the intermediate camera model, including:

[0021] Determine the latitude rotation and longitude rotation of the intermediate image relative to the original pose of the original image according to the camera internal parameters of the intermediate camera model;

[0022] Determine the rotation matrix of the intermediate image relative to the original image through the latitude rotation and the longitude rotation;

[0023] Determine the intermediate pose of the intermediate image through the rotation matrix and the original pose;

[0024] Convert the original image into the intermediate image through the rotation matrix, the back-projection function corresponding to the original camera model, and the projection function of the intermediate camera model.

[0025] Optionally, the intermediate image includes a reference intermediate image with texture mapping and a to-be-optimized intermediate image without texture mapping, and it further includes:

[0026] According to the intermediate camera model, determine the adaptive mapping direction of the intermediate image to be optimized relative to the reference intermediate image, and the image transformation parameters corresponding to each adaptive mapping direction;

[0027] Determine the texture region of the reference intermediate image, and determine the intermediate pose corresponding to the reference intermediate image;

[0028] According to the texture region of the reference intermediate image and the image transformation parameters, determine the adaptive mapping images in each adaptive mapping direction;

[0029] Use the adaptive mapping images to replace the intermediate image to be optimized, and obtain the updated intermediate image to be optimized;

[0030] According to the intermediate pose corresponding to the reference intermediate image and the adaptive mapping direction, determine the intermediate pose corresponding to the updated intermediate image to be optimized;

[0031] According to the intermediate image and the corresponding intermediate pose, construct a three-dimensional Gaussian sputtering model, including:

[0032] Construct a three-dimensional Gaussian sputtering model according to the reference intermediate image and its corresponding intermediate pose and the updated intermediate image to be optimized and its corresponding intermediate pose.

[0033] Optionally, according to the intermediate image and the corresponding intermediate pose, construct a three-dimensional Gaussian sputtering model, including:

[0034] According to the intermediate image and the corresponding intermediate pose, construct a first initial three-dimensional Gaussian sputtering model;

[0035] Through the first initial three-dimensional Gaussian sputtering model, obtain a first initial target intermediate image with the same pose as the intermediate image;

[0036] Determine a first loss through the intermediate image and the first initial target intermediate image;

[0037] Adjust the parameters of the first initial three-dimensional Gaussian sputtering model through the first loss;

[0038] Adjust the parameters of the first initial three-dimensional Gaussian sputtering model multiple times according to the above steps to obtain a three-dimensional Gaussian sputtering model.

[0039] Optionally, according to the intermediate image and the corresponding intermediate pose, construct a three-dimensional Gaussian sputtering model, including:

[0040] According to the intermediate image and the corresponding intermediate pose, construct a second initial three-dimensional Gaussian sputtering model;

[0041] Obtain a second initial target original image with the same pose as the original image through the second initial three-dimensional Gaussian sputtering model;

[0042] Obtain an initial restored image and an initial target transformation corresponding to the second initial target original image through an initial optimization network;

[0043] Determine an optimized second initial target original image according to the initial restored image, the initial target transformation, and the second initial target original image;

[0044] Determine a distortion loss according to the initial restored image and the original image;

[0045] Determine a second loss through the optimized second initial target original image and the original image;

[0046] Adjust the parameters of the initial optimization network and the second initial three-dimensional Gaussian sputtering model respectively according to the distortion loss and the second loss;

[0047] Adjust the parameters of the initial optimization network and the second initial three-dimensional Gaussian sputtering model multiple times according to the above steps to obtain the optimization network and the three-dimensional Gaussian sputtering model.

[0048] According to the second aspect of the embodiments of the present disclosure, there is provided a cross-device rendering apparatus, including:

[0049] An acquisition module for acquiring original images from different camera devices;

[0050] A mapping module for converting the original image into an intermediate image according to the mapping relationship between the original camera model corresponding to the original image and the intermediate camera model, and the intermediate image carries the corresponding intermediate pose;

[0051] A construction module for constructing a three-dimensional Gaussian sputtering model according to the intermediate image and the corresponding intermediate pose;

[0052] A determination module for rendering a target intermediate image by using the three-dimensional Gaussian sputtering model according to the rendering direction of the target intermediate image, and inverse mapping the target intermediate image to the inverse mapping area corresponding to the target original image to obtain the target original image.

[0053] According to the third aspect of the embodiments of the present disclosure, there is provided an electronic device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor. When the computer program is executed by the processor, the steps of the cross-device rendering method described in the first aspect are implemented.

[0054] According to a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided. A computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, the steps of the cross-device rendering method described in the first aspect are implemented.

[0055] According to a fifth aspect of the embodiments of the present disclosure, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the steps of the cross-device rendering method described in the first aspect are implemented.

[0056] By introducing an intermediate camera model, the present disclosure can process raw images from different camera devices, enhancing the generalization ability of 3D-GS and enabling it to be applicable to more types of camera devices. Using the intermediate camera model as a bridge, the cumbersome process of separately deriving the gradient flow for each camera model is avoided, thus simplifying the entire model construction process. This method can perform rendering according to the rendering direction of the target intermediate image and inverse-map the rendering result to the inverse-mapped area of the target raw image to obtain the target raw image, flexibly supporting various rendering requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] To more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for the description of the embodiments of the present disclosure will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present disclosure. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0058] Figure 1 It is a schematic diagram of the steps of a cross-device rendering method shown in the embodiments of the present disclosure;

[0059] Figure 2 It is a schematic diagram of a cross-device rendering system shown in the embodiments of the present disclosure;

[0060] Figure 3 It is a schematic diagram of a cross-device rendering device shown in the embodiments of the present disclosure;

[0061] Figure 4 It is a schematic diagram of an electronic device shown in the embodiments of the present disclosure. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0062] The technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are some, but not all, of the embodiments of the present disclosure. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present disclosure without creative efforts fall within the scope of protection of the present disclosure.

[0063] The terms "first", "second", etc. in the specification and claims of the present disclosure are used to distinguish similar objects, rather than to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type, and the number of objects is not limited. For example, the first object can be one or multiple. In addition, "and / or" in the specification and claims means at least one of the connected objects, and the character " / " generally indicates an "or" relationship between the associated objects before and after.

[0064] To solve the problems existing in the related art, the present disclosure provides a cross-device rendering method, which can construct a camera-independent three-dimensional Gaussian sputtering model, so as to efficiently process multiple camera source images.

[0065] Figure 1 It is a schematic diagram of the steps of a cross-device rendering method shown in the embodiments of the present disclosure. According to Figure 1 as shown, the method may specifically include the following steps:

[0066] Step S11: Obtain the original images from different camera devices.

[0067] The camera devices may include various camera types, for example, fisheye cameras, panoramic cameras, etc. The original images from different camera devices refer to the images captured by different camera devices. Due to the different camera types of the camera devices, the characteristics of the captured images are also different. When different camera devices capture the same object, the information that can be captured by one captured image is different.

[0068] Step S12: Convert the original image into an intermediate image according to the mapping relationship between the original camera model corresponding to the original image and the intermediate camera model, and the intermediate image carries the corresponding intermediate pose.

[0069] When converting any original image into an intermediate image, determine the original camera model corresponding to the original image. The original camera model is related to the camera device used to capture this original image, and the original camera model corresponding to the original image can be determined through the camera type of the camera device. For example, if an original image is captured by a fisheye camera, then this original image corresponds to a fisheye camera model.

[0070] The intermediate camera model is the camera model corresponding to the intermediate image, and the intermediate camera model is unified. When converting any original image into an intermediate image, the same intermediate camera model is used. The intermediate camera model can be set according to the actual situation.

[0071] An original image can be converted into one or more intermediate images. The intermediate images have a unified image format and can provide a standardized representation. Converting the original images captured by different camera devices with different perspectives and projection rules into unified intermediate images can ensure the consistency of processing.

[0072] Each intermediate image has its corresponding intermediate pose. The intermediate pose can represent the spatial positioning of the intermediate image and provide the spatial information and orientation information of the intermediate image in three-dimensional space.

[0073] Step S13: Construct a three-dimensional Gaussian sputtering model according to the intermediate image and the corresponding intermediate pose.

[0074] After obtaining the intermediate image and its corresponding intermediate pose, use the intermediate image and its corresponding intermediate pose to construct a three-dimensional Gaussian sputtering model. Through the original images from different camera devices, a unified three-dimensional Gaussian sputtering model is established. The three-dimensional Gaussian sputtering model is represented as a set of 3D Gaussian points.

[0075] Step S14: According to the rendering direction of the target intermediate image, use the three-dimensional Gaussian sputtering model to render the target intermediate image, and inverse-map the target intermediate image to the inverse-mapping area corresponding to the target original image to obtain the target original image.

[0076] The rendering direction refers to the process of determining from which angle and direction to generate an image during the image rendering process.

[0077] The inverse-mapping area refers to a specific area in the target original image obtained by inverse-mapping the target intermediate image. The inverse-mapping area is the part of the target original image corresponding to the target intermediate image. The inverse-mapping area can ensure that the rendered target intermediate image is aligned with the finally obtained target original image and avoid distortion of the target original image.

[0078] The rendering direction and the inverse-mapping area are pre-calculated according to the mapping relationship between the original camera model and the intermediate camera model. First, the two-dimensional plane corresponding to the target original image that is finally required can be determined. After determining this two-dimensional plane, according to the mapping relationship between the original camera model and the intermediate camera model, it can be determined which intermediate images with which rendering directions have a mapping relationship with this two-dimensional plane, and which area of this two-dimensional plane these intermediate images are mapped to. A target original image can be obtained by inverse-mapping from one or more target intermediate images.

[0079] Render according to the rendering direction through the three-dimensional Gaussian sputtering model to obtain the target intermediate image. The target intermediate image is obtained by rendering through the three-dimensional Gaussian sputtering model.

[0080] After obtaining the target intermediate image, in order to transform the target intermediate image back to the space of the original image, an inverse mapping is required. The inverse mapping refers to the mapping from the target intermediate image to the target original image. By mapping the target intermediate image back to the inverse mapping region corresponding to the target original image, the target intermediate image is precisely transformed back to the original image space. After being transformed back to the original image space, the final target original image is obtained. The inverse mapping mechanism can ensure that the finally obtained target original image can maintain the geometric shape and camera characteristics of the original image.

[0081] By adopting the embodiments of the present disclosure, original images are obtained from different camera devices, and these original images from different sources are converted into a unified intermediate image through the mapping relationship between the original camera model and the intermediate camera model, ensuring the consistency and unified processing framework of the original images of different devices during the processing, solving the problems of different perspectives, field of view angles, distortions, etc. of the images captured by different camera devices, and guaranteeing the efficiency of cross-device rendering. Thus, a unified 3D Gaussian sputtering model independent of the original camera model can be constructed based on these intermediate images and the corresponding intermediate poses. During the rendering process, according to the rendering direction of the target intermediate image, the unified 3D Gaussian sputtering model is used to render the target intermediate image, and the rendered target intermediate image will be mapped to the inverse mapping region corresponding to the target original image through the inverse mapping technology, enabling the target intermediate image to be restored to the target original image.

[0082] Among them, in an optional embodiment, converting the original image into an intermediate image according to the mapping relationship between the original camera model corresponding to the original image and the intermediate camera model includes: determining the latitude rotation and longitude rotation of the intermediate image relative to the original pose of the original image according to the camera internal parameters of the intermediate camera model; determining the rotation matrix of the intermediate image relative to the original image through the latitude rotation and the longitude rotation; determining the intermediate pose of the intermediate image through the rotation matrix and the original pose; and converting the original image into the intermediate image through the rotation matrix, the back-projection function corresponding to the original camera model, and the projection function of the intermediate camera model.

[0083] The original camera model is the camera model used when shooting the original image, including the internal parameters and external parameters of the camera. The internal parameters define the internal characteristics of the camera, and these characteristics determine how the camera projects points in the three-dimensional world onto the two-dimensional image plane. Specifically, they can include information such as focal length, principal point coordinates, and distortion coefficients. The external parameters describe the position and orientation of the camera in the world coordinate system, determining how the camera shoots the same scene from different angles and positions. The external parameters can be understood as the pose of the camera.

[0084] The intermediate camera model is a standardized camera model for constructing a three-dimensional Gaussian sputtering model. The internal parameters of the intermediate camera model can be expressed by the following formula:

[0085]

[0086] In the above formula, represents the width of the reference intermediate image, represents the height of the reference intermediate image.

[0087] Through the internal parameters of the intermediate camera model, six fixed rendering directions perpendicular to each other can be determined, namely the front, back, left, right, up, and down directions. The width and height of the intermediate image corresponding to each rendering direction are equal, which is 1024 pixels. The intermediate images in the six rendering directions can be stitched into a cube, and each face of the cube is the two-dimensional plane of the corresponding intermediate image. The intermediate image can be the pinhole image corresponding to the pinhole camera.

[0088] The rendering direction corresponding to the internal parameters of the intermediate camera model can be determined as the fixed mapping direction, and the fixed mapping direction is used to convert the original image into the intermediate image. The fixed mapping direction corresponds to an intermediate image.

[0089] The original pose corresponding to the original image represents the position and orientation of the original camera model in the three-dimensional space. The pose can include the position of the camera in the three-dimensional space and the rotation angle of the camera.

[0090] It is necessary to determine the rotation transformation relationship between the original image and the intermediate image, so as to convert the original image to a specific fixed mapping direction. The intermediate image corresponding to each fixed mapping direction has a specific rotation transformation relationship relative to the original image, and the rotation transformation relationship changes the viewing angle and perspective of the original image. A corresponding rotation transformation relationship can be determined for each fixed mapping direction.

[0091] Specifically, the latitude rotation and longitude rotation of each intermediate image relative to the original pose can be determined. The latitude rotation and longitude rotation represent different dimensions of rotation. The latitude rotation refers to the angle of rotation along the horizontal direction. For example, rotating the original image along the horizontal direction by a certain angle from the camera position of the original image to obtain different latitude rotation angles. The longitude rotation refers to the angle of rotation along the vertical direction. For example, rotating the original image along the vertical direction of the image by the camera pose from the camera position of the original image to obtain different longitude rotation angles.

[0092] A rotation matrix can be determined through the latitude rotation and longitude rotation. The rotation matrix is used to represent the rotation transformation relationship of the intermediate image relative to the original image. The rotation matrix is used to describe how the original image rotates relative to the intermediate image in the three-dimensional space.

[0093] After obtaining the rotation matrix, in combination with the original pose corresponding to the original image, determine the intermediate pose corresponding to the intermediate image. The intermediate pose can be specifically obtained through the following formula:

[0094]

[0095] In the above formula, represents the intermediate pose corresponding to the intermediate image, represents the original pose corresponding to the original image, represents the rotation matrix of the intermediate image relative to the original image, where, represents the latitude rotation, represents the longitude rotation.

[0096] The rotation matrix is used to convert the coordinates in the original camera coordinate system to the coordinates in the target coordinate system. Through the rotation matrix, points in the original camera coordinate system, specifically pixel points in the original image, can be rotated into the intermediate image coordinate system. Therefore, after obtaining the rotation matrix for any fixed rendering direction, according to the rotation matrix, the back-projection function corresponding to the original camera model, and the projection function of the intermediate camera model, the original image is converted into the intermediate image in this fixed rendering direction. The intermediate image can be specifically obtained through the following formula:

[0097]

[0098] In the above formula, represents the projection function of the intermediate camera model, represents the back-projection function of the original camera model, [[ID=3�]]represents the original image, represents the original image to the intermediate image mapping, represents the rotation matrix of the intermediate image relative to the original image.

[0099] Among them, for the blank area of the intermediate image, bilinear interpolation can be used for filling. The blank area of the intermediate image refers to the area in the image conversion process where, due to reasons such as the viewing angle or projection method, some areas of the intermediate image are not covered by the data of the original image, manifested as blank or informationless areas in the image.

[0100] Adopting the embodiments of the present disclosure, through the calculation of the rotation matrix, the coordinate systems of the original image and the intermediate image are accurately aligned, making the conversion from the original image to the intermediate image accurate. By combining the back-projection and projection functions, it can further ensure that points in the three-dimensional space can be correctly mapped to the two-dimensional image plane, avoiding distortion and information loss. According to the camera of the intermediate camera model,

[0101] Among them, in an optional embodiment, the intermediate image includes a reference intermediate image with texture mapping and an intermediate image to be optimized without texture mapping, and further includes: determining an adaptive mapping direction of the intermediate image to be optimized relative to the reference intermediate image according to the intermediate camera model, and image transformation parameters corresponding to each adaptive mapping direction; determining a texture region of the reference intermediate image, and determining an intermediate pose corresponding to the reference intermediate image; determining an adaptive mapping image for each adaptive mapping direction according to the texture region of the reference intermediate image and the image transformation parameters; using the adaptive mapping image to replace the intermediate image to be optimized to obtain the updated intermediate image to be optimized; determining an intermediate pose corresponding to the updated intermediate image to be optimized according to the intermediate pose corresponding to the reference intermediate image and the adaptive mapping direction; constructing a three-dimensional Gaussian sputtering model according to the intermediate image and the corresponding intermediate pose, including: constructing a three-dimensional Gaussian sputtering model according to the reference intermediate image and its corresponding intermediate pose and the updated intermediate image to be optimized and its corresponding intermediate pose.

[0102] After mapping an original image into one or more intermediate images, due to different field-of-view angles of the original camera model, there are intermediate images with less or even no texture regions in the intermediate images obtained based on a fixed mapping direction. The intermediate images can be classified according to the texture regions each intermediate image has, including a reference intermediate image with texture mapping and an intermediate image to be optimized without texture mapping. A texture threshold can be set, and the intermediate images are classified according to the comparison result between the texture region of the intermediate image and the texture threshold.

[0103] After determining the reference intermediate image and the intermediate image to be optimized, it is necessary to update the intermediate image to be optimized according to the reference intermediate image. Specifically, it is processed according to the texture region of the reference intermediate image and converted to the intermediate image to be optimized, so as to realize the update of the intermediate image to be optimized.

[0104] Determine the texture region of the reference intermediate image and the corresponding intermediate pose. The texture region can be represented by the width and height of the circumscribed rectangle of the texture region. After determining the texture region of the reference intermediate image, convert the texture region of the reference intermediate image according to the relative position relationship between the reference intermediate image and the intermediate image to be optimized.

[0105] Determine the adaptive mapping direction between the reference intermediate image and the intermediate image to be optimized. The adaptive mapping direction can represent the relative position relationship between the reference intermediate image and the intermediate image to be optimized. Specifically, when the camera internal parameters of the intermediate camera model can determine 6 fixed mapping directions that are perpendicular to each other in pairs, the adaptive mapping direction between the reference intermediate image and the intermediate image to be optimized can be divided into three categories: front-back, left-right, and up-down.

[0106] Corresponding image transformation parameters can be formulated for each category of adaptive mapping direction respectively. The image transformation parameters are used to represent the transformation of the texture region between the reference intermediate image and the intermediate image to be optimized.

[0107] The adaptive mapping image represents the result after converting the texture region in the reference original image to the intermediate image to be optimized. In the adaptive mapping image, the texture region is adjacent to the image boundary. After replacing the intermediate image to be optimized with the adaptive mapping image, the update of the intermediate image to be optimized is realized. The intermediate pose corresponding to the updated optimized intermediate image can be determined by the intermediate pose corresponding to the reference intermediate image and the adaptive mapping direction.

[0108] After updating the intermediate image to be optimized, a three-dimensional Gaussian sputtering model is constructed by the updated intermediate image to be optimized and its corresponding intermediate pose, as well as the reference intermediate image and its corresponding intermediate pose.

[0109] Specifically, when the camera internal parameters of the intermediate camera model can determine 6 fixed mapping directions that are perpendicular to each other in pairs, the intermediate image to be optimized in each category of adaptive mapping direction can be updated in the following way.

[0110] (1) The adaptive mapping direction is the front-back direction

[0111] When the adaptive mapping direction between the reference intermediate image and the intermediate image to be optimized is the front-back direction, the intermediate pose corresponding to the intermediate image to be optimized does not need to be updated. Only the adaptive mapping image of the intermediate image to be optimized needs to be determined. The adaptive mapping image can be determined by the converted texture region, and the conversion of the texture region is determined based on the image transformation parameters. The image transformation parameters in the front-back direction can not modify the size of the texture region in the reference intermediate image. The texture region of the intermediate image to be optimized can be updated by the following formula:

[0112]

[0113] In the above formula, represents the width of the circumscribed rectangle of the texture region corresponding to the reference intermediate image, represents the height of the circumscribed rectangle of the texture region corresponding to the reference intermediate image, represents the width of the updated intermediate image to be optimized, Represents the height of the intermediate image to be optimized after update.

[0114] (2) The adaptive mapping direction is the left - right direction

[0115] When the adaptive mapping direction between the reference intermediate image and the intermediate image to be optimized is the left - right direction, it is necessary to update the intermediate pose and texture region corresponding to the intermediate image to be optimized.

[0116] The intermediate pose corresponding to the intermediate image to be optimized can be determined by the following formula.

[0117]

[0118] Among them, Represents the longitude rotation between the reference intermediate image and the intermediate image to be optimized, Represents the original pose corresponding to the original images of the reference intermediate image and the intermediate image to be optimized, Represents the latitude rotation of the reference intermediate image with respect to the pose of the original image, Represents the rotation matrix of the intermediate image to be optimized with respect to the original image.

[0119] The longitude rotation between the reference intermediate image and the intermediate image to be optimized can be determined by the following formula:

[0120]

[0121] In the above formula, Represents the width of the circumscribed rectangle of the texture region corresponding to the reference intermediate image, Represents the focal length of the intermediate camera on the x - axis of the image plane.

[0122] The texture region of the adaptive mapping image corresponding to the intermediate image to be optimized can be determined by the following formula:

[0123]

[0124] In the above formula, Represents the width of the circumscribed rectangle of the texture region corresponding to the reference intermediate image, Represents the height of the circumscribed rectangle of the texture region corresponding to the reference intermediate image, Represents the width of the intermediate image to be optimized after update, Represents the height of the intermediate image to be optimized after update, Represents the width of the reference intermediate image, Represents the focal length of the intermediate camera on the x - axis of the image plane.

[0125] (3) The adaptive mapping direction is the up - down direction

[0126] When the adaptive mapping direction between the reference intermediate image and the intermediate image to be optimized is the up-down direction, it is necessary to update the intermediate pose and texture region corresponding to the intermediate image to be optimized.

[0127] The intermediate pose corresponding to the intermediate image to be optimized can be determined by the following formula.

[0128]

[0129] Where, represents the latitude rotation between the reference intermediate image and the intermediate image to be optimized, represents the original pose corresponding to the original images of the reference intermediate image and the intermediate image to be optimized, represents the longitude rotation of the reference intermediate image with respect to the pose of the original image, represents the rotation matrix of the intermediate image to be optimized with respect to the original image.

[0130] The latitude rotation between the reference intermediate image and the intermediate image to be optimized can be determined by the following formula:

[0131]

[0132] In the above formula, represents the height of the circumscribed rectangle of the texture region corresponding to the reference intermediate image, represents the focal length of the intermediate camera in the y-axis direction of the image plane.

[0133] The texture region of the adaptive mapping image corresponding to the intermediate image to be optimized can be determined by the following formula:

[0134]

[0135] In the above formula, represents the width of the circumscribed rectangle of the texture region corresponding to the reference intermediate image, represents the height of the circumscribed rectangle of the texture region corresponding to the reference intermediate image, represents the width of the updated intermediate image to be optimized, represents the height of the updated intermediate image to be optimized, represents the height of the reference intermediate image, represents the focal length of the intermediate camera in the y-axis direction of the image plane.

[0136] By adopting the embodiments of the present disclosure, through the calculation of the adaptive mapping direction and the image transformation parameters, the intermediate image to be optimized can be updated, the textureless region of the intermediate image to be optimized can be filled, and it can be aligned with the texture region of the reference intermediate image, reducing the textureless region in the intermediate image and optimizing the image quality, making the final image smoother and more detailed.

[0137] Among them, in an optional embodiment, it further includes: filtering out redundant three-dimensional Gaussians in the three-dimensional Gaussian sputtering model through the inverse mapping region and the rendering direction to obtain a filtered three-dimensional Gaussian sputtering model, where the redundant three-dimensional Gaussians are those that do not participate in generating the target original image; rendering a target intermediate image using the three-dimensional Gaussian sputtering model through the rendering direction, including: based on the filtered three-dimensional Gaussian sputtering model, allocating effective three-dimensional Gaussians to corresponding rendering directions to obtain the three-dimensional Gaussians corresponding to each rendering direction, where the effective three-dimensional Gaussians are those that participate in rendering the intermediate image of the rendering direction; and rendering a target intermediate image using the three-dimensional Gaussians corresponding to each rendering direction through the three-dimensional Gaussian sputtering model.

[0138] According to the mapping relationship between the original camera model and the intermediate camera model, the correspondence between the intermediate image and the original image needs to be determined. Based on the correspondence between the intermediate image and the original image, the rendering direction and the inverse mapping region are pre-calculated. The rendering direction is used to obtain a target intermediate image through the three-dimensional Gaussian sputtering model, and the inverse mapping region is used to obtain a target original image through the target intermediate image.

[0139] The three-dimensional Gaussian sputtering model can be represented by a series of three-dimensional Gaussians. Through the inverse mapping region and the rendering direction, redundant three-dimensional Gaussians can be determined. For example, in the case of obtaining a target intermediate image based on the rendering direction through the three-dimensional Gaussian sputtering model, the three-dimensional Gaussians that do not participate in rendering the target intermediate image can be determined. In the case of inverse mapping the target intermediate image into a target original image, the pixels in the target intermediate image that will not be inverse mapped to the inverse mapping region can be determined, thereby further determining the three-dimensional Gaussians that do not participate in the inverse mapping. These three-dimensional Gaussians that do not participate in rendering the target intermediate image and those that do not participate in the inverse mapping can be determined as redundant three-dimensional Gaussians.

[0140] Filter out the redundant three-dimensional Gaussians from the three-dimensional Gaussian sputtering model to obtain a filtered three-dimensional Gaussian sputtering model.

[0141] Allocate the effective three-dimensional Gaussians in the filtered three-dimensional Gaussian sputtering model to the corresponding rendering directions, and each rendering direction will have a set of three-dimensional Gaussians. The effective three-dimensional Gaussians represent the three-dimensional Gaussians that participate in rendering to generate the target intermediate image, and these three-dimensional Gaussians can have a substantial impact on the result of the target original image.

[0142] On each rendering direction, perform rendering calculations based on the corresponding three-dimensional Gaussians to generate different parts of the target intermediate image. Each effective three-dimensional Gaussian affects a certain area in the target intermediate image, and the contributions of all these effective Gaussians will be combined together to form the final rendering effect and obtain the target intermediate image.

[0143] With the embodiments of the present disclosure, by filtering redundant three-dimensional Gaussians and only retaining the three-dimensional Gaussians that contribute to the target original image, the computational amount and memory consumption can be significantly reduced. After filtering the redundant three-dimensional Gaussians, the remaining three-dimensional Gaussians will be assigned to different rendering directions, and the three-dimensional Gaussians assigned to different rendering directions will only participate in the calculations in that rendering direction, which can further reduce the computational complexity and ensure that the computing resources are only used for the parts that truly affect the rendering result. During the rendering process, by assigning the Gaussians to the corresponding rendering directions, it can be ensured that the three-dimensional Gaussians used in each rendering direction are closely related to the rendering requirements of that direction. By reducing the computational amount in each rendering direction, the overall rendering time will be significantly shortened.

[0144] Among them, in an optional embodiment, it further includes: initializing a learnable camera embedding through the number of original camera models; the number of the original camera models is consistent with the number of categories of the camera device; splicing the learnable camera embedding and the target original image along the image channel direction to obtain a spliced image; inputting the spliced image into an optimization network to output a pixel-level target transformation and a restored image, where the optimization network is used to optimize the target original image; obtaining an optimized target original image through the target transformation, the restored image, and the target original image.

[0145] In cross-device rendering, the original images may come from different original camera models, such as fisheye cameras, panoramic cameras, pinhole cameras, etc. Different original camera models have different characteristics such as field of view angles and imaging methods. Therefore, a learnable camera embedding is initialized for each original camera model. The learnable camera embedding is a vector obtained through training. In this embodiment, the vector can be a vector with a length of 64. The learnable camera embedding contains the characteristics and information of the camera model. By learning the learnable camera embedding, the optimization network can understand the characteristics of different original camera models, thereby improving the accuracy and consistency.

[0146] The target original image obtained through the inverse mapping may have problems such as distortion, noise, or detail loss, which can be processed by the optimization network. The optimization network is a neural network used to process the target original image, repair the distorted, noisy, or inaccurate areas of the target original image, so as to restore a clearer and more realistic target original image.

[0147] Splicing each learnable camera embedding with the target original image can specifically be performed along the channel direction of the image, adding the learnable camera embedding to the features of each pixel or region of the target original image to obtain a spliced image. The channel direction of the image represents the depth direction of the image. The spliced image is the input of the optimization network. After inputting the spliced image into the optimization network, the optimization network optimizes the target original image based on the spliced image.

[0148] The output of the optimized network includes pixel-level object transformation and restored images. Pixel-level object transformation means that through learning, the optimized network can adjust the color, brightness, or details of each pixel in the original object image, reduce distortion, and enhance details. Through pixel-level transformation, the optimized network can generate a restored image, which refers to the original object image with richer details and less distortion after optimization.

[0149] Using the pixel-level object transformation and restored images output by the optimized network, and the original object image obtained through inverse mapping, the finally optimized original object image is obtained, which can be specifically implemented through the following formula:

[0150]

[0151] In the above formula, represents the optimized original object image, represents the pixel-level object transformation, represents the restored image, represents the original object image obtained through inverse mapping, represents the edge region of the inverse mapping splicing of the intermediate image in the original object image, is the complementary set region of.

[0152] By adopting the embodiments of the present disclosure, the learnable camera embedding vectors are initialized according to the number of original camera models, which can model the particularities of each camera device, so that the features of each camera model can be obtained through learning, and thus the characteristics of different camera models can be identified and effective image optimization can be performed accordingly. By splicing the camera embedding and the original object image along the image channel direction, the spliced image can contain both the pixel information of the original object image and the feature information related to the camera model. Through the combination of object transformation, restored image, and original object image, the optimized original object image is generated, so that the finally obtained optimized original object image can effectively reduce distortion, improve image quality, and restore more real and delicate image details.

[0153] The three-dimensional Gaussian sputtering model is obtained through iterative training. The training process of the three-dimensional Gaussian sputtering model is introduced below.

[0154] (1) In the early stage of the training of the three-dimensional Gaussian sputtering model

[0155] Among them, in an optional embodiment, a three-dimensional Gaussian sputtering model is constructed according to the intermediate image and the corresponding intermediate pose, including: constructing a first initial three-dimensional Gaussian sputtering model according to the intermediate image and the corresponding intermediate pose; obtaining a first initial target intermediate image with the same pose as the intermediate image through the first initial three-dimensional Gaussian sputtering model; determining a first loss through the intermediate image and the first initial target intermediate image; adjusting the parameters of the first initial three-dimensional Gaussian sputtering model through the first loss; and adjusting the parameters of the first initial three-dimensional Gaussian sputtering model multiple times according to the above steps to obtain a three-dimensional Gaussian sputtering model.

[0156] In the early stage of training the three-dimensional Gaussian sputtering model, the difference between the intermediate image obtained by mapping the original image and the first initial target intermediate image rendered by the three-dimensional Gaussian sputtering model can be considered to update the three-dimensional Gaussian sputtering model. The intermediate image obtained by mapping the original image is the ground truth, and the first initial target intermediate image rendered by the three-dimensional Gaussian sputtering model is the prediction value.

[0157] First, determine the intermediate image and the corresponding intermediate pose. The specific steps can refer to the above step S12.

[0158] Construct a first initial three-dimensional Gaussian sputtering model according to the intermediate image and its corresponding intermediate pose. The first initial three-dimensional Gaussian sputtering model is a preliminary and inaccurate representation, but it provides a starting point for subsequent optimization.

[0159] Render the first initial target intermediate image through the first initial three-dimensional Gaussian sputtering model, and the pose of the first initial target intermediate image is the same as that of the intermediate image.

[0160] Calculate the difference between the intermediate image and the first initial target intermediate image, and the difference can be expressed as a first loss. The first loss can be determined by the following formula:

[0161]

[0162] Among them, represents the image structure loss between the intermediate image and the first initial target intermediate image, represents the pixel-level loss between the intermediate image and the first initial target intermediate image, is the weight coefficient.

[0163] Adjust the parameters of the first initial three-dimensional Gaussian sputtering model according to the calculated loss, thereby completing one iteration of optimizing the first initial three-dimensional Gaussian sputtering model.

[0164] The three-dimensional Gaussian sputtering model is obtained by repeatedly adjusting the parameters of the first initial three-dimensional Gaussian sputtering model until the loss reaches an acceptable threshold or the model performance no longer improves significantly.

[0165] Using the embodiments of the present disclosure, first, a first initial three-dimensional Gaussian sputtering model is quickly constructed based on the intermediate image and the corresponding intermediate pose, providing a basis for subsequent model optimization and adjustment. Through the first initial three-dimensional Gaussian sputtering model, a first initial target intermediate image with the same pose as the intermediate image can be obtained. By comparing the intermediate image and the first initial target intermediate image, a first loss is determined, which quantifies the difference between the image generated by the model and the real image and is the key to subsequent model optimization. Repeatedly adjusting the parameters of the initial three-dimensional Gaussian sputtering model, this iterative optimization method can gradually approach the optimal solution and improve the accuracy and robustness of the model.

[0166] (2) In the later stage of training the three-dimensional Gaussian sputtering model

[0167] Among them, in an optional embodiment, constructing a three-dimensional Gaussian sputtering model according to the intermediate image and the corresponding intermediate pose includes: constructing a second initial three-dimensional Gaussian sputtering model according to the intermediate image and the corresponding intermediate pose; obtaining a second initial target original image with the same pose as the original image through the second initial three-dimensional Gaussian sputtering model; obtaining an initial restored image and an initial target transformation corresponding to the second initial target original image through an initial optimization network; determining an optimized second initial target original image according to the initial restored image, the initial target transformation, and the second initial target original image; determining a distortion loss according to the initial restored image and the original image; determining a second loss through the optimized second initial target original image and the original image; adjusting the parameters of the initial optimization network and the second initial three-dimensional Gaussian sputtering model respectively according to the distortion loss and the second loss; repeatedly adjusting the parameters of the initial optimization network and the second initial three-dimensional Gaussian sputtering model according to the above steps to obtain the optimized network and the three-dimensional Gaussian sputtering model.

[0168] In the later stage of training the three-dimensional Gaussian sputtering model, the trained three-dimensional Gaussian sputtering model can be determined as the second initial three-dimensional Gaussian sputtering model. The parameters of the second initial three-dimensional Gaussian sputtering model can be adjusted according to the difference between the original image and the finally obtained second initial target original image. The original image is the image captured by the original camera and is the real value; the second initial target original image is the image obtained through the second initial three-dimensional Gaussian sputtering model and is the predicted value.

[0169] In the case of optimizing the second initial target original image using the initial optimization network, the initial optimization network can be jointly trained on the basis of training the second initial three-dimensional Gaussian sputtering model, and the parameters of the initial optimization network can be adjusted in each iteration to obtain the optimization network.

[0170] First, a second initial three-dimensional Gaussian sputtering model is constructed through the intermediate image and the corresponding intermediate pose. The intermediate image is obtained by mapping the original image.

[0171] The second initial target original image with the same pose as the original image is generated through the second initial three-dimensional Gaussian sputtering model. The generation of the second initial target original image involves rendering the second target intermediate image through the second initial three-dimensional Gaussian sputtering model, and obtaining the second target original image by inverse mapping of the second target intermediate image.

[0172] Subsequently, the initial restored image and the initial target transformation corresponding to the second initial target original image are obtained through the initial optimization network; according to the initial restored image, the initial target transformation, and the second initial target original image, the optimized second initial target original image is determined. The optimized second target original image is used to determine the second loss, which reflects the difference between the real image and the predicted image. The real image refers to the original image, and the predicted image refers to the optimized second target original image.

[0173] In order to focus on optimizing the distortion of the stitching edge in the inverse mapping area of the intermediate image in the second target original image, a distortion loss can be set. The following formula represents the specific method for determining the distortion loss through the initial restored image and the original image:

[0174]

[0175] In the above formula, represents the optimized target original image, represents the pixel-level target transformation, represents the restored image, represents the target original image obtained by inverse mapping, represents the stitching edge area of the inverse mapping of the intermediate image in the target original image, is the complementary set area of

[0176] The second loss is determined through the optimized second initial target original image and the original image.

[0177] Finally, according to the second loss and the distortion loss, the total loss is determined, and the total loss is represented by the following formula:

[0178]

[0179] Among them, represents the second loss, represents the pixel-level loss between the original image and the optimized second initial target original image, represents the image structure loss between the original image and the optimized second initial target original image, is the weight coefficient, is the distortion loss.

[0180] According to the calculated total loss, the parameters of the second initial three-dimensional Gaussian sputtering model and the initial optimization network are adjusted, thus completing an iteration of the adjustment for the second initial three-dimensional Gaussian sputtering model and the initial optimization network.

[0181] Through multiple adjustments of the parameters of the first initial three-dimensional Gaussian sputtering model and the initial optimization network until the total loss reaches an acceptable threshold or the model performance no longer improves significantly, the three-dimensional Gaussian sputtering model and the initial network are obtained.

[0182] By adopting the embodiments of the present disclosure, by specifically setting the distortion loss to focus on optimizing the stitching edge, the distortion of the stitching edge in the second initial target original image can be reduced targeted, so that the stitched image is more natural and the transition is smoother. In the later stage of training, the rendering quality of the second initial three-dimensional Gaussian sputtering model has been initially improved, and the model begins to have a certain ability to render more complex scenes. By jointly optimizing the initial optimization network, the realism and details of the rendering can be greatly improved.

[0183] To evaluate the cross-device rendering performance of the present disclosure, a cross-device rendering evaluation dataset is constructed. The present disclosure adopts the commonly used performance evaluation criteria for image rendering: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS). The test environment of the present disclosure is Ubuntu 20.04, equipped with a central processing unit of the Intel Gold 6330 series, with a processing frequency of 2.00 GHz, and additionally equipped with an NVIDIA GTX4090 graphics processing unit, with a core frequency of 2235 MHz and a video memory capacity of 24 GB.

[0184] The present disclosure is tested in a cross-device dataset. The quantitative results are shown in Table 1. The present disclosure outperforms the advanced 3D-GS method, achieving state-of-the-art rendering quality and demonstrating its superior performance and strong generalization ability in cross-device rendering tasks. Notably, the present disclosure can fully rely on a unified intermediate camera model based on a simple pinhole camera. This model can be easily integrated with pinhole camera-based methods widely used in the field of vision, further enhancing the scalability of the present disclosure. In addition, the present disclosure can achieve a rendering speed of 0.022085 seconds on an NVIDIA GTX 4090 graphics processor, meeting the requirements of real-time rendering.

[0185] Table 1: Quantitative test results of the present disclosure in the cross-device dataset

[0186]

[0187] Based on the same technical concept, the present disclosure provides a cross-device rendering system. The cross-device rendering system includes an adaptive mapping module, a pre-filtering and allocation module, and a camera-based optimization module. Refer to Figure 2 as shown Figure 2 is a schematic diagram of a cross-device rendering system shown in an embodiment of the present disclosure.

[0188] The adaptive mapping module is responsible for converting the original images from different cameras into unified intermediate images. Based on the intermediate images, a general three-dimensional Gaussian sputtering model is constructed and trained to achieve effective generalization processing of multiple camera source images.

[0189] The pre-filtering and allocation module is used to render the target intermediate images based on the unified three-dimensional Gaussian sputtering model and reverse-map these intermediate images back to the original images. To improve the rendering efficiency, the present disclosure proposes a pre-filtering and allocation module to optimize the processing process and avoid processing all three-dimensional Gaussian points every time rendering is performed.

[0190] The camera-based optimization module is used to, after obtaining the original images with inverse mapping, since there may be losses during the mapping process, the generated original images may be relatively rough. Considering that the mapping loss is closely related to the camera model, it is necessary to optimize the original images based on the camera model to reduce image distortion and improve the final effect.

[0191] Based on the same technical concept, the present disclosure provides a cross-device rendering device. Figure 3 is a schematic diagram of a cross-device rendering device shown in an embodiment of the present disclosure. As shown in Figure 3 the device includes:

[0192] An acquisition module 310, configured to acquire original images from different camera devices;

[0193] A mapping module 320, configured to convert the original image into an intermediate image according to the mapping relationship between the original camera model and the intermediate camera model corresponding to the original image, where the intermediate image carries the corresponding intermediate pose;

[0194] A construction module 330, configured to construct a three-dimensional Gaussian sputtering model according to the intermediate image and the corresponding intermediate pose;

[0195] A determination module 340, configured to render a target intermediate image by using the three-dimensional Gaussian sputtering model according to the rendering direction of the target intermediate image, and inverse-map the target intermediate image to an inverse mapping area corresponding to the target original image to obtain the target original image.

[0196] The embodiments of the present disclosure further provide an electronic device. Refer to Figure 4 , Figure 4 is a schematic diagram of an electronic device shown in the embodiments of the present disclosure. As Figure 4 shown, the electronic device 400 includes: a memory 410 and a processor 420. The memory 410 is communicatively connected to the processor 420 through a bus. A computer program is stored in the memory 410, and the computer program can run on the processor 420, thereby implementing the steps in the cross-device rendering method disclosed in the embodiments of the present disclosure.

[0197] The embodiments of the present disclosure further provide a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in the cross-device rendering method disclosed in the embodiments of the present disclosure are implemented.

[0198] The embodiments of the present disclosure further provide a computer program product, including a computer program. When the computer program is executed by a processor, the steps in the cross-device rendering method disclosed in the embodiments of the present disclosure are implemented.

[0199] The embodiments in this specification are all described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The same and similar parts among the embodiments can be referred to each other.

[0200] Those skilled in the art should understand that the embodiments of the present disclosure can be provided as a method, a device, or a computer program product. Therefore, the embodiments of the present disclosure can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the embodiments of the present disclosure can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories, CD-ROMs, optical memories, etc.) that contain computer-usable program codes.

[0201] Embodiments of the present disclosure are described with reference to the flowcharts and / or block diagrams of methods, apparatuses, electronic devices, and computer program products according to embodiments of the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram, and the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing terminal devices generate a device for implementing the functions specified in one process Figure 1 one process or multiple processes and / or blocks Figure 1 or a device for implementing the functions specified in multiple blocks.

[0202] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device implements the functions specified in one process Figure 1 one process or multiple processes and / or blocks Figure 1 or the functions specified in multiple blocks.

[0203] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device, such that a series of operation steps are executed on the computer or other programmable terminal device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable terminal device provide steps for implementing the functions specified in one process Figure 1 one process or multiple processes and / or blocks Figure 1 or the functions specified in multiple blocks.

[0204] Although some embodiments of the present disclosure have been described, those skilled in the art can make additional changes and modifications once they learn the basic creative concepts. Therefore, the appended claims are intended to be construed as including the preferred embodiments and all changes and modifications falling within the scope of the embodiments of the present disclosure.

[0205] The above has introduced in detail a cross-device rendering method, apparatus, device, and medium provided by the present disclosure. Specific examples are used herein to elaborate on the principles and implementation manners of the present disclosure. The description of the above embodiments is only used to help understand the method and its core idea of the present disclosure; at the same time, for those of ordinary skill in the art, according to the idea of the present disclosure, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to the present disclosure.

Claims

1. A cross-device rendering method, characterized in that, Including: Obtaining original images from different camera devices; Converting the original images into intermediate images according to the mapping relationship between the original camera models corresponding to the original images and the intermediate camera models, where the intermediate images carry corresponding intermediate poses; Constructing a three-dimensional Gaussian sputtering model based on the intermediate images and the corresponding intermediate poses; Rendering a target intermediate image by using the three-dimensional Gaussian sputtering model according to the rendering direction of the target intermediate image, and inverse mapping the target intermediate image to the inverse mapping area corresponding to the target original image to obtain the target original image; Rendering a target intermediate image by using the three-dimensional Gaussian sputtering model through the rendering direction, including: Filtering out redundant three-dimensional Gaussians in the three-dimensional Gaussian sputtering model through the inverse mapping area and the rendering direction, where the redundant three-dimensional Gaussians are those that do not participate in generating the target original image; Based on the filtered three-dimensional Gaussian sputtering model, allocating effective three-dimensional Gaussians to corresponding rendering directions to obtain three-dimensional Gaussians corresponding to each rendering direction, where the effective three-dimensional Gaussians are those that participate in rendering the intermediate image of the rendering direction; Through the three-dimensional Gaussians corresponding to each rendering direction, in each rendering direction, performing rendering calculations based on the corresponding three-dimensional Gaussians to generate different parts of the target intermediate image. Each three-dimensional Gaussian affects a certain part of the area in the target intermediate image, and the contributions of all these effective Gaussians are combined together to form the final rendering effect to obtain the target intermediate image.

2. The method according to claim 1, characterized in that Also including: Initializing a learnable camera embedding through the number of original camera models; the number of original camera models is consistent with the number of categories of the camera devices; Concatenating the learnable camera embedding and the target original image along the image channel direction to obtain a concatenated image; Inputting the concatenated image into an optimization network to output a pixel-level target transformation and a restored image, where the optimization network is used to optimize the target original image; Obtaining an optimized target original image through the target transformation, the restored image, and the target original image.

3. The method according to claim 1, characterized in that, Converting the original image into an intermediate image according to the mapping relationship between the original camera model corresponding to the original image and the intermediate camera model, including: Determining the latitude rotation and longitude rotation of the intermediate image relative to the original pose of the original image according to the camera internal parameters of the intermediate camera model; Determining the rotation matrix of the intermediate image relative to the original image through the latitude rotation and the longitude rotation; Determining the intermediate pose of the intermediate image through the rotation matrix and the original pose; Converting the original image into the intermediate image through the rotation matrix, the back-projection function corresponding to the original camera model, and the projection function of the intermediate camera model.

4. The method according to claim 1, characterized in that, The intermediate image includes a reference intermediate image with texture mapping and an intermediate image to be optimized without texture mapping, and also includes: Determine the adaptive mapping direction of the intermediate image to be optimized relative to the reference intermediate image according to the intermediate camera model, and the image transformation parameters corresponding to each adaptive mapping direction; Determine the texture region of the reference intermediate image and the intermediate pose corresponding to the reference intermediate image; Determine the adaptive mapping images in each adaptive mapping direction according to the texture region of the reference intermediate image and the image transformation parameters; Replace the intermediate image to be optimized with the adaptive mapping image to obtain the updated intermediate image to be optimized; Determine the intermediate pose corresponding to the updated intermediate image to be optimized according to the intermediate pose corresponding to the reference intermediate image and the adaptive mapping direction; Construct a three-dimensional Gaussian sputtering model according to the intermediate image and the corresponding intermediate pose, including: Construct a three-dimensional Gaussian sputtering model according to the reference intermediate image and its corresponding intermediate pose and the updated intermediate image to be optimized and its corresponding intermediate pose.

5. The method according to any one of claims 1-4, characterized in that Construct a three-dimensional Gaussian sputtering model according to the intermediate image and the corresponding intermediate pose, including: Construct a first initial three-dimensional Gaussian sputtering model according to the intermediate image and the corresponding intermediate pose; Obtain a first initial target intermediate image with the same pose as the intermediate image through the first initial three-dimensional Gaussian sputtering model; Determine a first loss through the intermediate image and the first initial target intermediate image; Adjust the parameters of the first initial three-dimensional Gaussian sputtering model through the first loss; Adjust the parameters of the first initial three-dimensional Gaussian sputtering model multiple times according to the above steps to obtain a three-dimensional Gaussian sputtering model.

6. The method according to claim 2, characterized in that Construct a three-dimensional Gaussian sputtering model according to the intermediate image and the corresponding intermediate pose, including: Construct a second initial three-dimensional Gaussian sputtering model according to the intermediate image and the corresponding intermediate pose; Obtain a second initial target original image with the same pose as the original image through the second initial three-dimensional Gaussian sputtering model; Obtain an initial restored image and an initial target transformation corresponding to the second initial target original image through an initial optimization network; Determine an optimized second initial target original image according to the initial restored image, the initial target transformation, and the second initial target original image; Determine a distortion loss according to the initial restored image and the original image; Determine a second loss through the optimized second initial target original image and the original image; Adjust the parameters of the initial optimization network and the second initial three-dimensional Gaussian sputtering model respectively according to the distortion loss and the second loss; Adjust the parameters of the initial optimization network and the second initial three-dimensional Gaussian sputtering model multiple times according to the above steps to obtain the optimization network and the three-dimensional Gaussian sputtering model.

7. A cross-device rendering device, characterized in that, Including: An acquisition module for acquiring original images from different camera devices; A mapping module for converting the original image into an intermediate image according to the mapping relationship between the original camera model corresponding to the original image and the intermediate camera model, and the intermediate image carries the corresponding intermediate pose; A construction module for constructing a three-dimensional Gaussian sputtering model according to the intermediate image and the corresponding intermediate pose; A determination module for rendering a target intermediate image by using the three-dimensional Gaussian sputtering model according to the rendering direction of the target intermediate image, and inverse mapping the target intermediate image to an inverse mapping area corresponding to the target original image to obtain the target original image; The determination module is further configured to: Filter out redundant three-dimensional Gaussians in the three-dimensional Gaussian sputtering model through the inverse mapping area and the rendering direction, where the redundant three-dimensional Gaussians are three-dimensional Gaussians that do not participate in generating the target original image; Based on the filtered three-dimensional Gaussian sputtering model, allocate effective three-dimensional Gaussians to corresponding rendering directions to obtain three-dimensional Gaussians corresponding to each rendering direction, where the effective three-dimensional Gaussians are three-dimensional Gaussians that participate in rendering the intermediate image of the rendering direction; Through the three-dimensional Gaussians corresponding to each rendering direction, in each rendering direction, perform rendering calculations based on the corresponding three-dimensional Gaussians to generate different parts of the target intermediate image. Each three-dimensional Gaussian affects a certain part of the area in the target intermediate image, and the contributions of all these effective Gaussians are combined together to form the final rendering effect, thereby obtaining the target intermediate image.

8. An electronic device, characterized in that, Including: A memory, a processor, and a computer program stored on the memory and executable on the processor. When the computer program is executed by the processor, the steps of the cross-device rendering method according to any one of claims 1-6 are implemented.

9. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium. When the computer program is executed by the processor, the steps of the cross-device rendering method according to any one of claims 1-6 are implemented.

Images