Method and device for synthesizing new perspective of road surface based on motion information and gaussian sputtering

Abstract

The application provides a pavement new perspective synthesis method based on motion information and Gaussian sputtering, comprising the following steps: acquiring a pavement point cloud and a pavement track point set; performing segmented processing on the pavement track point set to obtain a track rotation matrix of each track; generating a Gaussian ball at the position of each data point of the pavement point cloud and initializing attribute information of the Gaussian ball, wherein the track rotation matrix of each track is set as the rotation matrix of the Gaussian ball generated by the point cloud corresponding to each track; iteratively optimizing the attribute information of the Gaussian ball except the rotation matrix until a stop optimization condition is reached; and taking a rendering image generated by the attribute of the Gaussian ball when the iteration optimization is stopped as a pavement new perspective synthesis image. The application also provides a pavement new perspective synthesis device based on motion information and Gaussian sputtering, a computer program product and an electronic device, and the application makes the pavement synthesis image of the new perspective clearer.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and graphics, and in particular to a method and device for synthesizing a new road perspective based on motion information and Gaussian sputtering. Background Art

[0002] New road perspective synthesis technology reconstructs high-fidelity 3D road scenes by combining road image information from different perspectives, achieving realistic image synthesis of the road surface from different perspectives. It is widely used in scenarios such as autonomous driving, intelligent transportation, urban planning, and games.

[0003] Existing new perspective synthesis solutions for roads include the NeRF (Neural Radiance Fields) algorithm and the 3DGS (3D Gaussian Splatting) algorithm. The NeRF algorithm learns the continuous volume field of the scene through a multi-layer perceptron (MLP) network, and then uses volume rendering technology to synthesize images from a new perspective. Although it performs well in new perspective synthesis and three-dimensional reconstruction, it usually requires a lot of computing resources and time for training and rendering, and its real-time performance is poor. The 3DGS algorithm combines a Gaussian point cloud model with fast differentiable rasterization technology to estimate the point cloud from a set of multi-perspective images and create a Gaussian point cloud model, and then performs parameter optimization and fast rendering to achieve new perspective synthesis. The 3DGS algorithm has significantly improved rendering speed compared to the NeRF algorithm and is more suitable for real-time rendering applications. However, the existing 3DGS algorithm uses a random method to initialize the rotation matrix in the Gaussian sphere covariance matrix when synthesizing new perspectives of the road surface. Under the new perspective, it often manifests itself in problems such as loss of geometric information, blurred object outlines, and inaccurate depth perception. In particular, it performs poorly when surface details (such as lane lines and road surfaces) need to be accurately presented. Summary of the Invention

[0004] The present invention aims to solve the technical problems that the existing new road perspective synthesis scheme based on the 3DGS algorithm often manifests as loss of geometric information, blurred object contours, inaccurate depth perception, and insufficiently precise presentation of surface details, and provides a new road perspective synthesis method and device based on motion information and Gaussian sputtering.

[0005] In order to achieve the above-mentioned purpose of the present invention, according to the first aspect of the present invention, the present invention provides a method for synthesizing a new perspective of a road surface based on motion information and Gaussian sputtering, including: obtaining a road surface point cloud and a road surface trajectory point set; segmenting the road surface trajectory point set to obtain a trajectory rotation matrix for each trajectory segment; generating a Gaussian sphere at each data point position in the road surface point cloud, and initializing the attribute information of the Gaussian sphere, wherein the trajectory rotation matrix of each trajectory segment is set to the rotation matrix of the Gaussian sphere generated by the point cloud corresponding to each trajectory segment; iteratively optimizing the attribute information of the Gaussian sphere except the rotation matrix until the optimization stopping condition is reached; and using the rendered image corresponding to the Gaussian sphere attribute generated when the iterative optimization stops as the road surface new perspective synthesized image.

[0006] In order to achieve the above-mentioned purpose of the present invention, according to the second aspect of the present invention, the present invention provides a new road perspective synthesis device based on motion information and Gaussian sputtering, which is used to implement the new road perspective synthesis method based on motion information and Gaussian sputtering described in the first aspect of the present invention, including: an acquisition module, which acquires a road point cloud and a road trajectory point set; a trajectory rotation matrix acquisition module, which performs segmented processing on the road trajectory point set and obtains the trajectory rotation matrix of each trajectory; an initialization module, which generates a Gaussian sphere at each data point position of the road point cloud and initializes the attribute information of the Gaussian sphere, wherein the trajectory rotation matrix of each trajectory is set to the rotation matrix of the Gaussian sphere generated by the point cloud corresponding to each trajectory; an iterative optimization module, which iteratively optimizes the attribute information of the Gaussian sphere except the rotation matrix until the optimization stopping condition is reached; a road surface new perspective synthesis image acquisition module, which uses the rendered image generated corresponding to the Gaussian sphere attributes when the iterative optimization stops as the road surface new perspective synthesis image.

[0007] In order to achieve the above-mentioned purpose of the present invention, according to the third aspect of the present invention, the present invention provides a computer program product, including a computer program, characterized in that when the computer program is executed by a processor, it implements the steps of the new road surface perspective synthesis method based on motion information and Gaussian sputtering described in the first aspect of the present invention.

[0008] In order to achieve the above-mentioned purpose of the present invention, according to the fourth aspect of the present invention, the present invention provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the new road perspective synthesis method based on motion information and Gaussian sputtering as described in the first aspect of the present invention.

[0009] The beneficial technical effects of the present invention are as follows: the rotation matrix of the Gaussian sphere is constrained by the trajectory rotation matrix obtained based on the set of road trajectory points, and the constraint is adopted in a segmented manner, which limits the rotation of the Gaussian sphere in iterative optimization, ensuring that the Gaussian sphere distribution can learn and maintain the accurate geometric information of the point cloud during the reconstruction process, so that after switching the perspective, the Gaussian sphere will not be misplaced or the direction will be confused, making the road surface composite image from the new perspective clearer. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 1 is a flow chart of a method for synthesizing a new road surface perspective based on motion information and Gaussian sputtering in a preferred embodiment of the present invention;

[0011] Figure 2 This is a schematic diagram of a process for synthesizing a new road perspective based on motion information and Gaussian sputtering in an application scenario of the present invention;

[0012] Figure 3 This is a comparison chart of rendering effects in an application scenario of the present invention;

[0013] Figure 4 It is a structural diagram of an electronic device in a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0014] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.

[0015] In the description of the present invention, it should be understood that the terms "longitudinal", "transverse", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention.

[0016] In the description of the present invention, unless otherwise specified and limited, it should be noted that the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection, or it can be the internal communication between two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to the specific circumstances.

[0017] The execution subject of the method for synthesizing a new perspective of the road surface based on motion information and Gaussian sputtering includes but is not limited to at least one of the electronic devices such as a server and a terminal that can be configured to execute the method provided in the embodiment of the present application. In other words, the method for synthesizing a new perspective of the road surface based on motion information and Gaussian sputtering can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc. The server can be an independent server, or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0018] The present invention discloses a method for synthesizing a new road view based on motion information and Gaussian sputtering. In a preferred embodiment, Figure 1 As shown, Figure 2 A schematic diagram of the process of synthesizing a new road view based on motion information and Gaussian sputtering in an application scenario is presented, including:

[0019] Step S1: Obtain a road surface point cloud and a road surface trajectory point set.

[0020] The road surface point cloud is preferably, but not limited to, road surface point cloud data acquired by a point cloud acquisition device such as a lidar, or a group of road surface images from several perspectives acquired by a camera, and the group of road surface multi-perspective images is processed according to the Structure-from-Motion (SfM) algorithm to obtain road surface point cloud data. Preferably, the real image of the road surface is acquired synchronously for use in the subsequent Gaussian sphere iterative optimization. The road surface trajectory point set is a set of all road segment characterization points (such as the geometric center point of the road segment) within the three-dimensional space range corresponding to the road surface point cloud. Specifically, it can be obtained through the road trajectory on the map, or the road direction trajectory can be obtained by a collection vehicle carrying equipment such as a camera or lidar, or it can be obtained by derivation through road surface point cloud analysis. Therefore, the road surface point cloud and the road surface trajectory point set describe the road in the same three-dimensional space range, and there is a position correspondence. The road surface trajectory point set can be expressed as:

[0021] CP={CP1,…,CP N}

[0022] Indicates that there are N road trajectory points, CP1 indicates the first road trajectory point, CP N Represents the Nth road trajectory point.

[0023] Step S2: segment the road trajectory point set and obtain the trajectory rotation matrix of each trajectory segment.

[0024] The set of road trajectory points is divided into multiple segments along the trajectory direction. The segmentation can be uniform or non-uniform. For non-uniform segmentation, for road sections where the curvature of the road trajectory is greater than a preset curvature threshold, the first segment length is used for segmentation. For road sections where the curvature of the road trajectory is not greater than the preset curvature threshold, the second segment length is used for segmentation, where the first segment length is less than the second segment length.

[0025] In step S3, a Gaussian sphere is generated at each data point in the road surface point cloud. A Gaussian sphere is generated for each point in the road surface point cloud, and the attribute information of the Gaussian sphere is initialized. The rotation matrix of each trajectory segment is set to the rotation matrix of the Gaussian sphere generated by the point cloud corresponding to each trajectory segment. In one example, the rotation matrix of the first trajectory segment is set to the rotation matrix of all Gaussian spheres generated by the point cloud corresponding to the first trajectory segment.

[0026] In this embodiment, the attribute information of the Gaussian sphere includes a rotation matrix, a scaling matrix, opacity, and spherical harmonics. The rotation matrix and the scaling matrix constitute a covariance matrix. Initializing the attribute information of the Gaussian sphere in step S3 includes: randomly initializing the scaling matrix S and constructing the covariance matrix Σ of the Gaussian sphere:

[0027] Σ=RSS T R T

[0028] Here, R represents the rotation matrix of the Gaussian sphere, and the superscript T represents the transposed matrix. By introducing the road surface trajectory to constrain the rotation of the Gaussian sphere, the Gaussian sphere will not be misaligned or oriented after switching perspectives, making the road surface clearer from the new perspective.

[0029] Step S4, iteratively optimizing the attribute information of the Gaussian sphere except the rotation matrix until the optimization stop condition is reached.

[0030] In this embodiment, the optimization stopping condition is preferably but not limited to that the number of optimizations reaches a preset maximum number of optimizations, or the value of the loss function of the iterative optimization is less than a preset loss threshold, or the reduction rate of the value of the loss function of the iterative optimization is less than a preset reduction rate threshold.

[0031] In this embodiment, during the iterative optimization process of the Gaussian sphere attributes in step S4, each time optimization is performed, the Gaussian sphere is projected and rasterized to obtain a rendered image, and the loss function and the gradient of the loss function are calculated. If the optimization stopping condition is met, step S5 is entered, and the rendered image generated corresponding to the current Gaussian sphere attributes is used as a synthetic image of the new perspective of the road surface. If the optimization stopping condition is not met, the gradient descent method is used to optimize the attribute information of the Gaussian sphere except the rotation matrix, and the next optimization is entered.

[0032] In this embodiment, the Gaussian sphere projection and rasterization process involves projecting the Gaussian sphere onto a two-dimensional image plane and then, through the rasterization process, treating it as a "snowball" thrown into the image, leaving a diffusion trace to form a rendered image. Preferably, alpha blending is used during the rasterization process to combine the projections of different Gaussian spheres, ensuring accurate color and transparency information in the final image.

[0033] In this embodiment, to achieve a satisfactory rendering effect, Gaussian sphere property optimization and Gaussian sphere density control are performed alternately during the Gaussian sphere property iterative optimization process. For example, after a preset number of Gaussian sphere property iterative optimizations (e.g., 100 times), Gaussian sphere density control is performed once. After executing Gaussian sphere density control, Gaussian sphere property iterative optimization is continued, and this process is repeated alternately. Preferably, the Gaussian sphere density control process is as follows: when a small-scale Gaussian sphere splatting fails to cover the image, the corresponding Gaussian sphere is cloned to fill the image. If the image covers too much, the Gaussian sphere is split into two.

[0034] Step S5: Use the rendered image corresponding to the Gaussian sphere attributes when the iterative optimization stops as the new perspective synthetic image of the road surface, and output the new perspective synthetic image.

[0035] Existing point cloud rendering methods often suffer from problems such as loss of geometric information, blurred object outlines, and inaccurate depth perception under new perspectives. This is especially true when surface details (such as lane lines and road surfaces) need to be accurately presented. To address this issue, the present invention introduces road trajectory information to obtain the trajectory direction of the road, and uses these road rules as the true value. Trajectory constraints are introduced during iterative optimization to impose rotation restrictions on each Gaussian sphere. This ensures that the Gaussian distribution can learn and maintain the accurate geometric information of the point cloud during the reconstruction process, so that after switching perspectives, the Gaussian sphere will not be misplaced or have confused directions, making the road surface clearer from the new perspective. Figure 3This image demonstrates the difference between a new perspective synthesized image obtained using the new perspective synthesis method for road surfaces based on motion information and Gaussian sputtering, as provided by the present invention, and a rendered image obtained using a traditional 3D Gaussian sputtering algorithm. The image on the left is a rendered image obtained using a traditional 3D Gaussian sputtering algorithm, while the image on the right is a new perspective synthesized image obtained using the present invention. It can be seen that the left image (especially the selected area) is significantly less clear than the right image.

[0036] In a preferred embodiment, to improve rendering quality, during the iterative optimization of the attribute information of the Gaussian sphere, the loss function is:

[0037]

[0038] in, Represents the real image I and the rendered image The average value of the pixel difference; Represents the real image I and the rendered image The structural loss; λ represents the first weight coefficient; α represents the second weight coefficient; L scale represents the vertical scale loss of the Gaussian sphere distribution, L represents the vertical scale of the mth Gaussian ball, and M represents the number of Gaussian balls. scale The size of the Gaussian sphere in the road normal direction is constrained to ensure the smoothness of the road surface, thereby solving the problem of rough road surface when the viewpoint changes.

[0039] Specifically, S represents the number of pixels in the real image / rendered image, y s Represents the pixel value of the sth pixel in the real image, Represents the pixel value of the s-th pixel of the rendered image.

[0040] Specifically,

[0041] Among them, μ1 and μ2 represent the mean pixel values ​​of the real image and the rendered image respectively, σ1 and σ2 represent the pixel value variance of the real image and the rendered image respectively, and σ 12 Represents the pixel value covariance between the real image and the rendered image, C1 represents the first stability constant, and C2 represents the second stability constant. C1=0.01 2 , C2=0.03 2 .

[0042] In this embodiment, the value range of λ is 0.2-0.4, preferably 0.3. The value range of α is 30-100, preferably 50. The real image is the real road surface image corresponding to the road surface point cloud position.

[0043] In a preferred embodiment, in step S2, obtaining the trajectory rotation matrix of each trajectory segment includes:

[0044] In step S21 , the coordinates of the center of mass of each trajectory segment are determined, and the vector of each trajectory point in each trajectory segment relative to the center of mass is obtained, which is recorded as the first vector.

[0045] If a trajectory includes n trajectory points, n is greater than or equal to 1, then the coordinates of the center of mass CP' are:

[0046]

[0047] The coordinates of the i-th track point in this track are (x i ,y i ,z i ).

[0048] If two adjacent trajectory points in a certain trajectory are CP i and CP i+1 , then the trajectory point CP i The first vector is Track point CP i+1 The first vector is

[0049] Step S22, determining the rotation quaternion of the two adjacent trajectory points by the first vectors of the two adjacent trajectory points. Further preferably, step S22 includes:

[0050] Step A, obtaining the angle between the first vectors of two adjacent trajectory points;

[0051] In one example, the trajectory point CP i and CP i+1 The angle θ between the first vectors of two adjacent trajectory points ij The cosine of is:

[0052]

[0053] j=i+1. θ is obtained by the arc cosine function. ij .

[0054] Step B: normalize the first vectors of two adjacent trajectory points.

[0055] In the above example, After normalization, After normalization,

[0056]

[0057] Step C: obtain the rotation axis vector by cross-producting the normalized first vectors of two adjacent trajectory points, and normalize the rotation axis vector.

[0058] In the above example, the trajectory point CP i and CP i+1 The rotation axis vector for:

[0059]

[0060] Rotation axis vector Normalization is performed to obtain

[0061]

[0062] In step D, the rotation quaternion of two adjacent trajectory points is obtained by using the included angle and the normalized rotation axis vector.

[0063] In the above example, two adjacent trajectory points CP i and CP i+1 The rotation quaternion is q ij =[q w ,q x ,q y ,q z ], j=i+1:

[0064]

[0065] q w is the real part, q x ,q y ,q z are all imaginary parts, They are The x-axis coordinate, y-axis coordinate, and z-axis coordinate.

[0066] Step S23, based on the rotation quaternions of all two adjacent trajectory points in each trajectory, the average rotation quaternion of each trajectory is obtained. Preferably, the average rotation quaternion q of each trajectory is obtained according to the following formula: avg :

[0067]

[0068] Where n represents the number of trajectory points in each trajectory, i and j represent the trajectory point index, i∈[1,n-1],j=i+1; q ij Represents the rotation quaternion of adjacent trajectory points i and j.

[0069] Step S24: convert the average rotation quaternion into a trajectory rotation matrix.

[0070] Specifically, step S24 includes:

[0071] Step a, calculate the average rotation quaternion q avg =[q aw ,q ax ,q ay ,q az ] norm:

[0072]

[0073] Step b, use the norm to normalize the average rotation quaternion q avg :

[0074]

[0075] Step c, calculate the trajectory rotation matrix

[0076] R 11 =1-2(y' 2 +z' 2 )

[0077] R 12 =2(x'y'-z'ω')

[0078] R 13 =2(x'z'+y'ω')

[0079] R 21 =2(x'y'+z'ω')

[0080] R 22 =1-2(x' 2 +z' 2 )

[0081] R 23 =2(y'z'-x'ω')

[0082] R 31 =2(x'z'-y'ω')

[0083] R 32 =2(y'z'+x'ω')

[0084] R 33 =1-2(x' 2 +y' 2 )

[0085] This paper discloses a new road view synthesis method based on motion information and Gaussian sputtering. This method uses motion information to adjust the Gaussian sphere. During the rendering process, the position, shape, and color of the 3D Gaussian sphere are dynamically adjusted based on the captured vehicle's motion information, such as changes in perspective and path. This ensures that the synthesized image accurately reflects the overall direction of the road surface, improving rendering accuracy. The method also has excellent generalization capabilities and can adapt to different road conditions and viewing environments. It is applicable not only to straight lanes, but also to uphill and downhill slopes and curves, thus enhancing the algorithm's practicality and applicability.

[0086] The present invention further discloses a device for synthesizing a new road view perspective based on motion information and Gaussian sputtering, which is used to implement the method for synthesizing a new road view perspective based on motion information and Gaussian sputtering described in the first aspect of the present invention, comprising:

[0087] Acquisition module, obtains road surface point cloud and road surface trajectory point set;

[0088] The trajectory rotation matrix acquisition module processes the road trajectory point set in segments and obtains the trajectory rotation matrix of each segment;

[0089] An initialization module generates a Gaussian sphere at each data point position in the road surface point cloud and initializes the attribute information of the Gaussian sphere, wherein the trajectory rotation matrix of each trajectory segment is set to the rotation matrix of the Gaussian sphere generated by the point cloud corresponding to each trajectory segment;

[0090] Iterative optimization module, iteratively optimizes the attribute information of the Gaussian sphere except the rotation matrix until the optimization stop condition is reached;

[0091] The road surface new perspective synthetic image acquisition module uses the rendered image corresponding to the Gaussian sphere attributes when the iterative optimization stops as the road surface new perspective synthetic image.

[0092] In this embodiment, the acquisition module, trajectory rotation matrix acquisition module, initialization module, iterative optimization module and road surface new perspective synthetic image acquisition module correspond one-to-one to steps S1, S2, S3, S4 and S5 of the above-mentioned road surface new perspective synthesis method based on motion information and Gaussian sputtering, and will not be repeated here.

[0093] The present invention also discloses a computer program product, including a computer program, which, when executed by a processor, implements the steps of the above-mentioned method for synthesizing a new road surface perspective based on motion information and Gaussian sputtering provided by the present invention. The computer program product should be understood as a software product that mainly implements its solution through a computer program, such as a program product integrated in the cloud or a software library.

[0094] The present invention also discloses an electronic device. In one embodiment, the electronic device includes at least one processor; and a memory connected to the at least one processor; wherein,

[0095] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the new road perspective synthesis method based on motion information and Gaussian sputtering provided by the present invention.

[0096] like Figure 4 FIG2 is a schematic diagram of the structure of an electronic device for synthesizing a new road view based on motion information and Gaussian sputtering, according to an embodiment of the present invention. The electronic device may include a processor 10, a memory 11, a communication bus 12, and a communication interface 13. It may also include a computer program stored in the memory 11 and executable on the processor 10, such as a program for synthesizing a new road view based on motion information and Gaussian sputtering.

[0097] In some embodiments, the processor 10 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and a combination of various control chips. The processor 10 is the control core (Control Unit) of the electronic device, connecting the various components of the entire electronic device using various interfaces and lines, and executing the programs or modules stored in the memory 11 (for example, executing a new road perspective synthesis method based on motion information and Gaussian sputtering, etc.), as well as calling the data stored in the memory 11, to execute various functions of the electronic device and process data.

[0098] The memory 11 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory (for example, an SD or DX memory, etc.), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the memory 11 may be an internal storage unit of an electronic device, such as a mobile hard disk of the electronic device. In other embodiments, the memory 11 may also be an external storage device of an electronic device, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Furthermore, the memory 11 may also include both an internal storage unit and an external storage device of the electronic device. The memory 11 can not only be used to store application software and various types of data installed in the electronic device, such as the code of the program of the new perspective synthesis method of the road surface based on motion information and Gaussian sputtering, but can also be used to temporarily store data that has been output or is to be output.

[0099] The communication bus 12 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable communication between the memory 11 and at least one processor 10, etc.

[0100] The communication interface 13 is used for communication between the above-mentioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device and other electronic devices. The user interface may be a display (Display), an input unit (such as a keyboard (Keyboard)), optionally, the user interface may also be a standard wired interface, a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode, organic light-emitting diode) touch device, etc. Among them, the display may also be appropriately referred to as a display screen or a display unit, for displaying information processed in the electronic device and for displaying a visual user interface.

[0101] Figure 4 Only the electronic device with components is shown, and it can be understood by those skilled in the art that Figure 4 The structure shown does not limit the electronic device, and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0102] For example, although not shown, the electronic device may further include a power source (such as a battery) for powering various components. Preferably, the power source may be logically connected to at least one processor 10 via a power management device, thereby implementing functions such as charge management, discharge management, and power consumption management through the power management device. The power source may further include any components such as one or more DC or AC power sources, a recharging device, a power failure detection circuit, a power converter or inverter, a power status indicator, etc. The electronic device may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0103] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0104] Furthermore, if the modules / units integrated into the electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0105] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "example," "specific example," "one implementation," "a preferred implementation," or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0106] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

Claims

1. A new road view synthesis method based on motion information and Gaussian sputtering, characterized by: include: Obtain road surface point cloud and road surface trajectory point set; The road trajectory point set is segmented and the trajectory rotation matrix of each segment is obtained; Generate a Gaussian sphere at each data point position in the road surface point cloud, initialize the attribute information of the Gaussian sphere, wherein the trajectory rotation matrix of each trajectory segment is set to the rotation matrix of the Gaussian sphere generated by the point cloud corresponding to each trajectory segment; Iteratively optimize the attribute information of the Gaussian sphere except the rotation matrix until the optimization stop condition is reached; The rendered image corresponding to the Gaussian sphere attributes when the iterative optimization stops is used as the new perspective synthetic image of the road surface; The step of obtaining the trajectory rotation matrix of each trajectory segment includes: Determine the coordinates of the center of mass of each trajectory segment, and obtain the vector of each trajectory point relative to the center of mass in each trajectory segment, which is recorded as the first vector; Determine the rotation quaternion of the two adjacent trajectory points by using the first vectors of the two adjacent trajectory points; The average rotation quaternion of each trajectory segment is obtained based on the rotation quaternions of all two adjacent trajectory points of each trajectory segment; Convert the average rotation quaternion to a trajectory rotation matrix; During the iterative optimization of the Gaussian sphere attributes, each time the optimization is performed, the Gaussian sphere is projected and rasterized to obtain a rendered image, and the loss function is calculated. If the optimization stop condition is met, the rendered image corresponding to the current Gaussian sphere attribute is used as the new perspective composite image of the road surface. If the optimization stop condition is not met, the gradient descent method is used to optimize the Gaussian sphere attribute information except the rotation matrix, and the next optimization is started. The loss function is: in, Represents the real image I and the rendered image The average value of the pixel difference; Represents the real image I and the rendered image The structural loss; λ represents the first weight coefficient; α represents the second weight coefficient; L scale represents the vertical scale loss of the Gaussian sphere distribution, scale φm Represents the vertical scale of m Gaussian balls, and M represents the number of Gaussian balls.

2. The method for synthesizing a new road view based on motion information and Gaussian sputtering according to claim 1, wherein: The determining the rotation quaternion of the two adjacent trajectory points by using the first vectors of the two adjacent trajectory points includes: Get the angle between the first vectors of two adjacent trajectory points; Normalize the first vector of two adjacent trajectory points; The rotation axis vector is obtained by cross-producting the normalized first vectors of two adjacent trajectory points, and the rotation axis vector is normalized; The rotation quaternion of two adjacent trajectory points is obtained through the included angle and the normalized rotation axis vector.

3. The method for synthesizing a new road view based on motion information and Gaussian sputtering according to claim 1, wherein: The average rotation quaternion q of each trajectory is obtained according to the following formula avg : Where n represents the number of trajectory points in each trajectory, i and j represent the trajectory point index, i∈[1,n-1],j=i+1; q ij represents the rotation quaternion of adjacent trajectory points i and j.

4. The method for synthesizing a new road view based on motion information and Gaussian sputtering according to any one of claims 1 to 3, wherein: During the iterative optimization process of Gaussian sphere properties, Alternate between Gaussian sphere property optimization and Gaussian sphere density control.

5. The method for synthesizing a new road view based on motion information and Gaussian sputtering according to claim 4, wherein: The attribute information of the Gaussian sphere includes rotation matrix, scaling matrix, opacity and spherical harmonics; Initializing the attribute information of the Gaussian sphere includes randomly initializing the scaling matrix S and constructing the covariance matrix Σ of the Gaussian sphere: S = RSS T R T Where R represents the rotation matrix of the Gaussian sphere.

6. A device for synthesizing a new road view based on motion information and Gaussian sputtering, used to implement the method for synthesizing a new road view based on motion information and Gaussian sputtering according to any one of claims 1 to 5, characterized in that: include: Acquisition module, obtains road surface point cloud and road surface trajectory point set; The trajectory rotation matrix acquisition module processes the road trajectory point set in segments and obtains the trajectory rotation matrix of each segment; An initialization module generates a Gaussian sphere at each data point position in the road surface point cloud and initializes the attribute information of the Gaussian sphere, wherein the trajectory rotation matrix of each trajectory segment is set to the rotation matrix of the Gaussian sphere generated by the point cloud corresponding to each trajectory segment; Iterative optimization module, iteratively optimizes the attribute information of the Gaussian sphere except the rotation matrix until the optimization stop condition is reached; The road surface new perspective synthetic image acquisition module uses the rendered image corresponding to the Gaussian sphere attributes when the iterative optimization stops as the road surface new perspective synthetic image; Among them, in the trajectory rotation matrix acquisition module, the trajectory rotation matrix of each trajectory is obtained, including: Determine the coordinates of the center of mass of each trajectory segment, and obtain the vector of each trajectory point relative to the center of mass in each trajectory segment, which is recorded as the first vector; Determine the rotation quaternion of the two adjacent trajectory points by using the first vectors of the two adjacent trajectory points; The average rotation quaternion of each trajectory segment is obtained based on the rotation quaternions of all two adjacent trajectory points of each trajectory segment; Convert the average rotation quaternion to a trajectory rotation matrix; In the iterative optimization module, during the iterative optimization of the Gaussian sphere attributes, each time the optimization is performed, the Gaussian sphere is projected and rasterized to obtain a rendered image, and the loss function is calculated. If the optimization stop condition is met, the rendered image corresponding to the current Gaussian sphere attribute is used as the new perspective composite image of the road surface. If the optimization stop condition is not met, the gradient descent method is used to optimize the attribute information of the Gaussian sphere except the rotation matrix, and the next optimization is started. The loss function is: in, Represents the real image I and the rendered image The average value of the pixel difference; Represents the real image I and the rendered image The structural loss; λ represents the first weight coefficient; α represents the second weight coefficient; L scale represents the vertical scale loss of the Gaussian sphere distribution, scale φm Represents the vertical scale of m Gaussian balls, and M represents the number of Gaussian balls.

7. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

8. An electronic device, characterized in that: The electronic device comprises: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for synthesizing a new perspective of a road surface based on motion information and Gaussian sputtering as described in any one of claims 1 to 5.

Images