Pavement new view angle synthesis method and device based on motion information and Gaussian sputtering

Abstract

The invention provides a pavement new view angle synthesis method based on motion information and Gaussian sputtering. The pavement new view angle synthesis method comprises the following steps: acquiring a pavement point cloud and a pavement track point set; segmenting the road track point set to obtain a track rotation matrix of each segment of track; a Gaussian ball is generated at each data point position of the road surface point cloud, attribute information of the Gaussian balls is initialized, and a track rotation matrix of each section of track is set as a rotation matrix of the Gaussian balls generated by the point cloud corresponding to each section of track; iteratively optimizing attribute information of the Gaussian ball except for the rotation matrix until an optimization stopping condition is met; and taking a rendered image correspondingly generated by the Gaussian ball attribute when the iterative optimization stops as a new visual angle synthetic image of the road surface. The invention further provides a pavement new view angle synthesis device based on the motion information and Gaussian sputtering, a computer program product and electronic equipment. The pavement new view angle synthesis method based on the motion information and Gaussian sputtering enables a pavement synthesis image with a new view angle to be clearer.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and graphics, and in particular to a method and device for synthesizing a new road surface perspective based on motion information and Gaussian sputtering. Background Art

[0002] The new road perspective synthesis technology reconstructs high-fidelity 3D road scenes by combining road image information from different perspectives, achieving realistic image synthesis of the road surface from different perspectives. It is widely used in scenarios such as autonomous driving, intelligent transportation, urban planning, and games.

[0003] Existing new perspective synthesis solutions for roads include the NeRF (Neural Radiance Fields) algorithm and the 3DGS (3D Gaussian Splatting) algorithm. The NeRF algorithm learns the continuous volume field of the scene through a multi-layer perceptron (MLP) network, and then uses volume rendering technology to synthesize images from a new perspective. Although it performs well in new perspective synthesis and three-dimensional reconstruction, it usually requires a lot of computing resources and time for training and rendering, and its real-time performance is poor. The 3DGS algorithm combines the Gaussian point cloud model and fast differentiable rasterization technology to estimate the point cloud from a set of multi-perspective images, create a Gaussian point cloud model, and then perform parameter optimization and fast rendering to achieve new perspective synthesis. The 3DGS algorithm has significantly improved the rendering speed compared to the NeRF algorithm and is more suitable for real-time rendering applications. However, the existing 3DGS algorithm uses a random method to initialize the rotation matrix in the Gaussian spherical covariance matrix when synthesizing a new perspective of the road. Under the new perspective, it often manifests itself in problems such as loss of geometric information, blurred object contours, and inaccurate depth perception. In particular, it performs poorly when surface details (such as lane lines and road surfaces) need to be accurately presented. Summary of the invention

[0004] The present invention aims to solve the technical problems that the existing new perspective synthesis scheme for the road surface based on the 3DGS algorithm often manifests itself as loss of geometric information, blurred object contours, inaccurate depth perception, and insufficiently precise presentation of surface details, and provides a new perspective synthesis method and device for the road surface based on motion information and Gaussian sputtering.

[0005] In order to achieve the above-mentioned purpose of the present invention, according to the first aspect of the present invention, the present invention provides a method for synthesizing a new perspective of a road surface based on motion information and Gaussian sputtering, including: obtaining a road surface point cloud and a road surface trajectory point set; segmenting the road surface trajectory point set to obtain a trajectory rotation matrix for each trajectory segment; generating a Gaussian sphere at each data point position of the road surface point cloud, and initializing the attribute information of the Gaussian sphere, wherein the trajectory rotation matrix of each trajectory segment is set to the rotation matrix of the Gaussian sphere generated by the point cloud corresponding to each trajectory segment; iteratively optimizing the attribute information of the Gaussian sphere except the rotation matrix until the optimization stopping condition is reached; and using the rendered image corresponding to the Gaussian sphere attribute generated when the iterative optimization stops as the road surface new perspective synthesized image.

[0006] In order to achieve the above-mentioned purpose of the present invention, according to the second aspect of the present invention, the present invention provides a new road perspective synthesis device based on motion information and Gaussian sputtering, which is used to implement the new road perspective synthesis method based on motion information and Gaussian sputtering described in the first aspect of the present invention, including: an acquisition module, which acquires a road point cloud and a road trajectory point set; a trajectory rotation matrix acquisition module, which performs segmented processing on the road trajectory point set to obtain the trajectory rotation matrix of each trajectory; an initialization module, which generates a Gaussian sphere at each data point position of the road point cloud and initializes the attribute information of the Gaussian sphere, wherein the trajectory rotation matrix of each trajectory is set to the rotation matrix of the Gaussian sphere generated by the point cloud corresponding to each trajectory; an iterative optimization module, which iteratively optimizes the attribute information of the Gaussian sphere except the rotation matrix until the optimization stopping condition is reached; a road surface new perspective synthesis image acquisition module, which uses the rendered image generated corresponding to the Gaussian sphere attributes when the iterative optimization stops as the road surface new perspective synthesis image.

[0007] In order to achieve the above-mentioned purpose of the present invention, according to the third aspect of the present invention, the present invention provides a computer program product, including a computer program, characterized in that when the computer program is executed by a processor, the steps of the method for synthesizing a new perspective of the road surface based on motion information and Gaussian sputtering described in the first aspect of the present invention are implemented.

[0008] In order to achieve the above-mentioned purpose of the present invention, according to the fourth aspect of the present invention, the present invention provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for synthesizing a new perspective of the road surface based on motion information and Gaussian sputtering as described in the first aspect of the present invention.

[0009] The beneficial technical effects of the present invention are as follows: the rotation matrix of the Gaussian sphere is constrained by the trajectory rotation matrix obtained based on the road surface trajectory point set, and the constraint is adopted in a segmented manner, which limits the rotation of the Gaussian sphere in iterative optimization, and ensures that the Gaussian sphere distribution can learn and maintain the accurate geometric information of the point cloud during the reconstruction process, so that after switching the perspective, the Gaussian ball will not be misplaced or confused in direction, making the road surface composite image of the new perspective clearer. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is a flow chart of a method for synthesizing a new road surface perspective based on motion information and Gaussian sputtering in a preferred embodiment of the present invention;

[0011] Figure 2 It is a schematic diagram of a process of synthesizing a new road surface perspective based on motion information and Gaussian sputtering in an application scenario of the present invention;

[0012] Figure 3 This is a comparison chart of rendering effects in an application scenario of the present invention;

[0013] Figure 4 It is a schematic structural diagram of an electronic device in a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0014] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0015] In the description of the present invention, it is necessary to understand that the terms "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0016] In the description of the present invention, unless otherwise specified and limited, it should be noted that the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection, or it can be the internal connection between two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to the specific circumstances.

[0017] The execution subject of the new road surface perspective synthesis method based on motion information and Gaussian sputtering includes but is not limited to at least one of the electronic devices such as the server, terminal, etc. that can be configured to execute the method provided in the embodiment of the present application. In other words, the new road surface perspective synthesis method based on motion information and Gaussian sputtering can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc. The server can be an independent server, or it can be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms.

[0018] The present invention discloses a method for synthesizing a new road view based on motion information and Gaussian sputtering. In a preferred embodiment, Figure 1 As shown, Figure 2 The schematic diagram of the process of synthesizing a new road view based on motion information and Gaussian sputtering in an application scenario is shown, including:

[0019] Step S1, obtaining a road surface point cloud and a road surface trajectory point set.

[0020] The road surface point cloud is preferably, but not limited to, road surface point cloud data acquired by a point cloud acquisition device such as a laser radar, or a group of road surface images from several perspectives acquired by a camera, and the group of road surface multi-perspective images are processed according to the Structure-from-Motion (SfM) algorithm to obtain road surface point cloud data. Preferably, the real image of the road surface is acquired synchronously for use in the subsequent Gaussian sphere iterative optimization. The road surface trajectory point set is a set of all road segment characterization points (such as the geometric center point of the road segment) within the three-dimensional space range corresponding to the road surface point cloud. Specifically, it can be obtained through the road trajectory on the map, or it can also be obtained by a collection vehicle carrying equipment such as a camera or a laser radar to obtain the road direction trajectory, or it can also be obtained by analyzing and deriving the road surface point cloud. Therefore, the road surface point cloud and the road surface trajectory point set describe the road in the same three-dimensional space range, and there is a position correspondence. The road surface trajectory point set can be expressed as:

[0021] CP={CP1,…,CP N}

[0022] Indicates that there are N road trajectory points, CP1 indicates the first road trajectory point, CP N Represents the Nth road trajectory point.

[0023] Step S2, segmenting the road trajectory point set to obtain a trajectory rotation matrix for each trajectory segment.

[0024] The road surface trajectory point set is divided into a plurality of segments along the trajectory direction, which can be evenly divided or unevenly divided. For uneven division, for a road section area where the curvature of the road surface trajectory is greater than a preset curvature threshold, the first segment length is used for segmentation, and for a road section area where the curvature of the road surface trajectory is not greater than the preset curvature threshold, the second segment length is used for segmentation, and the first segment length is less than the second segment length.

[0025] Step S3, a Gaussian sphere is generated at each data point position of the road surface point cloud, a Gaussian sphere is generated for all points in the road surface point cloud, and the attribute information of the Gaussian sphere is initialized, wherein the trajectory rotation matrix of each segment of the trajectory is set to the rotation matrix of the Gaussian sphere generated by the point cloud corresponding to each segment of the trajectory. In an example, the trajectory rotation matrix of the first segment of the trajectory is set to the rotation matrix of all Gaussian spheres generated by the point cloud corresponding to the first segment of the trajectory.

[0026] In this embodiment, the attribute information of the Gaussian sphere includes a rotation matrix, a scaling matrix, opacity and a spherical harmonic function, and the rotation matrix and the scaling matrix constitute a covariance matrix; the attribute information of the Gaussian sphere is initialized in step S3, including: randomly initializing the scaling matrix S, and constructing the covariance matrix Σ of the Gaussian sphere:

[0027] Σ=RSS T R T

[0028] Among them, R represents the rotation matrix of the Gaussian ball, and the superscript T represents the transposed matrix of the matrix. By introducing the road surface trajectory to restrict the rotation of the Gaussian ball, the Gaussian ball will not be misplaced or confused in direction after switching the perspective, making the road surface clearer in the new perspective.

[0029] Step S4, iteratively optimizing the attribute information of the Gaussian sphere except the rotation matrix until the optimization stop condition is reached.

[0030] In this embodiment, the optimization stopping condition is preferably but not limited to that the number of optimizations reaches a preset maximum number of optimizations, or the value of the loss function of the iterative optimization is less than a preset loss threshold, or the reduction rate of the value of the loss function of the iterative optimization is less than a preset reduction rate threshold.

[0031] In this embodiment, during the iterative optimization process of the Gaussian sphere attributes in step S4, each time the optimization is performed, the Gaussian sphere is projected and rasterized to obtain a rendered image, and the loss function and the gradient of the loss function are calculated. If the optimization stopping condition is met, the process proceeds to step S5, and the rendered image generated corresponding to the current Gaussian sphere attributes is used as a synthetic image of the road surface from a new perspective. If the optimization stopping condition is not met, the gradient descent method is used to optimize the attribute information of the Gaussian sphere except the rotation matrix, and the next optimization is performed.

[0032] In this embodiment, the process of projecting and rasterizing the Gaussian sphere is as follows: projecting the Gaussian sphere onto a two-dimensional image plane, and treating it as a "snowball" thrown at the image through the rasterization process, leaving diffusion traces to form a rendered image. Preferably, the Alpha Blending technology is used to fuse the projections of different Gaussian spheres during the rasterization process to ensure that the color and transparency information of the final image are accurate.

[0033] In this embodiment, in order to obtain a satisfactory rendering effect, during the Gaussian sphere attribute iterative optimization process, Gaussian sphere attribute optimization and Gaussian sphere density control are performed alternately. For example, after the Gaussian sphere attribute iterative optimization is performed for a preset number of times (e.g., 100 times), Gaussian sphere density control is performed once, and after the Gaussian sphere density control is performed, the Gaussian sphere attribute iterative optimization is continued, and the process is performed alternately. Preferably, the process of Gaussian sphere density control is as follows: when the small-scale Gaussian sphere splatting cannot cover the image, the corresponding Gaussian splatting method is used to fill it, and if it covers too much, it is divided into two.

[0034] Step S5, using the rendered image corresponding to the Gaussian sphere attributes generated when the iterative optimization stops as the new perspective synthetic image of the road surface, and outputting the new perspective synthetic image.

[0035] Existing point cloud rendering methods often suffer from problems such as loss of geometric information, blurred object contours, and inaccurate depth perception under new perspectives, especially when surface details (such as lane lines and road surfaces) need to be accurately presented. To solve this problem, the present invention introduces road trajectory information, obtains the trajectory direction of the road, and uses these road rules as the true value. Trajectory constraints are introduced during iterative optimization to restrict the rotation of each Gaussian sphere, which ensures that the Gaussian distribution can learn and maintain the accurate geometric information of the point cloud during the reconstruction process, so that after switching the perspective, the Gaussian sphere will not be misplaced or have a confused direction, making the road surface clearer in the new perspective. Figure 3The difference between the new perspective synthetic image obtained by the new perspective synthesis method of the road surface based on motion information and Gaussian sputtering provided by the present invention and the rendered image obtained by the traditional 3D Gaussian sputtering algorithm is shown. The left image is the rendered image obtained by the traditional 3D Gaussian sputtering algorithm, and the right image is the new perspective synthetic image obtained by the method of the present invention. It can be seen that the clarity of the left image (especially the framed part) is significantly lower than that of the right image.

[0036] In a preferred embodiment, in order to improve the rendering quality, during the iterative optimization process of the attribute information of the Gaussian sphere, the loss function is:

[0037]

[0038] in, Represents the real image I and the rendered image The average value of the pixel difference; Represents the real image I and the rendered image The structural loss; λ represents the first weight coefficient; α represents the second weight coefficient; L scale represents the vertical scale loss of the Gaussian sphere distribution, L represents the vertical scale of the mth Gaussian ball, and M represents the number of Gaussian balls. scale The size of the Gaussian sphere of the road is constrained in the normal direction of the road, ensuring the flatness of the road surface, thereby solving the problem of rough road surface when the viewpoint changes.

[0039] Specifically, S represents the number of pixels in the real image / rendered image, y s Represents the pixel value of the sth pixel of the real image, Represents the pixel value of the sth pixel of the rendered image.

[0040] Specifically,

[0041] Among them, μ1 and μ2 represent the mean pixel values ​​of the real image and the rendered image, σ1 and σ2 represent the variance of the pixel values ​​of the real image and the rendered image, respectively. 12 Represents the pixel value covariance between the real image and the rendered image, C1 represents the first stable constant, and C2 represents the second stable constant. C1=0.01 2 , C2=0.03 2 .

[0042] In this embodiment, the value range of λ is 0.2-0.4, preferably 0.3. The value range of α is 30-100, preferably 50. The real image is a real road surface image corresponding to the road surface point cloud position.

[0043] In a preferred embodiment, in step S2, obtaining the trajectory rotation matrix of each trajectory segment includes:

[0044] Step S21, determine the coordinates of the center of mass of each trajectory segment, obtain the vector of each trajectory point in each trajectory segment relative to the center of mass, and record it as the first vector.

[0045] If a certain trajectory includes n trajectory points, n is greater than or equal to 1, then the coordinates of the center of mass CP' are:

[0046]

[0047] The coordinates of the i-th track point in this track are (x i ,y i ,z i ).

[0048] If two adjacent trajectory points in a certain trajectory are CP i and CP i+1 , then the trajectory point CP i The first vector of Track point CP i+1 The first vector of

[0049] Step S22, determining the rotation quaternion of the two adjacent trajectory points by the first vectors of the two adjacent trajectory points. Further preferably, step S22 includes:

[0050] Step A, obtaining the angle between the first vectors of two adjacent trajectory points;

[0051] In one example, the trajectory point CP i and CP i+1 The angle θ between the first vectors of two adjacent trajectory points ij The cosine of is:

[0052]

[0053] j=i+1. Obtain θ by using the arccosine function ij .

[0054] Step B: normalize the first vector of two adjacent trajectory points.

[0055] In the above example, After normalization, After normalization,

[0056]

[0057] Step C, using the cross product of the normalized first vectors of two adjacent trajectory points to obtain the rotation axis vector, and normalizing the rotation axis vector.

[0058] In the above example, the trajectory point CP i and CP i+1 The rotation axis vector for:

[0059]

[0060] Rotation axis vector After normalization, we get

[0061]

[0062] Step D, obtaining the rotation quaternion of two adjacent trajectory points through the included angle and the normalized rotation axis vector.

[0063] In the above example, two adjacent trajectory points CP i and CP i+1 The rotation quaternion is q ij =[q w ,q x ,q y ,q z ], j=i+1:

[0064]

[0065] q w is the real part, q x ,q y ,q z are all imaginary parts, They are The x-axis coordinate, y-axis coordinate, and z-axis coordinate.

[0066] Step S23, obtaining the average rotation quaternion of each trajectory segment based on the rotation quaternions of all two adjacent trajectory points of each trajectory segment. Preferably, the average rotation quaternion q of each trajectory segment is obtained according to the following formula: avg :

[0067]

[0068] Where n represents the number of trajectory points in each trajectory, i and j represent trajectory point indexes, i∈[1,n-1], j=i+1; q ij Represents the rotation quaternion of adjacent trajectory point i and trajectory point j.

[0069] Step S24, converting the average rotation quaternion into a trajectory rotation matrix.

[0070] Specifically, step S24 includes:

[0071] Step a, calculate the average rotation quaternion q avg =[q aw ,q ax ,q ay ,q az ]’s norm:

[0072]

[0073] Step b, use the norm to normalize the average rotation quaternion q avg :

[0074]

[0075] Step c: Calculate the trajectory rotation matrix

[0076] R 11 =1-2(y' 2 +z' 2 )

[0077] R 12 =2(x'y'-z'ω')

[0078] R 13 =2(x'z'+y'ω')

[0079] R 21 =2(x'y'+z'ω')

[0080] R 22 =1-2(x' 2 +z' 2 )

[0081] R 23 =2(y'z'-x'ω')

[0082] R 31 =2(x'z'-y'ω')

[0083] R 32 =2(y'z'+x'ω')

[0084] R 33 =1-2(x' 2 +y' 2 )

[0085] The new road view synthesis method based on motion information and Gaussian sputtering disclosed in the present invention introduces motion information to adjust the Gaussian ball. During the rendering process, the position, shape and color of the 3D Gaussian ball are dynamically adjusted according to the motion information of the acquisition vehicle, such as the change of the view angle and the change of the path, so as to ensure that the synthesized image can timely reflect the overall direction of the road surface and improve the accuracy of rendering. It has good generalization ability and can adapt to different road conditions and observation environments. It is not only suitable for straight lanes, but also for uphill and downhill roads and curves, which improves the practicality and scope of application of the algorithm.

[0086] The present invention also discloses a new road view perspective synthesis device based on motion information and Gaussian sputtering, which is used to implement the new road view perspective synthesis method based on motion information and Gaussian sputtering described in the first aspect of the present invention, comprising:

[0087] Acquisition module, to obtain road surface point cloud and road surface trajectory point set;

[0088] The trajectory rotation matrix acquisition module processes the road trajectory point set in sections and obtains the trajectory rotation matrix of each section of the trajectory;

[0089] An initialization module generates a Gaussian sphere at each data point position of the road surface point cloud, and initializes the attribute information of the Gaussian sphere, wherein the trajectory rotation matrix of each trajectory segment is set to the rotation matrix of the Gaussian sphere generated by the point cloud corresponding to each trajectory segment;

[0090] Iterative optimization module, iteratively optimizes the attribute information of the Gaussian ball except the rotation matrix until the optimization stop condition is reached;

[0091] The road surface new perspective synthetic image acquisition module uses the rendered image corresponding to the Gaussian sphere attributes generated when the iterative optimization stops as the road surface new perspective synthetic image.

[0092] In this embodiment, the acquisition module, trajectory rotation matrix acquisition module, initialization module, iterative optimization module and road surface new perspective synthetic image acquisition module correspond one by one to steps S1, S2, S3, S4 and S5 of the above-mentioned road surface new perspective synthesis method based on motion information and Gaussian sputtering, and will not be repeated here.

[0093] The present invention also discloses a computer program product, including a computer program. When the computer program is executed by a processor, it implements the steps of the above-mentioned method for synthesizing a new road surface perspective based on motion information and Gaussian sputtering provided by the present invention. The computer program product should be understood as a software product that mainly implements its solution through a computer program, such as a program product integrated in the cloud or a software library.

[0094] The present invention also discloses an electronic device. In one embodiment, the electronic device includes at least one processor; and a memory connected to the at least one processor in communication; wherein:

[0095] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for synthesizing a new road surface perspective based on motion information and Gaussian sputtering provided by the present invention.

[0096] like Figure 4 , which is a schematic diagram of the structure of an electronic device for synthesizing a road surface new perspective based on motion information and Gaussian sputtering according to an embodiment of the present invention. The electronic device may include a processor 10, a memory 11, a communication bus 12, and a communication interface 13, and may also include a computer program stored in the memory 11 and executable on the processor 10, such as a program for synthesizing a road surface new perspective based on motion information and Gaussian sputtering.

[0097] Among them, the processor 10 may be composed of an integrated circuit in some embodiments, for example, it may be composed of a single packaged integrated circuit, or it may be composed of multiple integrated circuits packaged with the same function or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and a combination of various control chips, etc. The processor 10 is the control core (Control Unit) of the electronic device, and uses various interfaces and lines to connect various components of the entire electronic device, and executes or executes programs or modules stored in the memory 11 (for example, executing a new road view synthesis method based on motion information and Gaussian sputtering, etc.), and calls data stored in the memory 11 to execute various functions of the electronic device and process data.

[0098] The memory 11 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory (for example, SD or DX memory, etc.), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the memory 11 may be an internal storage unit of an electronic device, such as a mobile hard disk of the electronic device. In other embodiments, the memory 11 may also be an external storage device of an electronic device, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Further, the memory 11 may also include both an internal storage unit of the electronic device and an external storage device. The memory 11 can not only be used to store application software and various types of data installed in the electronic device, such as the code of the program of the new perspective synthesis method of the road surface based on motion information and Gaussian sputtering, but also can be used to temporarily store data that has been output or is to be output.

[0099] The communication bus 12 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is configured to realize connection and communication between the memory 11 and at least one processor 10, etc.

[0100] The communication interface 13 is used for communication between the above-mentioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device and other electronic devices. The user interface may be a display (Display), an input unit (such as a keyboard (Keyboard)), and optionally, the user interface may also be a standard wired interface, a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode, organic light-emitting diode) touch device, etc. Among them, the display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device and to display a visual user interface.

[0101] Figure 4 Only an electronic device with components is shown, and those skilled in the art will understand that Figure 4 The structure shown does not constitute a limitation on the electronic device, and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0102] For example, although not shown, the electronic device may also include a power source (such as a battery) for supplying power to various components. Preferably, the power source may be logically connected to at least one processor 10 through a power management device, so that the power management device can realize functions such as charging management, discharging management, and power consumption management. The power source may also include any components such as one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, and power status indicators. The electronic device may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0103] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited by this structure.

[0104] Furthermore, if the module / unit integrated in the electronic device 1 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).

[0105] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", "an implementation", "a preferred implementation" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0106] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. A new road view synthesis method based on motion information and Gaussian sputtering, characterized in that: include: Obtain road surface point cloud and road surface trajectory point set; The road surface trajectory point set is processed in segments to obtain the trajectory rotation matrix of each segment of the trajectory; Generate a Gaussian sphere at each data point position of the road surface point cloud, initialize the attribute information of the Gaussian sphere, wherein the trajectory rotation matrix of each trajectory segment is set to the rotation matrix of the Gaussian sphere generated by the point cloud corresponding to each trajectory segment; Iteratively optimize the attribute information of the Gaussian ball except the rotation matrix until the optimization stop condition is reached; The rendered image corresponding to the Gaussian sphere attributes when the iterative optimization stops is used as the new perspective synthetic image of the road surface.

2. The method for synthesizing a new road view based on motion information and Gaussian sputtering according to claim 1, characterized in that: The step of obtaining the trajectory rotation matrix of each trajectory segment includes: Determine the coordinates of the center of mass of each trajectory segment, and obtain the vector of each trajectory point in each trajectory segment relative to the center of mass, recorded as the first vector; Determine the rotation quaternion of the two adjacent trajectory points by using the first vectors of the two adjacent trajectory points; The average rotation quaternion of each trajectory is obtained based on the rotation quaternions of all two adjacent trajectory points of each trajectory; Convert the average rotation quaternion to a trajectory rotation matrix.

3. The method for synthesizing a new road view based on motion information and Gaussian sputtering according to claim 2, characterized in that: The step of determining the rotation quaternion of the two adjacent trajectory points by using the first vectors of the two adjacent trajectory points comprises: Get the angle between the first vectors of two adjacent trajectory points; Normalize the first vector of two adjacent trajectory points; The rotation axis vector is obtained by cross-producting the normalized first vectors of two adjacent trajectory points, and the rotation axis vector is normalized; The rotation quaternion of two adjacent trajectory points is obtained through the included angle and the normalized rotation axis vector.

4. The method for synthesizing a new road view based on motion information and Gaussian sputtering according to claim 2, characterized in that: The average rotation quaternion q of each trajectory is obtained according to the following formula avg : Where n represents the number of trajectory points in each trajectory, i, j represent the trajectory point index, i∈[1,N-1], J=I+1; q ij represents the rotation quaternion of adjacent trajectory point i and trajectory point j.

5. The method for synthesizing a new road view based on motion information and Gaussian sputtering according to any one of claims 1 to 4, characterized in that: During the iterative optimization process of Gaussian sphere properties, Gaussian sphere property optimization and Gaussian sphere density control are performed alternately.

6. The method for synthesizing a new road view based on motion information and Gaussian sputtering according to claim 5, characterized in that: In the iterative optimization process of Gaussian sphere attributes, each time the optimization is performed, the Gaussian sphere is projected and rasterized to obtain a rendered image, and the loss function is calculated. If the optimization stop condition is met, the rendered image corresponding to the current Gaussian sphere attribute is used as the synthetic image of the new road surface perspective. If the optimization stop condition is not met, the attribute information of the Gaussian sphere except the rotation matrix is ​​optimized using the gradient descent method, and the next optimization is performed. The loss function is: in, Represents the real image I and the rendered image The average value of the pixel difference; Represents the real image I and the rendered image The structural loss; λ represents the first weight coefficient; α represents the second weight coefficient; L scale represents the vertical scale loss of the Gaussian sphere distribution, scale φm represents the vertical scale of m Gaussian balls, and M represents the number of Gaussian balls.

7. The method for synthesizing a new road view based on motion information and Gaussian sputtering according to claim 1, 2, 3, 4, or 6, characterized in that: The attribute information of the Gaussian sphere includes rotation matrix, scaling matrix, opacity and spherical harmonics; The attribute information of the initialized Gaussian ball includes: randomly initializing the scaling matrix S and constructing the covariance matrix Σ of the Gaussian ball: S = RSS T R T Where R represents the rotation matrix of the Gaussian sphere.

8. A device for synthesizing a new road view based on motion information and Gaussian sputtering, used to implement the method for synthesizing a new road view based on motion information and Gaussian sputtering according to any one of claims 1 to 7, characterized in that: include: Acquisition module, to obtain road surface point cloud and road surface trajectory point set; The trajectory rotation matrix acquisition module processes the road trajectory point set in sections and obtains the trajectory rotation matrix of each section of the trajectory; An initialization module generates a Gaussian sphere at each data point position of the road surface point cloud, and initializes the attribute information of the Gaussian sphere, wherein the trajectory rotation matrix of each trajectory segment is set to the rotation matrix of the Gaussian sphere generated by the point cloud corresponding to each trajectory segment; Iterative optimization module, iteratively optimizes the attribute information of the Gaussian ball except the rotation matrix until the optimization stop condition is reached; The road surface new perspective synthetic image acquisition module uses the rendered image corresponding to the Gaussian sphere attributes generated when the iterative optimization stops as the road surface new perspective synthetic image.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. An electronic device, characterized in that: The electronic device comprises: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for synthesizing a new perspective of a road surface based on motion information and Gaussian sputtering as described in any one of claims 1 to 7.

Images