Methods, systems, equipment, and storage media for improving the quality of unmanned vehicle landscape scanning videos.

Abstract

This invention discloses a method, system, device, and storage medium for improving the quality of unmanned vehicle scene scanning videos. The method involves extracting frames from a video to obtain multiple images; segmenting each dynamic object image from each image to obtain a main subject image; extracting first feature points from each main subject image based on a pre-set first algorithm and determining the pose information of each main subject image; training multiple main subject images and their corresponding pose information using 3DGS to obtain a scene rendering model; constructing a bounding sphere that encloses the scene rendering model; adding multiple second feature points to the surface of the bounding sphere; and parametrically training the scene rendering model based on these second feature points to obtain an optimized scene rendering model. This improves the scene rendering quality and efficiency in the video, thereby enhancing the autonomous driving performance of the unmanned vehicle.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method, system, device, and storage medium for improving the quality of unmanned vehicle scene scanning videos. Background Technology

[0002] Currently, autonomous vehicles need to use image acquisition devices such as cameras to capture external scenes and model them during autonomous driving. Then, autonomous driving algorithms are used to simulate driving in these scenes before the autonomous vehicle's route can be determined. However, the existing scene modeling and rendering quality for videos is poor, which affects the simulation effect of autonomous driving algorithms. Summary of the Invention

[0003] The purpose of this invention is to address the technical problems existing in the background art by proposing a method, system, device, and storage medium for improving the quality of unmanned vehicle scene scanning videos.

[0004] To achieve the above-mentioned technical objectives, the technical solution adopted by the present invention is as follows:

[0005] A first implementation of the first aspect of the present invention provides a method for improving the quality of unmanned vehicle scene scanning videos, comprising:

[0006] S101. Perform frame extraction on the video to obtain multiple images;

[0007] S102. Segment each dynamic object image in each image to obtain multiple subject images;

[0008] S103, and based on the preset first algorithm, extract the first feature points of each subject image and determine the pose information of each subject image;

[0009] S104. Train multiple subject images and their corresponding pose information using 3DGS to obtain a scene rendering model.

[0010] S105. Construct a bounding sphere that encloses the scene rendering model, and add multiple second feature points on the surface of the bounding sphere. Based on the multiple second feature points, perform parameterized training on the scene rendering model to obtain an optimized scene rendering model.

[0011] Optionally, in a second implementation of the first aspect of the present invention, S101 includes:

[0012] S1011. Based on the preset wheel speed meter and preset GPS positioning information in the unmanned vehicle, the video is processed by frame extraction at one-meter intervals to obtain multiple images.

[0013] Optionally, in a third implementation of the first aspect of the present invention, S102 includes:

[0014] S1021. A pre-set YOLOv6 network is used to identify and track each dynamic object in each image, and to extract the image of each dynamic object, wherein the dynamic object image is the image of the object in motion.

[0015] S1022. Classify multiple dynamic object images and summarize them according to the same type of object to obtain multiple dynamic object datasets.

[0016] Optionally, in a fourth implementation of the first aspect of the present invention, S103 includes:

[0017] S1031. Based on the preset SFM algorithm, extract the first feature point of each subject image and match each first feature point to determine the relationship between each subject image;

[0018] S1032. Perform pose estimation on each first feature point, GPS positioning information, and preset IMU information to determine the pose information of each subject image.

[0019] Optionally, in a fifth implementation of the first aspect of the present invention, S105 includes:

[0020] S1051. Calculate the size of the preset scene bounding box and construct a bounding sphere with a size greater than a specified multiple of the scene bounding box, wherein the bounding sphere encloses the scene rendering model;

[0021] S1052. Add 360 second feature points evenly on the surface of the sphere surrounding the sphere.

[0022] Optionally, in a sixth implementation of the first aspect of the invention, the size of the bounding sphere is greater than three times the size of the scene bounding box.

[0023] Optionally, in a seventh implementation of the first aspect of the invention, the parameterized training includes deep learning of spherical harmonic functions, rotation matrices, scale, and opacity.

[0024] A first implementation of the second aspect of the present invention provides a system for improving the quality of unmanned vehicle scene scanning videos, comprising:

[0025] The frame extraction module is used to extract frames from a video to obtain multiple images;

[0026] The first extraction module is used to segment each dynamic object image in each image to obtain multiple subject images;

[0027] The second extraction module is used to extract the first feature points of each subject image based on a preset first algorithm, and to determine the pose information of each subject image.

[0028] The training module is used to train multiple subject images and their corresponding pose information using 3DGS to obtain a scene rendering model.

[0029] The optimization module is used to construct a bounding sphere that encloses the scene rendering model, and add multiple second feature points to the surface of the bounding sphere. Based on the multiple second feature points, the scene rendering model is parametrically trained to obtain an optimized scene rendering model.

[0030] A first implementation of the third aspect of the present invention provides an unmanned vehicle scene scanning video quality improvement device, the unmanned vehicle scene scanning video quality improvement device comprising: a memory and at least one processor, the memory storing instructions, the memory and the at least one processor being interconnected via a circuit;

[0031] The at least one processor invokes the instructions in the memory to cause the unmanned vehicle scene scanning video quality improvement device to execute the unmanned vehicle scene scanning video quality improvement method as described in any one of the first aspects of the present invention.

[0032] A first implementation of the fourth aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the unmanned vehicle scene scanning video quality improvement method as described in any one of the first aspects of the present invention.

[0033] Compared with existing technologies, this invention has the following beneficial technical effects: It extracts frames from a video to obtain multiple images; it segments each dynamic object image in each image to obtain a main image; based on a preset first algorithm, it extracts the first feature points of each main image and determines the pose information of each main image; it trains multiple main images and their corresponding pose information using 3DGS to obtain a scene rendering model; it constructs a bounding sphere that encloses the scene rendering model, and adds multiple second feature points to the surface of the bounding sphere; based on these multiple second feature points, it performs parameterized training on the scene rendering model to obtain an optimized scene rendering model, thereby improving the quality and efficiency of scene rendering in the video, and thus enhancing the autonomous driving effect of unmanned vehicles. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the first embodiment of the unmanned vehicle scene scanning video quality improvement method in this invention;

[0035] Figure 2 This is a schematic diagram of the second embodiment of the unmanned vehicle scene scanning video quality improvement method in this invention;

[0036] Figure 3This is a schematic diagram of the third embodiment of the unmanned vehicle scene scanning video quality improvement method in the present invention;

[0037] Figure 4 This is a schematic diagram of the fourth embodiment of the unmanned vehicle scene scanning video quality improvement method in this invention;

[0038] Figure 5 This is a schematic diagram of the fifth embodiment of the unmanned vehicle scene scanning video quality improvement method in this invention;

[0039] Figure 6 This is a schematic diagram of one embodiment of the unmanned vehicle scene scanning video quality improvement system of the present invention;

[0040] Figure 7 This is a schematic diagram of one embodiment of the unmanned vehicle scene scanning video quality improvement device in the present invention;

[0041] Figure 8 This is a flowchart illustrating the process of improving the quality of unmanned vehicle scene scanning video in an embodiment of the present invention. Detailed Implementation

[0042] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0043] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are used only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or component referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, features defined with "first," "second," etc., may explicitly or implicitly include one or more features. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0044] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or a specific connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal communication between two components. Those skilled in the art will understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0045] For ease of understanding, the specific process of the embodiments of the present invention is described below. Please refer to [link / reference]. Figures 1-5 as well as Figure 8 The method for improving the quality of unmanned vehicle scene scanning videos in this embodiment of the invention is used for road detection. The method includes:

[0046] S101. Perform frame extraction on the video to obtain multiple images;

[0047] Furthermore, S101 can also specifically perform the following:

[0048] S1011. Based on the preset wheel speed meter and preset GPS positioning information in the unmanned vehicle, the video is processed by frame extraction at one-meter intervals to obtain multiple images.

[0049] In this embodiment, the video is captured by an unmanned vehicle shooting street scenes. Therefore, when performing frame extraction on the video, the wheel speed sensor and the built-in GPS positioning information on the unmanned vehicle are used for frame extraction. This allows each video frame to record the current wheel speed and GPS positioning information, which serves as the basis for subsequent 3D scene construction. This enables the acquisition of continuous and uniformly distributed street scene images, providing high-quality raw data for subsequent image processing and scene reconstruction.

[0050] S102. Segment each dynamic object image in each image to obtain multiple subject images;

[0051] Furthermore, S102 can also specifically perform:

[0052] S1021. A pre-set YOLOv6 network is used to identify and track each dynamic object in each image, and to extract the image of each dynamic object, wherein the dynamic object image is the image of the object in motion.

[0053] S1022. Classify multiple dynamic object images and summarize them according to the same type of object to obtain multiple dynamic object datasets.

[0054] In this embodiment, the images contain a large amount of information, which needs to be classified and organized. Therefore, the YOLOv6 network is used to identify and track moving objects in each frame, and to extract and integrate images of similar objects, such as the same car, the same bicycle, the same pedestrian, etc. It should be noted that these dynamic object images do not participate in the subsequent 3DGS training and need to be segmented in advance so that only scene images are trained later, reducing the training load and improving optimization efficiency.

[0055] S103, and based on the preset first algorithm, extract the first feature points of each subject image and determine the pose information of each subject image;

[0056] Furthermore, S103 can also specifically perform:

[0057] S1031. Based on the preset SFM algorithm, extract the first feature point of each subject image and match each first feature point to determine the relationship between each subject image;

[0058] S1032. Perform pose estimation on each first feature point, GPS positioning information, and preset IMU information to determine the pose information of each subject image.

[0059] S104. Train multiple subject images and their corresponding pose information using 3DGS to obtain a scene rendering model.

[0060] In this embodiment, the colmap tool is used to process multiple subject images and their corresponding pose information to obtain multiple 3D point clouds. Based on these point clouds, corresponding Gaussian spheres are initialized. A Gaussian sphere is placed at each point cloud position, with the center point position set as the point cloud position and the others randomly initialized.

[0061] Gaussian spheres are splatted onto the image, rasterized rendering is performed in the overlapping projection areas, and the alphablending algorithm is used to blend these Gaussian spheres. The process is differentiable, and the multidimensional coefficients of each Gaussian sphere are updated.

[0062] S105. Construct a bounding sphere that encloses the scene rendering model, and add multiple second feature points on the surface of the bounding sphere. Based on the multiple second feature points, perform parameterized training on the scene rendering model to obtain an optimized scene rendering model.

[0063] Furthermore, S105 can also specifically perform the following:

[0064] S1051. Calculate the size of the preset scene bounding box and construct a bounding sphere with a size greater than a specified multiple of the scene bounding box, wherein the bounding sphere encloses the scene rendering model;

[0065] S1052. Add 360 second feature points evenly on the surface of the sphere surrounding the sphere.

[0066] In this embodiment, the size of the bounding sphere is more than three times the size of the scene bounding box. The parameterized training includes deep learning of spherical harmonic functions, rotation matrices, scale, and opacity, which enhances the realism and integrity of the scene. At the same time, by learning parameters such as spherical harmonic functions and rotation matrices, the model's adaptability to complex lighting and viewpoint changes is further improved.

[0067] The video is processed by frame extraction to obtain multiple images; each dynamic object image in each image is segmented to obtain the main image; and based on a preset first algorithm, the first feature points of each main image are extracted, and the pose information of each main image is determined; multiple main images and their corresponding pose information are trained using 3DGS to obtain a scene rendering model; a bounding sphere is constructed to enclose the scene rendering model, and multiple second feature points are added to the surface of the bounding sphere; based on the multiple second feature points, the scene rendering model is parametrically trained to obtain an optimized scene rendering model, thereby improving the scene rendering quality and efficiency in the video, and thus improving the autonomous driving effect of the unmanned vehicle.

[0068] The above describes the method for improving the quality of unmanned vehicle scene scanning video in the embodiments of the present invention. The following describes the system for improving the quality of unmanned vehicle scene scanning video in the embodiments of the present invention. Please refer to [link / reference]. Figure 6 The unmanned vehicle scene scanning video quality improvement system includes:

[0069] The frame extraction module 201 is used to perform frame extraction processing on the video to obtain multiple images;

[0070] The first extraction module 202 is used to segment each dynamic object image in each image to obtain multiple subject images;

[0071] The second extraction module 203 is used to extract the first feature points of each subject image based on a preset first algorithm, and to determine the pose information of each subject image.

[0072] Training module 204 is used to train multiple subject images and their corresponding pose information using 3DGS to obtain a scene rendering model.

[0073] The optimization module 205 is used to construct a bounding sphere that encloses the scene rendering model, and add multiple second feature points on the surface of the bounding sphere. Based on the multiple second feature points, the scene rendering model is parametrically trained to obtain an optimized scene rendering model.

[0074] Specifically, the process involves frame extraction from the video to obtain multiple images; segmenting each dynamic object image within each image to obtain the main subject image; extracting the first feature points of each main subject image based on a pre-set first algorithm and determining the pose information of each main subject image; training multiple main subject images and their corresponding pose information using 3DGS to obtain a scene rendering model; constructing a bounding sphere that encloses the scene rendering model and adding multiple second feature points to the surface of the bounding sphere; and parametrically training the scene rendering model based on these second feature points to obtain an optimized scene rendering model. This improves the quality and efficiency of scene rendering in the video, thereby enhancing the autonomous driving performance of unmanned vehicles.

[0075] The frame extraction module 201 can also be specifically used for:

[0076] Based on the pre-set wheel speed meter and pre-set GPS positioning information in the unmanned vehicle, the video is processed by frame extraction at one-meter intervals to obtain multiple images.

[0077] The roads include main roads and non-main roads, and after the identification module 202, it also includes:

[0078] Based on a preset road segmentation algorithm, the roads in the image unit are segmented and filtered to obtain the main roads and traffic type information on the main roads.

[0079] The first extraction module 202 can also be specifically used for:

[0080] A pre-built YOLOv6 network is used to identify and track each dynamic object in each image, and to extract the image of each dynamic object. The object image is the image of the object in motion.

[0081] Multiple dynamic object images are classified and then summarized according to the same type of object to obtain multiple dynamic object datasets.

[0082] The second extraction module 203 can also be specifically used for:

[0083] Based on the preset SFM algorithm, the first feature point of each subject image is extracted, and each first feature point is matched to determine the relationship between each subject image;

[0084] Pose estimation is performed on each first feature point, GPS positioning information, and preset IMU information to determine the pose information of each subject image.

[0085] The optimization module 205 can also be specifically used for:

[0086] Calculate the size of the preset scene bounding box, and construct a bounding sphere whose size is a multiple of the scene bounding box. The bounding sphere encloses the scene rendering model.

[0087] 360 second feature points are uniformly added to the surface of the sphere surrounding the sphere.

[0088] The bounding sphere is larger than three times the size of the scene bounding box, and the parameterized training includes deep learning of spherical harmonics, rotation matrices, scale, and opacity.

[0089] above Figure 6The unmanned vehicle scene scanning video quality improvement system in this embodiment of the invention is described in detail from the perspective of modular functional entities. The unmanned vehicle scene scanning video quality improvement device in this embodiment of the invention is described in detail from the perspective of hardware processing.

[0090] Figure 7 This is a schematic diagram of the structure of an unmanned vehicle (UAV) landscape video quality enhancement device 300 provided in an embodiment of the present invention. The UAV landscape video quality enhancement device 300 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 310 (e.g., one or more processors) and a memory 320, and one or more storage media 330 (e.g., one or more mass storage devices) for storing application programs 333 or data 332. The memory 320 and storage media 330 can be temporary or persistent storage. The program stored in the storage media 330 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the UAV landscape video quality enhancement device 300. Furthermore, the processor 310 may be configured to communicate with the storage media 330 and execute the series of instruction operations in the storage media 330 on the UAV landscape video quality enhancement device 300.

[0091] The unmanned vehicle-based scene scanning video quality improvement device 300 may also include one or more power supplies 340, one or more wired or wireless network interfaces 350, one or more input / output interfaces 360, and / or one or more operating systems 331, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will understand that... Figure 3 The illustrated structure of the unmanned vehicle scene scanning video quality improvement device does not constitute a limitation on communication protocol devices based on local area network projection. It may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0092] The present invention also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein the computer-readable storage medium stores instructions that, when the instructions are executed on a computer, cause the computer to perform the steps of the unmanned vehicle scene scanning video quality improvement method.

[0093] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0094] The above describes a method or multiple implementations for improving the quality of unmanned vehicle-based scene scanning videos, but it is not intended that the specific implementation of this invention is limited to these descriptions. Any methods or structures that are similar to or identical to those of this invention, or any technical deductions or substitutions made based on the concept of this invention, should be considered within the scope of protection of this invention.

Claims

1. A method for improving the quality of unmanned vehicle scene scanning videos, characterized in that, include: S101. Perform frame extraction on the video to obtain multiple images; S102. Segment each dynamic object image in each of the images to obtain multiple subject images; S103. Based on a preset first algorithm, extract the first feature points of each subject image to determine the pose information of each subject image; S104. Perform 3DGS training on multiple subject images and their corresponding pose information to obtain a scene rendering model; S105. Construct a bounding sphere that encloses the scene rendering model, and add multiple second feature points on the surface of the bounding sphere. Based on the multiple second feature points, perform parameterized training on the scene rendering model to obtain an optimized scene rendering model. Wherein, S103 includes: S1031. Based on the preset SFM algorithm, extract the first feature point of each of the main images, and match each of the first feature points to determine the relationship between each of the main images; S1032. Perform pose estimation on each of the first feature points, GPS positioning information, and preset IMU information to determine the pose information of each of the main images.

2. The method for improving the quality of unmanned vehicle scene scanning video according to claim 1, characterized in that, S101 includes: S1011. Based on the preset wheel speed meter and preset GPS positioning information in the unmanned vehicle, the video is processed by frame extraction at intervals of one meter traveled by the unmanned vehicle to obtain multiple images.

3. The method for improving the quality of unmanned vehicle scene scanning video according to claim 2, characterized in that, S102 includes: S1021. A preset YOLOv6 network is used to identify and track each dynamic object in each of the images, and to extract the image of each dynamic object, wherein the dynamic object image is an image of an object in motion. S1022. Classify the multiple dynamic object images and summarize them according to the same type of object to obtain multiple dynamic object datasets.

4. The method for improving the quality of unmanned vehicle scene scanning video according to claim 1, characterized in that, S105 includes: S1051. Calculate the size of the preset scene bounding box, and construct a bounding sphere with a size larger than a specified multiple of the scene bounding box, wherein the bounding sphere encloses the scene rendering model; S1052. 360 second feature points are uniformly added to the surface of the sphere surrounding the sphere.

5. The method for improving the quality of unmanned vehicle scene scanning video according to claim 4, characterized in that, The size of the bounding sphere is more than three times the size of the scene bounding box.

6. The method for improving the quality of unmanned vehicle scene scanning video according to claim 4, characterized in that, The parametric training includes deep learning of spherical harmonic functions, rotation matrices, scale, and opacity.

7. A system for improving the quality of unmanned vehicle scene scanning videos, characterized in that, include: The frame extraction module is used to extract frames from the video to obtain multiple images; The first extraction module is used to segment each dynamic object image in each of the images to obtain multiple subject images; The second extraction module is used to extract the first feature points of each subject image based on a preset first algorithm, so as to determine the pose information of each subject image; The training module is used to train multiple subject images and their corresponding pose information using 3DGS to obtain a scene rendering model. An optimization module is used to construct a bounding sphere that encloses the scene rendering model, and add multiple second feature points on the surface of the bounding sphere. Based on the multiple second feature points, the scene rendering model is parametrically trained to obtain an optimized scene rendering model. Specifically, the second extraction module is used for: Based on a preset SFM algorithm, the first feature point of each subject image is extracted, and each first feature point is matched to determine the relationship between each subject image; Pose estimation is performed on each of the first feature points, GPS positioning information, and preset IMU information to determine the pose information of each subject image.

8. A device for improving the quality of unmanned vehicle scene scanning videos, characterized in that, The unmanned vehicle scene scanning video quality improvement device includes: a memory and at least one processor, wherein the memory stores instructions, and the memory and the at least one processor are interconnected via a circuit; The at least one processor invokes the instructions in the memory to cause the unmanned vehicle scene scanning video quality improvement device to execute the unmanned vehicle scene scanning video quality improvement method as described in any one of claims 1-6.

9. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by the processor, it implements the unmanned vehicle scene scanning video quality improvement method as described in any one of claims 1-6.

Images