Unmanned vehicle scene scanning video quality improving method, system and device and storage medium

Abstract

The invention discloses an unmanned vehicle scene scanning video quality improvement method, system and device and a storage medium, and the method is used for carrying out the frame extraction of a video, and obtaining a plurality of images. Segmenting each dynamic object image in each image to obtain a main body image; based on a preset first algorithm, extracting a first feature point of each main body image, and determining pose information of each main body image; performing 3DGS training on the plurality of main body images and the corresponding pose information to obtain a scene rendering model; according to the method, a surrounding ball wrapping a scene rendering model is constructed, a plurality of second feature points are added on the ball surface of the surrounding ball, and the scene rendering model is subjected to parametric training according to the plurality of second feature points to obtain an optimized scene rendering model, so that the scene rendering quality and efficiency in a video are improved, and the automatic driving effect of an unmanned vehicle is improved.

Description

[0001] Technology Neighborhood

[0002] The present invention relates to the field of image processing technology, and in particular to a method, system, device and storage medium for improving the quality of scene scanning videos of unmanned vehicles. Background Art

[0003] At present, when unmanned vehicles are driving autonomously, they need to use image acquisition devices such as cameras to capture external scenes, model the scenes, and use autonomous driving algorithms to model the scenes for driving simulation. Finally, the driving route of the unmanned vehicle can be determined. However, the existing mid-scene modeling and rendering quality for videos is poor, which affects the simulation effect of the autonomous driving algorithm. Summary of the invention

[0004] The purpose of the present invention is to address the technical problems existing in the background technology and to propose a method, system, device and storage medium for improving the quality of scene scanning video of an unmanned vehicle.

[0005] In order to achieve the above technical objectives, the technical solution adopted by the present invention is as follows:

[0006] A first implementation of the first aspect of the present invention provides a method for improving the quality of scene scanning videos of an unmanned vehicle, comprising:

[0007] S101, extracting frames from the video to obtain multiple images;

[0008] S102, segmenting each dynamic object image in each image to obtain a plurality of subject images;

[0009] S103, extracting first feature points of each subject image based on a preset first algorithm, and determining position and posture information of each subject image;

[0010] S104, performing 3DGS training on multiple subject images and corresponding pose information to obtain a scene rendering model;

[0011] S105, constructing a bounding sphere that encloses the scene rendering model, and adding a plurality of second feature points on the surface of the bounding sphere, and performing parameterized training on the scene rendering model according to the plurality of second feature points to obtain an optimized scene rendering model.

[0012] Optionally, in a second implementation of the first aspect of the present invention, S101 includes:

[0013] S1011. According to the preset wheel speed meter and preset GPS positioning information in the unmanned vehicle, every meter traveled by the unmanned vehicle is used as a cropping interval, and frame extraction is performed on the video to obtain multiple images.

[0014] Optionally, in a third implementation of the first aspect of the present invention, S102 includes:

[0015] S1021, using a preset YOLOv6 network to identify and track each dynamic object in each image, and extracting each dynamic object image, wherein the dynamic object image is an image of an object in motion;

[0016] S1022: Classify the multiple dynamic object images, and aggregate them according to the same type of objects to obtain multiple dynamic object data sets.

[0017] Optionally, in a fourth implementation of the first aspect of the present invention, S103 includes:

[0018] S1031, extracting the first feature point of each subject image based on a preset SFM algorithm, and matching each first feature point to determine the relationship between each subject image;

[0019] S1032, performing pose estimation on each first feature point, GPS positioning information, and preset IMU information to determine pose information of each subject image.

[0020] Optionally, in a fifth implementation of the first aspect of the present invention, S105 includes:

[0021] S1051, calculating the size of a preset scene bounding box, and constructing a bounding sphere whose size is greater than a specified multiple of the scene bounding box, wherein the bounding sphere encloses the scene rendering model;

[0022] S1052. Evenly add 360 second feature points on the surface of the sphere surrounding the sphere.

[0023] Optionally, in a sixth implementation manner of the first aspect of the present invention, the size of the bounding sphere is greater than three times the size of the scene bounding box.

[0024] Optionally, in a seventh implementation of the first aspect of the present invention, parameterized training includes deep learning of spherical harmonics, rotation matrices, scales, and opacity.

[0025] A first implementation of the second aspect of the present invention provides a system for improving the quality of scene scanning videos of an unmanned vehicle, comprising:

[0026] A frame extraction module is used to extract frames from the video to obtain multiple images;

[0027] A first extraction module is used to segment each dynamic object image in each image to obtain a plurality of subject images;

[0028] A second extraction module, configured to extract first feature points of each subject image based on a preset first algorithm, and determine position and posture information of each subject image;

[0029] A training module is used to perform 3DGS training on multiple subject images and corresponding pose information to obtain a scene rendering model;

[0030] The optimization module is used to construct a bounding sphere that wraps the scene rendering model, and add multiple second feature points on the surface of the bounding sphere. According to the multiple second feature points, the scene rendering model is parameterized and trained to obtain an optimized scene rendering model.

[0031] A first implementation of the third aspect of the present invention provides a device for improving the quality of scene scanning videos of an unmanned vehicle, the device comprising: a memory and at least one processor, the memory storing instructions, the memory and the at least one processor being interconnected via a line;

[0032] The at least one processor calls the instructions in the memory so that the unmanned vehicle scene scanning video quality improvement device executes the unmanned vehicle scene scanning video quality improvement method as described in any one of the first aspects of the present invention.

[0033] A first implementation method of the fourth aspect of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method for improving the quality of unmanned vehicle scene scanning video as described in any one of the first aspects of the present invention is implemented.

[0034] Compared with the prior art, the present invention has the following beneficial technical effects: extracting frames from a video to obtain a plurality of images; segmenting each dynamic object image in each image to obtain a main image; and based on a preset first algorithm, extracting a first feature point of each main image, and determining the position and posture information of each main image; performing 3DGS training on the plurality of main images and the corresponding position and posture information to obtain a scene rendering model; constructing an enclosing sphere that wraps the scene rendering model, and adding a plurality of second feature points on the surface of the enclosing sphere, and performing parameterized training on the scene rendering model according to the plurality of second feature points to obtain an optimized scene rendering model, thereby improving the quality and efficiency of scene rendering in the video, and thereby improving the automatic driving effect of the unmanned vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 Schematic diagram of a first embodiment of a method for improving the quality of scene scanning video of an unmanned vehicle in an embodiment of the present invention;

[0036] Figure 2 Schematic diagram of a second embodiment of the method for improving the quality of scene scanning video of an unmanned vehicle in an embodiment of the present invention;

[0037] Figure 3 Schematic diagram of a third embodiment of the method for improving the quality of scene scanning video of an unmanned vehicle in an embodiment of the present invention;

[0038] Figure 4 Schematic diagram of a fourth embodiment of the method for improving the quality of scene scanning video of an unmanned vehicle in an embodiment of the present invention;

[0039] Figure 5 Schematic diagram of a fifth embodiment of the method for improving the quality of scene scanning video of an unmanned vehicle in an embodiment of the present invention;

[0040] Figure 6 A schematic diagram of an embodiment of a system for improving the quality of scene scanning video of an unmanned vehicle in an embodiment of the present invention;

[0041] Figure 7 It is a schematic diagram of an embodiment of a device for improving the quality of scene scanning video of an unmanned vehicle in an embodiment of the present invention;

[0042] Figure 8 4 is a flowchart of an execution of a method for improving the quality of scene scanning video of an unmanned vehicle in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0044] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or component referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and the like are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first", "second", and the like may explicitly or implicitly include one or more features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0045] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or a specific connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood by specific circumstances.

[0046] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1-Figure 5 as well as Figure 8 In an embodiment of the present invention, a method for improving the quality of a scene-scanning video of an unmanned vehicle is used to detect a road. The method for improving the quality of a scene-scanning video of an unmanned vehicle comprises:

[0047] S101, extracting frames from the video to obtain multiple images;

[0048] Furthermore, S101 may also specifically execute:

[0049] S1011. According to the preset wheel speed meter and preset GPS positioning information in the unmanned vehicle, every meter traveled by the unmanned vehicle is used as a cropping interval, and frame extraction is performed on the video to obtain multiple images.

[0050] In this embodiment, the video is obtained by shooting street scenes by an unmanned vehicle. Therefore, when extracting frames from the video, the wheel speed meter on the unmanned vehicle and the built-in GPS positioning information are used for frame extraction, so that each frame of the video image can record the current wheel speed and GPS positioning information, and as the basis for subsequent three-dimensional scene construction, continuous and evenly distributed street scene images can be obtained, providing high-quality original data for subsequent image processing and scene reconstruction.

[0051] S102, segmenting each dynamic object image in each image to obtain a plurality of subject images;

[0052] Furthermore, S102 may further specifically execute:

[0053] S1021, using a preset YOLOv6 network to identify and track each dynamic object in each image, and extracting each dynamic object image, wherein the dynamic object image is an image of an object in motion;

[0054] S1022: Classify the multiple dynamic object images, and aggregate them according to the same type of objects to obtain multiple dynamic object data sets.

[0055] In this embodiment, the image contains a lot of information, which needs to be classified and organized. Therefore, the YOLOv6 network is used to identify and track the object images in motion in each frame of the image, and extract and integrate the same object images, such as the same car, the same bicycle, the same pedestrian, etc. It should be noted that these dynamic object images do not participate in the subsequent 3DGS training and need to be segmented in advance so that only the scene images can be trained later, reducing the amount of training and improving the optimization efficiency.

[0056] S103, extracting first feature points of each subject image based on a preset first algorithm, and determining position and posture information of each subject image;

[0057] Furthermore, S103 may further specifically execute:

[0058] S1031, extracting the first feature point of each subject image based on a preset SFM algorithm, and matching each first feature point to determine the relationship between each subject image;

[0059] S1032, performing pose estimation on each first feature point, GPS positioning information, and preset IMU information to determine pose information of each subject image.

[0060] S104, performing 3DGS training on multiple subject images and corresponding pose information to obtain a scene rendering model;

[0061] In this embodiment, the colmap tool is used to process multiple subject images and corresponding posture information to obtain multiple 3D point clouds, and the corresponding Gaussian spheres are initialized based on these point clouds. A Gaussian sphere is placed at each point cloud position, the center point position is set as the point cloud position, and the others are randomly initialized.

[0062] The Gaussian spheres are splatted onto the image, rasterized in the overlapping projection areas, and the alpha alpha blending algorithm is used to blend the Gaussian spheres in a differentiable process, and the multidimensional coefficients of each Gaussian sphere are updated.

[0063] S105, constructing a bounding sphere that encloses the scene rendering model, and adding a plurality of second feature points on the surface of the bounding sphere, and performing parameterized training on the scene rendering model according to the plurality of second feature points to obtain an optimized scene rendering model.

[0064] Furthermore, S105 may also specifically execute:

[0065] S1051, calculating the size of a preset scene bounding box, and constructing a bounding sphere whose size is greater than a specified multiple of the scene bounding box, wherein the bounding sphere encloses the scene rendering model;

[0066] S1052. Evenly add 360 second feature points on the surface of the sphere surrounding the sphere.

[0067] In this embodiment, the size of the bounding sphere is three times larger than the size of the scene bounding box. The parametric training includes deep learning of spherical harmonics, rotation matrices, scales, and opacity, which enhances the realism and integrity of the scene. At the same time, through the learning of parameters such as spherical harmonics and rotation matrices, the model's adaptability to complex lighting and perspective changes is further improved.

[0068] The video is processed by frame extraction to obtain multiple images; each dynamic object image in each image is segmented to obtain a main image; and based on a preset first algorithm, the first feature point of each main image is extracted, and the posture information of each main image is determined; the multiple main images and the corresponding posture information are trained by 3DGS to obtain a scene rendering model; a bounding sphere that wraps the scene rendering model is constructed, and multiple second feature points are added to the surface of the bounding sphere. According to the multiple second feature points, the scene rendering model is parameterized and trained to obtain an optimized scene rendering model, so as to improve the quality and efficiency of scene rendering in the video, thereby improving the automatic driving effect of the unmanned vehicle.

[0069] The above describes the unmanned vehicle scene scanning video quality improvement method in the embodiment of the present invention. The following describes the unmanned vehicle scene scanning video quality improvement system in the embodiment of the present invention. Figure 6 , the unmanned vehicle scene scanning video quality improvement system comprises:

[0070] The frame extraction module 201 is used to extract frames from the video to obtain multiple images;

[0071] A first extraction module 202, used to segment each dynamic object image in each image to obtain a plurality of subject images;

[0072] A second extraction module 203 is used to extract first feature points of each subject image based on a preset first algorithm, and determine position and posture information of each subject image;

[0073] The training module 204 is used to perform 3DGS training on the multiple subject images and the corresponding pose information to obtain a scene rendering model;

[0074] The optimization module 205 is used to construct a bounding sphere that encloses the scene rendering model, and add multiple second feature points on the surface of the bounding sphere. According to the multiple second feature points, the scene rendering model is parameterized and trained to obtain an optimized scene rendering model.

[0075] Specifically, the video is processed by frame extraction to obtain multiple images; each dynamic object image in each image is segmented to obtain a main image; and based on a preset first algorithm, the first feature point of each main image is extracted, and the posture information of each main image is determined; the multiple main images and the corresponding posture information are trained by 3DGS to obtain a scene rendering model; a bounding sphere that wraps the scene rendering model is constructed, and multiple second feature points are added to the surface of the bounding sphere. According to the multiple second feature points, the scene rendering model is parameterized and trained to obtain an optimized scene rendering model, so as to improve the quality and efficiency of scene rendering in the video, thereby improving the automatic driving effect of the unmanned vehicle.

[0076] The frame extraction module 201 may also be specifically used for:

[0077] According to the preset wheel speed meter and preset GPS positioning information in the unmanned vehicle, the video is frame-extracted with each meter traveled by the unmanned vehicle as the cropping interval to obtain multiple images.

[0078] The roads include trunk roads and non-truncal roads, and after the identification module 202, the following is also included:

[0079] Based on a preset road segmentation algorithm, the roads in the image units are segmented and screened to obtain trunk roads and traffic type information on the trunk roads.

[0080] The first extraction module 202 may also be specifically used for:

[0081] Use the preset YOLOv6 network to identify and track each dynamic object in each image, and extract each dynamic object image, where the object image is an image of an object in motion;

[0082] Multiple dynamic object images are classified and summarized in the form of objects of the same type to obtain multiple dynamic object data sets.

[0083] The second extraction module 203 may also be specifically used for:

[0084] Based on a preset SFM algorithm, extract the first feature point of each subject image, and match each first feature point to determine the relationship between each subject image;

[0085] Each first feature point, GPS positioning information, and preset IMU information are used to estimate the pose to determine the pose information of each subject image.

[0086] The optimization module 205 may also be specifically used for:

[0087] Calculate the size of the preset scene bounding box, and construct a bounding sphere whose size is greater than a specified multiple of the scene bounding box, wherein the bounding sphere wraps the scene rendering model;

[0088] 360 second feature points are uniformly added on the surface of the sphere surrounding the sphere.

[0089] The size of the bounding sphere is three times larger than the size of the scene bounding box, and parameterized training includes deep learning of spherical harmonics, rotation matrices, scales, and opacity.

[0090] above Figure 6 The unmanned vehicle scene scanning video quality improvement system in the embodiment of the present invention is described in detail from the perspective of modular functional entities. The unmanned vehicle scene scanning video quality improvement device in the embodiment of the present invention is described in detail from the perspective of hardware processing.

[0091] Figure 7 3 is a schematic diagram of the structure of an unmanned vehicle scene scanning video quality improvement device provided by an embodiment of the present invention. The unmanned vehicle scene scanning video quality improvement device 300 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 310 (for example, one or more processors) and a memory 320, and one or more storage media 330 (for example, one or more mass storage devices) storing application programs 333 or data 332. Among them, the memory 320 and the storage medium 330 can be short-term storage or permanent storage. The program stored in the storage medium 330 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the unmanned vehicle scene scanning video quality improvement device 300. Furthermore, the processor 310 can be configured to communicate with the storage medium 330 to execute a series of instruction operations in the storage medium 330 on the unmanned vehicle scene scanning video quality improvement device 300.

[0092] The unmanned vehicle-based sweeping video quality improvement device 300 may also include one or more power supplies 340, one or more wired or wireless network interfaces 350, one or more input and output interfaces 360, and / or one or more operating systems 331, such as Windows Serve, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will appreciate that Figure 3 The structure of the unmanned vehicle scene scanning video quality improvement device shown does not constitute a limitation on the communication protocol device based on LAN screen projection, and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0093] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the method for improving the quality of the unmanned vehicle scene scanning video.

[0094] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0095] The above is a method or multiple implementation methods for improving the quality of unmanned vehicle scene scanning video provided in combination with specific content, and it is not intended that the specific implementation of the present invention is limited to these descriptions. Any method, structure, etc. similar to or identical to the method, structure, etc. of the present invention, or any technical deduction or replacement based on the concept of the present invention, shall be deemed to be within the scope of protection of the present invention.

Claims

1. A method for improving the quality of scene scanning video of an unmanned vehicle, characterized in that: include: S101, performing frame extraction processing on the video to obtain multiple images; S102, segmenting each dynamic object image in each of the images to obtain a plurality of subject images; S103, extracting first feature points of each subject image based on a preset first algorithm, and determining position and posture information of each subject image; S104, performing 3DGS training on the plurality of subject images and the corresponding pose information to obtain a scene rendering model; S105, constructing a bounding sphere that encloses the scene rendering model, and adding a plurality of second feature points on the surface of the bounding sphere, and performing parameterized training on the scene rendering model according to the plurality of second feature points to obtain an optimized scene rendering model.

2. The method for improving the quality of scene scanning video of an unmanned vehicle according to claim 1, characterized in that: The S101 includes: S1011. According to the preset wheel speed meter and preset GPS positioning information in the unmanned vehicle, every meter traveled by the unmanned vehicle is used as a cropping interval, and frame extraction is performed on the video to obtain multiple images.

3. The method for improving the quality of scene scanning video of an unmanned vehicle according to claim 2, characterized in that: The S102 includes: S1021, using a preset YOLOv6 network to identify and track each dynamic object in each of the images, and extracting each of the dynamic object images, wherein the dynamic object image is an image of an object in motion; S1022: Classify the plurality of dynamic object images, and aggregate them according to the same type of objects to obtain a plurality of dynamic object data sets.

4. The method for improving the quality of scene scanning video of an unmanned vehicle according to claim 3, characterized in that: The S103 includes: S1031, extracting a first feature point of each of the subject images based on a preset SFM algorithm, and matching each of the first feature points to determine a relationship between each of the subject images; S1032. Perform pose estimation on each of the first feature points, the GPS positioning information, and the preset IMU information to determine pose information of each of the subject images.

5. The method for improving the quality of scene scanning video of an unmanned vehicle according to claim 1, characterized in that: The S105 includes: S1051, calculating the size of a preset scene bounding box, and constructing a bounding sphere whose size is greater than a specified multiple of the scene bounding box, wherein the bounding sphere encloses the scene rendering model; S1052. Evenly add 360 second feature points on the surface of the enclosing sphere.

6. The method for improving the quality of scene scanning video of an unmanned vehicle according to claim 5, characterized in that: The size of the bounding sphere is three times greater than the size of the scene bounding box.

7. The method for improving the quality of scene scanning video of an unmanned vehicle according to claim 5, characterized in that: The parameterized training includes deep learning of spherical harmonics, rotation matrices, scale, and opacity.

8. A system for improving the quality of scene scanning video of an unmanned vehicle, characterized in that: include: A frame extraction module, used for performing frame extraction processing on the video to obtain multiple images; A first extraction module, used for segmenting each dynamic object image in each of the images to obtain a plurality of subject images; A second extraction module, configured to extract first feature points of each subject image based on a preset first algorithm, and determine position and posture information of each subject image; A training module, used for performing 3DGS training on the plurality of subject images and the corresponding pose information to obtain a scene rendering model; The optimization module is used to construct a bounding sphere that encloses the scene rendering model, and add multiple second feature points on the surface of the bounding sphere. According to the multiple second feature points, the scene rendering model is parameterized and trained to obtain an optimized scene rendering model.

9. An unmanned vehicle scene scanning video quality improvement device, characterized in that: The unmanned vehicle scene scanning video quality improvement device comprises: a memory and at least one processor, the memory stores instructions, and the memory and the at least one processor are interconnected via a line; The at least one processor calls the instructions in the memory so that the unmanned vehicle scene scanning video quality improvement device executes the unmanned vehicle scene scanning video quality improvement method according to any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for improving the quality of the scene scanning video of an unmanned vehicle as described in any one of claims 1 to 7 is implemented.

Images