Scene model construction method and device, storage medium and electronic device

Abstract

The invention provides a scene model construction method and device, a storage medium and an electronic device.The method comprises the steps that multi-modal data around a vehicle are acquired, and the multi-modal data comprise image data, point cloud data and environment data; extracting a feature point set and texture information of the image data; a two-dimensional Gaussian splash model of the feature point set is constructed according to the environmental data, and the Gaussian splash model is used for simulating spatial distribution and influence range of feature points; and performing scene reconstruction on the Gaussian splash model by using the point cloud data and the texture information to obtain a three-dimensional scene model of the environment where the vehicle is located. According to the embodiment of the invention, the technical problem that the three-dimensional scene model of the environment where the vehicle is located is not accurate in the related technology is solved, the real-time performance and accuracy of reconstruction of the three-dimensional scene model are improved, and the reliability and safety of an automatic driving system are enhanced.

Description

Technical Field

[0001] The present invention relates to the field of automobile technology, and in particular to a method and device for constructing a scene model, a storage medium, and an electronic device. Background Art

[0002] Among related technologies, the rapid development of the new energy vehicle industry has promoted the rapid update and iteration of autonomous driving and advanced driver assistance system technologies, and accurate scene reconstruction has become one of the key technologies for achieving safe and efficient driving. Environmental perception technology plays an important role in assisted driving systems. Environmental perception technology includes but is not limited to sensors such as cameras, radars, and lidars, which collect data about the surrounding environment to build a model of the driving scene.

[0003] Related technologies In assisted driving systems, accurate understanding and reconstruction of the real-time driving environment is crucial to ensure driving safety. Assisted driving systems require accurate environmental perception and scene reconstruction capabilities for effective obstacle detection, lane keeping, automatic parking and navigation. However, the accuracy and reliability of these systems are limited by image noise, edge blur, insufficient real-time processing performance, and failure to consider the vehicle's environmental conditions.

[0004] With respect to the above-mentioned problems existing in the related technologies, no efficient and accurate solutions have been found yet. Summary of the invention

[0005] The present invention provides a method and device for constructing a scene model, a storage medium, and an electronic device to solve technical problems in related technologies.

[0006] According to one embodiment of the present invention, a method for constructing a scene model is provided, comprising: acquiring multimodal data around a vehicle, wherein the multimodal data includes image data, point cloud data, and environmental data; extracting a feature point set and texture information of the image data; constructing a two-dimensional Gaussian splash model of the feature point set based on the environmental data, wherein the Gaussian splash model is used to simulate the spatial distribution and influence range of the feature points; and reconstructing the scene of the Gaussian splash model using the point cloud data and the texture information to obtain a three-dimensional scene model of the environment in which the vehicle is located.

[0007] Optionally, extracting the feature point set and texture information of the image data includes: performing edge detection on the image data to obtain edge features of the image data; performing corner point detection on the image data to obtain corner point features of the image data; extracting the grayscale co-occurrence matrix of the image data to obtain texture information of the image data; wherein the feature point set includes the edge features and the corner point features.

[0008] Optionally, before extracting the set of feature points of the image data, the method also includes: performing image segmentation on the image data and selecting a number of regions of interest; parsing regional attribute information of each region of interest; setting filtering parameters of the Gaussian filter in the corresponding image region based on the regional attribute information; and using the Gaussian filter to denoise each region of interest to obtain optimized image data.

[0009] Optionally, setting filtering parameters of the Gaussian filter in the corresponding image area based on the regional attribute information includes: for each image area, extracting local contrast and noise level in the regional attribute information; setting filtering strength of the Gaussian filter in the corresponding image area based on the local contrast and the noise level, wherein the filtering parameters include the filtering strength, the filtering strength is negatively correlated with the local contrast, and the filtering strength is positively correlated with the noise level.

[0010] Optionally, a two-dimensional Gaussian splash model of the feature point set is constructed according to the environmental data, including: classifying the feature point set according to the feature object type to obtain multiple feature point subsets, wherein each feature point subset corresponds to a scene level; parsing the distribution density and environmental complexity of each feature point in the feature point subset; using the distribution density and the environmental complexity to configure the standard deviation of the Gaussian distribution function, wherein the standard deviation is used to control the width and influence range of the distribution; configuring the Gaussian distribution function using the standard deviation; generating multiple spatially distributed splash particles according to the Gaussian distribution function to obtain a particle distribution model; configuring a weight coefficient of each splash particle in the particle distribution model to obtain a two-dimensional Gaussian splash model corresponding to the scene level.

[0011] Optionally, using the standard deviation to configure the Gaussian distribution function includes: using the standard deviation to configure the following Gaussian distribution function G(x, y): Wherein, σ is the standard deviation, exp() is the natural exponential function, and x and y are the two-dimensional coordinate positions of the feature points.

[0012] Optionally, generating a plurality of spatially distributed splash particles according to the Gaussian distribution function includes: generating a plurality of spatially distributed splash particles P(x, y) using the following formula: (x i ,y i ) is the two-dimensional coordinate position of feature point i, n is the total number of feature points, (x, y) is the three-dimensional horizontal and vertical coordinate position of feature point i, and G() is the Gaussian distribution function.

[0013] Optionally, a weight coefficient of each splatter particle is configured in the particle distribution model to obtain a two-dimensional Gaussian splatter model corresponding to the scene level, including: for each splatter particle in the particle distribution model, analyzing the characteristic intensity of the corresponding feature point; configuring the weight coefficient of the splatter particle in the particle distribution model based on the characteristic intensity to obtain a two-dimensional Gaussian splatter model corresponding to the scene level.

[0014] Optionally, classifying the feature point set according to feature object type includes: for each feature point in the feature point set, determining the feature object type to which the feature point belongs, wherein the feature object types include roads, lane lines, buildings, traffic signs, obstacles, other vehicles, and living things; if the feature point is a road or a lane line, classifying the feature point into a first scene level; if the feature point is a building, a traffic sign, or an obstacle, classifying the feature point into a second scene level; if the feature point is another vehicle or a living thing, classifying the feature point into a third scene level.

[0015] Optionally, the point cloud data and the texture information are used to reconstruct the Gaussian splash model to obtain a three-dimensional scene model of the environment where the vehicle is located, including: mapping the particle distribution in the Gaussian splash model to the three-dimensional environment to generate a basic three-dimensional model; using the point cloud data to complete the details of the basic three-dimensional model to obtain an intermediate three-dimensional model; applying the texture information to the surface of the intermediate three-dimensional model to obtain a three-dimensional scene model of the environment where the vehicle is located.

[0016] Optionally, mapping the particle distribution in the Gaussian splash model to a three-dimensional environment to generate a basic three-dimensional model includes: for each particle point in the Gaussian splash model, calculating the depth coordinate Z of the particle point using the following formula: Wherein, D is the distance from the camera to the object corresponding to the particle point, f is the focal length of the camera, and d is the pixel distance of the particle point on the two-dimensional image; the corresponding particle point is converted into a three-dimensional grid using the depth coordinate to generate a basic three-dimensional model.

[0017] According to another embodiment of the present invention, a scene model construction device is provided, including: an acquisition module, used to acquire multimodal data around a vehicle, wherein the multimodal data includes image data, point cloud data, and environmental data; an extraction module, used to extract a feature point set and texture information of the image data; a construction module, used to construct a two-dimensional Gaussian splash model of the feature point set based on the environmental data, wherein the Gaussian splash model is used to simulate the spatial distribution and influence range of the feature points; a reconstruction module, used to reconstruct the Gaussian splash model using the point cloud data and the texture information to obtain a three-dimensional scene model of the environment where the vehicle is located.

[0018] Optionally, the extraction module includes: a first detection unit, used to perform edge detection on the image data to obtain edge features of the image data; a second detection unit, used to perform corner point detection on the image data to obtain corner point features of the image data; an extraction unit, used to extract the grayscale co-occurrence matrix of the image data to obtain texture information of the image data; wherein the feature point set includes the edge features and the corner point features.

[0019] Optionally, the device also includes: a segmentation module, used to perform image segmentation on the image data and select a number of regions of interest before the extraction module extracts the feature point set of the image data; a parsing module, used to parse the region attribute information of each region of interest; a setting module, used to set the filtering parameters of the Gaussian filter in the corresponding image region based on the region attribute information; and a denoising module, used to use the Gaussian filter to denoise each region of interest to obtain optimized image data.

[0020] Optionally, the setting module includes: an extraction unit, used to extract the local contrast and noise level in the regional attribute information for each image area; a setting unit, used to set the filtering strength of the Gaussian filter in the corresponding image area based on the local contrast and the noise level, wherein the filtering parameters include the filtering strength, the filtering strength is negatively correlated with the local contrast, and the filtering strength is positively correlated with the noise level.

[0021] Optionally, the construction module includes: a classification unit, used to classify the feature point set according to the feature object type to obtain multiple feature point subsets, wherein each feature point subset corresponds to a scene level; a parsing unit, used to parse the distribution density and environmental complexity of each feature point in the feature point subset; a configuration unit, used to configure the standard deviation of the Gaussian distribution function using the distribution density and the environmental complexity, wherein the standard deviation is used to control the width and influence range of the distribution;

[0022] The first configuration unit is used to configure the Gaussian distribution function using the standard deviation; the generation unit is used to generate a plurality of spatially distributed splash particles according to the Gaussian distribution function to obtain a particle distribution model; the second configuration unit is used to configure the weight coefficient of each splash particle in the particle distribution model to obtain a two-dimensional Gaussian splash model corresponding to the scene level.

[0023] Optionally, the first configuration unit includes: a configuration subunit, configured to configure the following Gaussian distribution function G(x, y) using the standard deviation: Wherein, σ is the standard deviation, exp() is the natural exponential function, and x and y are the two-dimensional coordinate positions of the feature points.

[0024] Optionally, the generating unit includes: a generating subunit, configured to generate a plurality of spatially distributed splash particles P(x, y) using the following formula: (x i ,y i ) is the two-dimensional coordinate position of feature point i, n is the total number of feature points, (x, y) is the three-dimensional horizontal and vertical coordinate position of feature point i, and G() is the Gaussian distribution function.

[0025] Optionally, the second configuration unit includes: an analysis subunit, used to analyze the characteristic intensity of the corresponding feature point for each splatter particle in the particle distribution model; and a configuration subunit, used to configure the weight coefficient of the splatter particle in the particle distribution model based on the characteristic intensity to obtain a two-dimensional Gaussian splatter model corresponding to the scene hierarchy.

[0026] Optionally, the classification unit includes: a determination subunit, used to determine, for each feature point in the feature point set, the feature object type to which the feature point belongs, wherein the feature object types include roads, lane lines, buildings, traffic signs, obstacles, other vehicles, and living things; a classification subunit, used to classify the feature point into a first scene level if the feature point is a road or a lane line; classify the feature point into a second scene level if the feature point is a building, a traffic sign, or an obstacle; and classify the feature point into a third scene level if the feature point is another vehicle or a living thing.

[0027] Optionally, the reconstruction module includes: a mapping unit, used to map the particle distribution in the Gaussian splash model into a three-dimensional environment to generate a basic three-dimensional model; a completion unit, used to use the point cloud data to complete the details of the basic three-dimensional model to obtain an intermediate three-dimensional model; and an application unit, used to apply the texture information to the surface of the intermediate three-dimensional model to obtain a three-dimensional scene model of the environment where the vehicle is located.

[0028] Optionally, the mapping unit includes: a calculation subunit, for calculating the depth coordinate Z of each particle point in the Gaussian splash model using the following formula: Wherein, D is the distance from the camera to the object corresponding to the particle point, f is the focal length of the camera, and d is the pixel distance of the particle point on the two-dimensional image; the conversion subunit is used to convert the corresponding particle point into a three-dimensional grid using the depth coordinate to generate a basic three-dimensional model.

[0029] According to another aspect of an embodiment of the present application, a storage medium is further provided, which includes a stored program, and the above steps are executed when the program is run.

[0030] According to another aspect of an embodiment of the present application, there is also provided an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; wherein: the memory is used to store computer programs; and the processor is used to execute the steps in the above method by running the program stored in the memory.

[0031] The embodiment of the present application also provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute the steps in the above method.

[0032] Beneficial effects of the present invention:

[0033] 1. Adopt adaptive Gaussian filter to dynamically adjust the filter strength according to the different characteristics of the image area. It retains more details in the areas with rich details, and performs stronger noise suppression in the areas with large noise, so as to balance the image detail retention and noise suppression;

[0034] 2. Support filter intensity adjustment, automatically adjust the filter parameters based on the local contrast and noise level of the image to adapt to different environmental conditions and image characteristics;

[0035] 3. Optimize the parameters of the traditional splash model to improve image reconstruction capabilities in complex driving scenarios. The improved model can more accurately handle lane markings, obstacles and other key elements, improving the accuracy of scene reconstruction;

[0036] 4. Dynamic scene modeling, combined with the dynamic Gaussian distribution model, improves the processing ability of dynamic objects, so that the algorithm can better cope with changes in the environment during vehicle movement;

[0037] 5. Introduce edge detection methods to enhance edge information in images. By increasing edge contrast, the scene reconstruction effect is improved, the edge information is clearer, and the recognition ability of key driving elements is improved;

[0038] 6. Combine data from cameras, radars, and lidars to generate accurate scene models. Fusion of data from different sensors can make up for the shortcomings of a single sensor and improve the accuracy and robustness of overall environmental perception.

[0039] 7. By performing consistency correction on sensor data, the data inconsistency problem between sensors is solved to ensure that the generated scene model is more realistic and reliable;

[0040] 8. Dynamically adjust the algorithm parameters according to the vehicle's real-time position and speed and other environmental complexity parameters to adapt to different driving environments. This adjustment can optimize the algorithm's performance under different driving conditions and improve the system's adaptability and stability;

[0041] 9. By monitoring environmental changes in real time, the algorithm settings are automatically adjusted to ensure high reconstruction accuracy under various driving conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0043] Figure 1 is a hardware structure block diagram of a vehicle according to an embodiment of the present invention;

[0044] Figure 2 is a flow chart of a method for constructing a scene model according to an embodiment of the present invention;

[0045] Figure 3 is a schematic diagram of data preprocessing in an embodiment of the present invention;

[0046] Figure 4 It is an overall schematic diagram of the system of the embodiment of the present invention;

[0047] Figure 5 It is a structural block diagram of a scene model construction device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only embodiments of a part of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present application. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0049] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0050] Example 1

[0051] The method embodiment provided in the first embodiment of the present application can be executed in a vehicle, a server, a processor or a similar processing device. Taking running on a vehicle as an example, Figure 1 FIG. 1 is a hardware structure diagram of a vehicle according to an embodiment of the present invention. Figure 1 As shown, the vehicle may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data. Optionally, the above vehicle may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is for illustration only and does not limit the structure of the above vehicle. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0052] The memory 104 can be used to store vehicle programs, for example, software programs and modules of application software, such as a vehicle program corresponding to a method for constructing a scene model of a vehicle in an embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the vehicle program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the vehicle via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0053] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the vehicle. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0054] In this embodiment, a method for constructing a scene model is provided. Figure 2is a flow chart of a method for constructing a scene model according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0055] Step S202, acquiring multimodal data around the vehicle, wherein the multimodal data includes image data, point cloud data, and environmental data;

[0056] The image data in this embodiment is data collected by the vehicle's image sensor, and the environmental data is data collected by the vehicle's radar sensor, position sensor, etc., including but not limited to laser radar (L IDAR), camera, radar, vehicle-mounted sensors, etc.

[0057] Step S204, extracting a feature point set and texture information of the image data;

[0058] Step S206, constructing a two-dimensional Gaussian splash model of the feature point set according to the environmental data, wherein the Gaussian splash model is used to simulate the spatial distribution and influence range of the feature points;

[0059] Step S208: Reconstruct the Gaussian splash model using the point cloud data and the texture information to obtain a three-dimensional scene model of the environment where the vehicle is located.

[0060] The three-dimensional scene model of this embodiment can be applied to assisted driving systems, such as high-speed navigation, lane keeping systems, automatic parking systems, obstacle detection systems, etc.

[0061] Through the above steps, multimodal data around the vehicle is obtained, wherein the multimodal data includes image data, point cloud data, and environmental data; a feature point set and texture information of the image data are extracted; a two-dimensional Gaussian splash model of the feature point set is constructed according to the environmental data, wherein the Gaussian splash model is used to simulate the spatial distribution and influence range of the feature points; the point cloud data and the texture information are used to perform scene reconstruction on the Gaussian splash model to obtain a three-dimensional scene model of the environment where the vehicle is located, and the Gaussian splash model is used to optimize the flexibility and accuracy of the three-dimensional scene model, thereby solving the technical problem of inaccurate construction of a three-dimensional scene model of the environment where the vehicle is located in the related technology, and improving the real-time and accuracy of the three-dimensional scene model reconstruction, thereby enhancing the reliability and safety of the autonomous driving system.

[0062] In this embodiment, multimodal data around the vehicle is obtained, wherein the multimodal data includes image data, point cloud data, and environmental data including: static, dynamic, speed and other related data around the vehicle are collected through various sensors on the vehicle body including but not limited to front-view cameras, side-view cameras, rear-view cameras, lidars, millimeter-wave radars, inertial measurement units, etc.

[0063] In terms of sensor layout, high-definition cameras are configured in the front, rear, left and right directions, mainly front-view, side-view and rear-view cameras, which are used to capture image data around the vehicle in real time. The cameras should have high dynamic range (HDR) function to collect images under different lighting conditions; LiDAR is configured on the top or front of the vehicle to obtain high-precision three-dimensional point cloud data; it can provide data information about the distance and shape of objects to help build accurate scene models; millimeter-wave radar is installed at the front and rear of the vehicle to detect the speed and distance of objects in front and behind, and can also support the collection of relevant data under low light or bad weather conditions;

[0064] In this embodiment, by setting a suitable data acquisition frequency, the acquisition frequency of image and point cloud data should be high enough to ensure the accuracy of data in dynamic scenes; the sensor is calibrated regularly, including the internal and external parameter calibration of the camera, the calibration of the laser radar and the debugging of the radar, to ensure the accuracy of the collected data; the collected data is stored in the processing unit of the vehicle or the cloud server, and the data storage system should have high speed and large capacity to cope with the storage requirements of large amounts of data; an efficient data stream processing system (such as Apache Kafka) is used to ensure that the data obtained from the sensor can be transmitted to the processing unit in a timely manner, and the data includes real-time sensor data and feedback of processing results; the data from different sensors are aligned using timestamps, and synchronization algorithms are applied to ensure the consistency of different sensor data in time; the synchronized data is transmitted to the central processing unit or the cloud server through high-speed communication interfaces such as CAN bus and Ethernet for storage and management, while ensuring the real-time and stability of data transmission; the data management includes preliminary processing of the collected data, including noise removal and data normalization, and storage and classification of the data for subsequent analysis and processing.

[0065] When capturing data around the vehicle in real time, it is necessary to set the camera's acquisition frequency to meet the needs of dynamic scenes; adjust the measurement parameters of the lidar and millimeter-wave radar to adapt to different environmental conditions; wherein the central processing unit or cloud server includes a data analysis module to annotate the image and point cloud data, identify important feature points, object boundaries and obstacles (including but not limited to lane lines, pedestrians, traffic lights, overpasses, obstacles, etc.), and generate an environmental map. The data analysis module can fuse data from different sensors to generate comprehensive environmental perception information. For example, combining image data and lidar point cloud data can provide a more comprehensive scene description, and the data can be converted into a unified format for subsequent processing and analysis.

[0066] In this embodiment, after the image data and the environmental data are acquired, the image data and the environmental data may be preprocessed, and the preprocessing includes image denoising, image enhancement, image standardization, image correction, feature extraction, data fusion, etc. Figure 3 It is a schematic diagram of data preprocessing in an embodiment of the present invention. Data from sensors such as cameras and radars are input. After the data preprocessing is completed, scene modeling and analysis are performed using the preprocessed data, and finally an environmental model (a three-dimensional scene model of the vehicle's environment) is output.

[0067] In an implementation of image denoising of the present embodiment, before extracting a set of feature points of the image data, it also includes: performing image segmentation on the image data and selecting a number of regions of interest; parsing the region attribute information of each region of interest; setting filtering parameters of the Gaussian filter in the corresponding image region based on the region attribute information; and using the Gaussian filter to denoise each region of interest to obtain optimized image data.

[0068] In this embodiment, noise is removed from the collected image data, and noise processing is performed using a spatial domain filtering method or a frequency domain filtering method. In one noise removal example, median filtering can be used to remove salt and pepper noise in an image, and the image is smoothed while retaining edge information by replacing the current pixel value with the median of the neighboring pixels. In another noise removal example, an adaptive Gaussian filter is used, and a sliding window technique can be used to apply the filter to a local area to calculate the filtering result; the local filtering results are synthesized into the final image to ensure that the details and edges of the image are retained and optimized; in another noise removal example, bilateral filtering is used to consider both spatial distance and pixel value differences, and to remove noise while retaining edges. Suitable for removing color noise and retaining edges;

[0069] In the example of filtering denoising, an adaptive Gaussian filter is designed according to the calculated parameters, and the adaptive parameters of the filter are calculated based on the local contrast and noise level of the image area. Higher contrast areas retain fewer details, while lower contrast areas require stronger filtering; the standard deviation σ of the filter is set as a function, σ=f(contrast, noise), where f is a function adjusted according to contrast and noise levels.

[0070] In one example, setting the filtering parameters of the Gaussian filter in the corresponding image area based on the regional attribute information includes: for each image area, extracting the local contrast and noise level in the regional attribute information; setting the filtering strength of the Gaussian filter in the corresponding image area based on the local contrast and the noise level, wherein the filtering parameters include the filtering strength, the filtering strength is negatively correlated with the local contrast, and the filtering strength is positively correlated with the noise level.

[0071] In this embodiment, the collected image data can also be enhanced and standardized by a method. In the example of image enhancement, for example, the contrast of the image can be adjusted by histogram equalization to make the brightness distribution of the image more uniform and enhance the visual effect of the image; for example, the contrast of the image can be adjusted by linear transformation or gamma correction to enhance the details and edge information of the image and make the image features more prominent.

[0072] Examples of image normalization include cropping, scaling, or padding images to a uniform size to ensure input consistency for subsequent processing and model training; converting images from RGB color space to grayscale images or other color spaces (such as HSV, Lab), as grayscale images can reduce computational complexity; and normalizing images by dividing by the maximum value or standard deviation of pixel values, scaling pixel values ​​to a uniform range (such as [0,1]) to improve algorithm stability and convergence speed.

[0073] In this embodiment, the collected image data can also be corrected. For example, the camera internal parameters and distortion model are used for correction. The camera calibration model is used for distortion correction to correct radial distortion and tangential distortion in the image; for example, the image problems caused by uneven illumination are processed by an illumination compensation algorithm (such as brightness equalization and reflection correction) to ensure the consistency of the image under different illumination conditions.

[0074] In this embodiment, the collected image data can also be segmented; for example, foreground and background segmentation can be performed using Otsu threshold segmentation, K-means clustering or deep learning segmentation network; and the region of interest (ROI) in the image can be extracted for subsequent processing.

[0075] In one example, extracting the feature point set and texture information of the image data includes: performing edge detection on the image data to obtain edge features of the image data; performing corner point detection on the image data to obtain corner point features of the image data; extracting the grayscale co-occurrence matrix of the image data to obtain texture information of the image data; wherein the feature point set includes the edge features and the corner point features.

[0076] In this embodiment, feature extraction may be performed on the collected image data, including edge features, corner point features, key point features, texture features, shape features, feature matching, and feature fusion.

[0077] By introducing edge detection methods (such as Canny edge detection or Sobel operator), the edge information in the image can be enhanced, the edges can be made clearer, the scene reconstruction effect can be improved, and the recognition ability of key driving elements can be improved. Use the Canny edge detection algorithm to apply Gaussian filtering to smooth the image, calculate the gradient amplitude and direction, use non-maximum suppression to refine the edge, and form the final edge map through double threshold detection and edge connection; at the same time, use the Sobel filter to calculate the horizontal and vertical gradients of the image, and then combine the gradient amplitude and direction for edge detection to extract the edges and contours in the image;

[0078] For corner features, the Harris corner detection algorithm can be used to calculate the corner response function of the image and determine the corner position by setting a threshold; or the Shi-Tomas i corner detection algorithm can detect the corners in the image through the corner response function and select the point with the largest response function value as the corner point;

[0079] In addition, you can use adaptive filtering technology and Laplace filters to enhance the edges of images and make image details clearer. You can also calculate the image gradient amplitude and apply the gradient amplitude to the image to enhance the edge effect; apply the edge detection results to the image to adjust the edge contrast and improve the visibility of key details;

[0080] For key point features, SI FT (Scale Invariant Feature Transform) is used to detect key points in the image, and scale and direction invariant feature descriptors are calculated for each key point; SURF (Speeded Up Robust Features) algorithm is applied to detect key points in the image, and its feature descriptors are calculated, and ORB (Oriented FAST and Rotated BRI EF) algorithm is used to identify key points;

[0081] For texture features, methods such as gray-level co-occurrence matrix are used to extract texture features in images, such as contrast, entropy, correlation, etc., to help identify and classify different types of surfaces or objects; or local binary pattern (LBP) is used: the image is divided into small blocks, and the local binary pattern histogram of each small block is calculated to describe the local texture information of the image;

[0082] For shape features, extract geometric shapes such as straight lines and circles in the image using Hough transform, and use contour detection algorithms (such as the findContours function in OpenCV) to extract the boundaries of objects in the image;

[0083] When matching features, feature descriptors (such as SI FT (Scale Invariant Feature Transform) or SURF (Speeded Up Robust Feature) descriptors) are used to match features and find corresponding feature points in different images; the RANSAC algorithm is used to verify the robustness of feature matching, remove incorrect matching points and estimate the transformation model (such as homography matrix);

[0084] The extracted features can also be extracted. Feature extraction also includes: dimensionality reduction processing of the extracted features to reduce the complexity of the computing environment and improve processing efficiency; applying principal component analysis) or linear discriminant analysis to perform feature dimensionality reduction; post-processing the extracted features to improve the stability and reliability of the features, and using feature screening and weighting algorithms to optimize the features; wherein the feature extraction step is combined with the subsequent scene reconstruction step to improve the accuracy and efficiency of reconstruction;

[0085] When fusion of features, the two-dimensional features of the image are fused with the three-dimensional depth data obtained by the depth sensor, and the depth image is used to enhance the positioning accuracy of key points; the feature information in the multi-view images can be combined to generate a consistent feature representation; feature matching algorithms (such as brute force matching, K nearest neighbor matching) are used to perform global matching of features; multi-view images are fused to generate panoramic images or multi-view images, and combined with depth images to generate a comprehensive environment representation;

[0086] Finally, image quality evaluation indicators (such as peak signal-to-noise ratio (PSNR) and structural similarity (SSIM)) are used to detect the quality of the processed images; image clarity, contrast and detail retention are evaluated.

[0087] In this embodiment, a two-dimensional Gaussian splash model of the feature point set is constructed according to the environmental data, including: classifying the feature point set according to the feature object type to obtain multiple feature point subsets, wherein each feature point subset corresponds to a scene level; parsing the distribution density and environmental complexity of each feature point in the feature point subset; configuring the standard deviation of the Gaussian distribution function using the distribution density and the environmental complexity, wherein the standard deviation is used to control the width and influence range of the distribution; configuring the Gaussian distribution function using the standard deviation; generating multiple spatially distributed splash particles according to the Gaussian distribution function to obtain a particle distribution model; configuring the weight coefficient of each splash particle in the particle distribution model to obtain a two-dimensional Gaussian splash model corresponding to the scene level.

[0088] Optionally, the above-mentioned environmental complexity is related to environmental complexity, such as the lighting and weather of the vehicle's environment. The parameters of the Gaussian distribution are adjusted according to different environmental conditions such as lighting and weather to improve the robustness of the model.

[0089] In one example, using the standard deviation to configure the Gaussian distribution function includes: using the standard deviation to configure the following Gaussian distribution function G(x, y): Wherein, σ is the standard deviation, exp() is the natural exponential function, and x and y are the two-dimensional coordinate positions of the feature points.

[0090] In this embodiment, the Gaussian kernel function is selected in the form of a two-dimensional Gaussian function formula G(x, y), where σ is the standard deviation of the Gaussian distribution, which controls the width and influence range of the distribution). The feature points are modeled using a Gaussian function, and the Gaussian distribution function is used to describe the spatial distribution and influence range of the feature points. According to the density of the feature points and the complexity of the environment, the standard deviation σ value of the kernel function is dynamically adjusted to optimize the flexibility and accuracy of the model. The parameters of the Gaussian distribution are adjusted according to different environmental conditions such as light and weather to improve the robustness of the model.

[0091] Dynamically adjust the standard deviation σ. By analyzing the distribution density of feature points and environmental data, adjust the σ value in real time to optimize model performance. Use an adaptive algorithm to dynamically calculate the optimal σ value to adapt to environmental changes.

[0092] In one example, generating a plurality of spatially distributed splash particles according to the Gaussian distribution function includes: generating a plurality of spatially distributed splash particles P(x, y) using the following formula: (x i ,y i ) is the two-dimensional coordinate position of feature point i, n is the total number of feature points, (x, y) is the three-dimensional horizontal and vertical coordinate position of feature point i, and G() is the Gaussian distribution function.

[0093] Generate splash particles according to the Gaussian distribution function. These particles are scattered in space according to the Gaussian distribution.

[0094] In one example, a weight coefficient of each splatter particle is configured in the particle distribution model to obtain a two-dimensional Gaussian splatter model corresponding to the scene level, including: for each splatter particle in the particle distribution model, analyzing the characteristic intensity of the corresponding feature point; configuring the weight coefficient of the splatter particle in the particle distribution model based on the characteristic intensity to obtain a two-dimensional Gaussian splatter model corresponding to the scene level.

[0095] Assign a weight to each particle. The weight formula is: a iis the weight of the i-th feature point, and the weight is related to the strength of the feature point; the particle weight is calculated according to the strength or importance of the feature point; the particle weight is integrated using the weighted average method to reflect the overall impact of the feature point on the model; finally, all the generated splash particles are integrated into a complete scene model, and the impact between particles can be completed by weighted average or accumulation. The formula of the two-dimensional Gaussian splash model is:

[0096]

[0097] In one example, classifying the feature point set according to feature object type includes: for each feature point in the feature point set, determining the feature object type to which the feature point belongs, wherein the feature object types include roads, lane lines, buildings, traffic signs, obstacles, other vehicles, and living things; if the feature point is a road or a lane line, classifying the feature point into a first scene level; if the feature point is a building, a traffic sign, or an obstacle, classifying the feature point into a second scene level; if the feature point is another vehicle or a living thing, classifying the feature point into a third scene level.

[0098] In this example, a multi-layer two-dimensional scene model (Gaussian splash model) is constructed in layers to describe different scene elements (such as roads, obstacles, vehicles, etc.). Each layer uses a different Gaussian distribution function and particle generation mechanism; hierarchical structure diagram, basic layer: describes roads and lane lines; middle layer: describes static obstacles (such as buildings, traffic signs, obstacles); advanced layer: describes dynamic objects (such as other vehicles, pedestrians, animals, etc.).

[0099] When constructing the Gaussian splash model, use real-time sensor data to dynamically adjust the model; update model parameters such as σ value, particle position and weight according to the newly acquired image data and sensor information; adjust model parameters according to actual environmental changes; dynamically update, update the model in real time to reflect new feature points and environmental information; when adaptively adjusting, adjust the model according to changes in scene lighting, object movement, etc. Adaptive adjustment is implemented to maintain the accuracy and stability of the model. The accuracy of the model is optimized through error analysis and correction mechanisms, and the actual scene and model reconstruction results are compared to adjust the Gaussian distribution function and particle generation mechanism to reduce errors; improve the efficiency of model construction, including accelerating particle generation and model fusion processes, and using efficient data structures and parallel computing to increase processing speed.

[0100] In one implementation of the present embodiment, the point cloud data and the texture information are used to reconstruct the Gaussian splash model to obtain a three-dimensional scene model of the environment where the vehicle is located, including: mapping the particle distribution in the Gaussian splash model to the three-dimensional environment to generate a basic three-dimensional model; using the point cloud data to complete the details of the basic three-dimensional model to obtain an intermediate three-dimensional model; applying the texture information to the surface of the intermediate three-dimensional model to obtain a three-dimensional scene model of the environment where the vehicle is located.

[0101] The preprocessed image data and depth data are input into the Gaussian splash model, and the particle distribution in the Gaussian splash model is mapped to the actual environment to generate the basic structure of the scene; the distribution of particles simulates the influence of feature points and objects in the environment, and generates three-dimensional representations of various objects and scenes in the environment according to the particle density and weight in the model.

[0102] Optionally, mapping the particle distribution in the Gaussian splash model to a three-dimensional environment to generate a basic three-dimensional model includes: for each particle point in the Gaussian splash model, calculating the depth coordinate Z of the particle point using the following formula: Wherein, D is the distance from the camera to the object corresponding to the particle point, f is the focal length of the camera, and d is the pixel distance of the particle point on the two-dimensional image; the corresponding particle point is converted into a three-dimensional grid using the depth coordinate to generate a basic three-dimensional model.

[0103] Use the generated particle data to create a 3D model. The particles can be converted into a 3D mesh through a voxelization algorithm or a meshing technique. Use the above formula to calculate the Z value of the particle point and obtain the 3D coordinates of each point to generate the 3D reconstruction data of the scene.

[0104] When reconstructing a scene, it also includes detail enhancement. Scene enhancement uses cloud interpolation, texture mapping, and intelligent repair methods to complete the details of the basic scene model, and uses deep learning models or interpolation algorithms to fill in the gaps in details caused by sparse or missing data. When reconstructing a scene, it also includes texture mapping, which applies the texture information extracted from the image data to the 3D model; the color and detail information of the image is mapped to the surface of the 3D model through texture coordinates. The generated 3D reconstruction data is fused with real-time environmental data to update and optimize the scene model.

[0105] The reconstructed data is fused with real-time data from sensors such as radar and lidar, and the scene model is optimized using algorithms such as Kalman filtering or weighted average data fusion. Finally, the generated 3D model is smoothed to remove noise and irregularities. Detail enhancement algorithms are applied to improve the accuracy and clarity of the scene model.

[0106] In this embodiment, after the reconstruction of the three-dimensional scene model is completed, the reconstructed scene data is output for decision-making and navigation of the assisted driving system; the reconstructed scene data is converted into the format required by the assisted driving system; a scene map, obstacle detection information or navigation path planning data is generated. The scene model is dynamically updated based on real-time sensor data; incremental learning or adaptive algorithms are used to optimize the accuracy of scene reconstruction; and the reconstruction results can also be verified and evaluated to ensure their accuracy and consistency to ensure that they meet actual environmental conditions.

[0107] In this embodiment, after the reconstruction of the three-dimensional scene model is completed, it is dynamically adjusted, the dynamic model is updated, and the incremental update is performed. The new data is integrated into the existing scene model through the incremental update algorithm. The incremental update includes the addition, deletion, and modification of feature points, the position adjustment of particles, and the local correction of the model. A real-time feedback mechanism can also be established, such as real-time correction of sensor data, adaptive adjustment of model parameters, and real-time monitoring and adjustment of the model. Based on the current sensor data and environmental changes, the Gaussian distribution parameters and particle weights are dynamically adjusted.

[0108] In an example of a dynamic parameter adjustment 3D scene model, it includes: model accuracy optimization to adjust the standard deviation of the Gaussian kernel function and other model parameters according to real-time environmental changes such as lighting and vehicle speed; analyze the monitored environmental data to identify the current environmental conditions and changes, such as detecting changes in lighting or road conditions; adjust the standard deviation of the Gaussian kernel function and other model parameters according to real-time environmental changes; apply real-time learning algorithms to dynamically optimize model parameters; transmit the optimized scene data to the assisted driving system, and update the decision model of the navigation system or the driver assistance system through the real-time data interface; collect feedback after implementation, evaluate the actual effect of the adjustment strategy, and further optimize the adjustment strategy and system parameters based on the feedback to achieve better performance;

[0109] Through model accuracy optimization, local optimization is performed on key areas of the model. For example, high-resolution modeling is performed in areas near the vehicle, while low-resolution modeling is performed in areas far from the vehicle. Through computing performance optimization, parallel computing technology is used to accelerate the model building and updating process. Multi-threaded or multi-core processors are used to share computing tasks and improve real-time processing speed; GPUs are used to accelerate image processing and 3D reconstruction to improve computing efficiency; efficient algorithms are used to optimize the model processing process, such as using fast particle generation algorithms and efficient feature matching algorithms (such as FLANN, Fast Library for Approximate Nearest Ne i ghbors) to improve performance; data compression and fast structure search algorithm optimization technology are applied to reduce computing and storage requirements.

[0110] Figure 4It is an overall schematic diagram of the system of an embodiment of the present invention. The system includes multiple functional modules, a data acquisition and sensor module, which acquires the assisted driving scene data to be constructed; a data preprocessing module, which is used to preprocess the collected data; a feature extraction and fusion module, which is used to extract the preprocessed data features and model and output the scene construction information; a scene reconstruction and analysis module, which is used to receive the scene construction information output by the algorithm module and construct the output scene; a real-time parameter and dynamic optimization module, which optimizes and adjusts the scene reconstruction results through a feedback mechanism; a data fusion and decision module, which reconstructs the scene according to the constructed Gaussian splash model; a control and execution module, which executes the driving control strategy according to the results of the scene reconstruction; a user feedback and display module, which displays the reconstructed driving scene information to the user, and the user can also feedback the system results through this module.

[0111] The solution of this embodiment provides an improved Gaussian splash algorithm for reconstruction of assisted driving scenes, aiming to solve the problems of image noise, edge blur, insufficient real-time processing performance and insufficient image reconstruction capability in complex scenes in the prior art. By improving the traditional Gaussian splash algorithm and combining multi-sensor data fusion and dynamic adjustment technology, the accuracy and real-time performance of scene reconstruction can be significantly improved, thereby enhancing the performance of the assisted driving system.

[0112] The present invention provides a method for assisting in reconstruction of a driving scene using an improved Gaussian splash algorithm, the process of which includes: collecting data of the vehicle's surrounding environment through a variety of sensors, including but not limited to laser radar, camera, radar, vehicle-mounted sensor, etc.; preprocessing the collected data, and improving the data quality by denoising, enhancing and standardizing the data; extracting key feature points from the preprocessed image data using an improved feature extraction algorithm, wherein the feature points include edges, corners and texture features of the image, and can accurately describe important information in the image; constructing a Gaussian splash model, and constructing a scene model according to the improved Gaussian splash algorithm, wherein the improvement includes an optimized Gaussian distribution function, an improved splash particle generation mechanism and a dynamic adjustment algorithm; reconstructing the scene according to the constructed Gaussian splash model, generating a scene model that is highly consistent with the actual environment by combining the reconstruction model with the actual image data, and mapping the reconstruction result to the actual environment; optimizing and adjusting the scene reconstruction result by establishing a real-time feedback mechanism to adapt to the dynamically changing driving environment.

[0113] The solution of this embodiment is adopted, by introducing an improved Gaussian splashing algorithm, including multimodal data fusion, improved Gaussian kernel function, enhanced image preprocessing, deep learning integration and real-time processing capability improvement. It can significantly improve the accuracy and real-time performance of the reconstruction of the three-dimensional scene assisted driving system. Multimodal data fusion and deep learning integration improve the performance of the algorithm, making it perform well in complex and dynamic environments, thereby enhancing the reliability and safety of the autonomous driving system.

[0114] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0115] Example 2

[0116] In this embodiment, a device for constructing a scene model is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0117] Figure 5 is a structural block diagram of a scene model construction device according to an embodiment of the present invention, such as Figure 5 As shown, the device comprises:

[0118] An acquisition module 50, for acquiring multimodal data around the vehicle, wherein the multimodal data includes image data, point cloud data, and environmental data;

[0119] An extraction module 52, used to extract a feature point set and texture information of the image data;

[0120] A construction module 54 is used to construct a two-dimensional Gaussian splash model of the feature point set according to the environmental data, wherein the Gaussian splash model is used to simulate the spatial distribution and influence range of the feature points;

[0121] The reconstruction module 56 is used to use the point cloud data and the texture information to perform scene reconstruction on the Gaussian splash model to obtain a three-dimensional scene model of the environment where the vehicle is located.

[0122] Optionally, the extraction module includes: a first detection unit, used to perform edge detection on the image data to obtain edge features of the image data; a second detection unit, used to perform corner point detection on the image data to obtain corner point features of the image data; an extraction unit, used to extract the grayscale co-occurrence matrix of the image data to obtain texture information of the image data; wherein the feature point set includes the edge features and the corner point features.

[0123] Optionally, the device also includes: a segmentation module, used to perform image segmentation on the image data and select a number of regions of interest before the extraction module extracts the feature point set of the image data; a parsing module, used to parse the region attribute information of each region of interest; a setting module, used to set the filtering parameters of the Gaussian filter in the corresponding image region based on the region attribute information; and a denoising module, used to use the Gaussian filter to denoise each region of interest to obtain optimized image data.

[0124] Optionally, the setting module includes: an extraction unit, used to extract the local contrast and noise level in the regional attribute information for each image area; a setting unit, used to set the filtering strength of the Gaussian filter in the corresponding image area based on the local contrast and the noise level, wherein the filtering parameters include the filtering strength, the filtering strength is negatively correlated with the local contrast, and the filtering strength is positively correlated with the noise level.

[0125] Optionally, the construction module includes: a classification unit, used to classify the feature point set according to the feature object type to obtain multiple feature point subsets, wherein each feature point subset corresponds to a scene level; a parsing unit, used to parse the distribution density and environmental complexity of each feature point in the feature point subset; a configuration unit, used to configure the standard deviation of the Gaussian distribution function using the distribution density and the environmental complexity, wherein the standard deviation is used to control the width and influence range of the distribution;

[0126] The first configuration unit is used to configure the Gaussian distribution function using the standard deviation; the generation unit is used to generate a plurality of spatially distributed splash particles according to the Gaussian distribution function to obtain a particle distribution model; the second configuration unit is used to configure the weight coefficient of each splash particle in the particle distribution model to obtain a two-dimensional Gaussian splash model corresponding to the scene level.

[0127] Optionally, the first configuration unit includes: a configuration subunit, configured to configure the following Gaussian distribution function G(x, y) using the standard deviation: Wherein, σ is the standard deviation, exp() is the natural exponential function, and x and y are the two-dimensional coordinate positions of the feature points.

[0128] Optionally, the generating unit includes: a generating subunit, configured to generate a plurality of spatially distributed splash particles P(x, y) using the following formula: (x i ,y i ) is the two-dimensional coordinate position of feature point i, n is the total number of feature points, (x, y) is the three-dimensional horizontal and vertical coordinate position of feature point i, and G() is the Gaussian distribution function.

[0129] Optionally, the second configuration unit includes: an analysis subunit, used to analyze the characteristic intensity of the corresponding feature point for each splatter particle in the particle distribution model; and a configuration subunit, used to configure the weight coefficient of the splatter particle in the particle distribution model based on the characteristic intensity to obtain a two-dimensional Gaussian splatter model corresponding to the scene hierarchy.

[0130] Optionally, the classification unit includes: a determination subunit, used to determine, for each feature point in the feature point set, the feature object type to which the feature point belongs, wherein the feature object types include roads, lane lines, buildings, traffic signs, obstacles, other vehicles, and living things; a classification subunit, used to classify the feature point into a first scene level if the feature point is a road or a lane line; classify the feature point into a second scene level if the feature point is a building, a traffic sign, or an obstacle; and classify the feature point into a third scene level if the feature point is another vehicle or a living thing.

[0131] Optionally, the reconstruction module includes: a mapping unit, used to map the particle distribution in the Gaussian splash model into a three-dimensional environment to generate a basic three-dimensional model; a completion unit, used to use the point cloud data to complete the details of the basic three-dimensional model to obtain an intermediate three-dimensional model; and an application unit, used to apply the texture information to the surface of the intermediate three-dimensional model to obtain a three-dimensional scene model of the environment where the vehicle is located.

[0132] Optionally, the mapping unit includes: a calculation subunit, for calculating the depth coordinate Z of each particle point in the Gaussian splash model using the following formula: Wherein, D is the distance from the camera to the object corresponding to the particle point, f is the focal length of the camera, and d is the pixel distance of the particle point on the two-dimensional image; the conversion subunit is used to convert the corresponding particle point into a three-dimensional grid using the depth coordinate to generate a basic three-dimensional model.

[0133] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0134] Example 3

[0135] An embodiment of the present invention further provides a storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0136] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps:

[0137] S1, acquiring multimodal data around the vehicle, wherein the multimodal data includes image data, point cloud data, and environmental data;

[0138] S2, extracting a feature point set and texture information of the image data;

[0139] S3, constructing a two-dimensional Gaussian splash model of the feature point set according to the environmental data, wherein the Gaussian splash model is used to simulate the spatial distribution and influence range of the feature points;

[0140] S4, using the point cloud data and the texture information to reconstruct the Gaussian splash model to obtain a three-dimensional scene model of the environment where the vehicle is located.

[0141] Optionally, in this embodiment, the above-mentioned storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store computer programs.

[0142] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0143] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0144] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0145] S1, acquiring multimodal data around the vehicle, wherein the multimodal data includes image data, point cloud data, and environmental data;

[0146] S2, extracting a feature point set and texture information of the image data;

[0147] S3, constructing a two-dimensional Gaussian splash model of the feature point set according to the environmental data, wherein the Gaussian splash model is used to simulate the spatial distribution and influence range of the feature points;

[0148] S4, using the point cloud data and the texture information to reconstruct the Gaussian splash model to obtain a three-dimensional scene model of the environment where the vehicle is located.

[0149] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0150] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0151] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0152] It should be understood that the terms used herein are only for the purpose of describing specific example embodiments and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used herein may also be meant to include plural forms. The terms "include", "comprise", "contain", and "have" are inclusive, and therefore specify the existence of stated features, steps, operations, elements and / or parts, but do not exclude the existence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not interpreted as necessarily requiring them to be performed in the specific order described or illustrated, unless the execution order is clearly indicated. It should also be understood that additional or alternative steps may be used.

[0153] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for constructing a scene model, characterized in that: include: Acquire multimodal data around the vehicle, wherein the multimodal data includes image data, point cloud data, and environmental data; Extracting a feature point set and texture information of the image data; Constructing a two-dimensional Gaussian splash model of the feature point set according to the environmental data, wherein the Gaussian splash model is used to simulate the spatial distribution and influence range of the feature points; The point cloud data and the texture information are used to reconstruct the Gaussian splash model to obtain a three-dimensional scene model of the environment where the vehicle is located.

2. The method according to claim 1, characterized in that: Extracting the feature point set and texture information of the image data includes: Performing edge detection on the image data to obtain edge features of the image data; Performing corner point detection on the image data to obtain corner point features of the image data; Extracting a gray level co-occurrence matrix of the image data to obtain texture information of the image data; The feature point set includes the edge features and the corner point features.

3. The method according to claim 1, characterized in that Before extracting the feature point set of the image data, the method further includes: Performing image segmentation on the image data and selecting a number of regions of interest; Parse the regional attribute information of each region of interest; Setting filtering parameters of the Gaussian filter in the corresponding image area based on the area attribute information; The Gaussian filter is used to denoise each region of interest to obtain optimized image data.

4. The method according to claim 3, characterized in that Setting the filtering parameters of the Gaussian filter in the corresponding image area based on the area attribute information includes: For each image region, extracting local contrast and noise level in the region attribute information; The filtering strength of the Gaussian filter in the corresponding image area is set based on the local contrast and the noise level, wherein the filtering parameters include the filtering strength, the filtering strength is negatively correlated with the local contrast, and the filtering strength is positively correlated with the noise level.

5. The method according to claim 1, characterized in that Constructing a two-dimensional Gaussian splash model of the feature point set according to the environmental data, including: Classifying the feature point set according to the feature object type to obtain multiple feature point subsets, wherein each feature point subset corresponds to a scene level; Analyzing the distribution density and environmental complexity of each feature point in the feature point subset; The distribution density and the environment complexity are used to configure the standard deviation of the Gaussian distribution function, wherein the standard deviation is used to control the width and influence range of the distribution; configuring the Gaussian distribution function using the standard deviation; generating a plurality of splash particles distributed in space according to the Gaussian distribution function to obtain a particle distribution model; The weight coefficient of each splash particle is configured in the particle distribution model to obtain a two-dimensional Gaussian splash model corresponding to the scene level.

6. The method according to claim 5, characterized in that Configuring the Gaussian distribution function using the standard deviation includes: The following Gaussian distribution function G(x,y) is configured using the standard deviation: Wherein, σ is the standard deviation, exp() is the natural exponential function, and x and y are the two-dimensional coordinate positions of the feature points.

7. The method according to claim 5, characterized in that Generating a plurality of spatially distributed splash particles according to the Gaussian distribution function includes: The following formula is used to generate multiple spatially distributed splash particles P(x,y): (x i ,y i ) is the two-dimensional coordinate position of feature point i, n is the total number of feature points, (x, y) is the three-dimensional horizontal and vertical coordinate position of feature point i, and G() is the Gaussian distribution function.

8. The method according to claim 5, characterized in that The weight coefficient of each splash particle is configured in the particle distribution model to obtain a two-dimensional Gaussian splash model corresponding to the scene level, including: For each splash particle in the particle distribution model, analyzing the characteristic intensity of the corresponding characteristic point; The weight coefficient of the splash particle in the particle distribution model is configured based on the characteristic intensity to obtain a two-dimensional Gaussian splash model corresponding to the scene level.

9. The method according to claim 5, characterized in that Classifying the feature point set according to the feature object type includes: For each feature point in the feature point set, determine the feature object type to which the feature point belongs, wherein the feature object type includes roads, lane lines, buildings, traffic signs, obstacles, other vehicles, and living things; If the feature point is a road or lane line, the feature point is classified as the first scene level; if the feature point is a building, a traffic sign, or an obstacle, the feature point is classified as the second scene level; if the feature point is another vehicle or a living body, the feature point is classified as the third scene level.

10. The method according to claim 1, characterized in that The point cloud data and the texture information are used to reconstruct the Gaussian splash model to obtain a three-dimensional scene model of the environment where the vehicle is located, including: Mapping the particle distribution in the Gaussian splash model to a three-dimensional environment to generate a basic three-dimensional model; Using the point cloud data to complete the details of the basic three-dimensional model to obtain an intermediate three-dimensional model; The texture information is applied to the surface of the intermediate three-dimensional model to obtain a three-dimensional scene model of the environment where the vehicle is located.

11. The method according to claim 10, characterized in that Mapping the particle distribution in the Gaussian splash model to a three-dimensional environment to generate a basic three-dimensional model includes: For each particle point in the Gaussian splash model, the depth coordinate Z of the particle point is calculated using the following formula: Where D is the distance from the camera to the object corresponding to the particle point, f is the focal length of the camera, and d is the pixel distance of the particle point on the two-dimensional image; The corresponding particle points are converted into a three-dimensional grid using the depth coordinates to generate a basic three-dimensional model.

12. A device for constructing a scene model, characterized in that: include: An acquisition module, used to acquire multimodal data around the vehicle, wherein the multimodal data includes image data, point cloud data, and environmental data; An extraction module, used to extract a feature point set and texture information of the image data; A construction module, used to construct a two-dimensional Gaussian splash model of the feature point set according to the environmental data, wherein the Gaussian splash model is used to simulate the spatial distribution and influence range of the feature points; A reconstruction module is used to use the point cloud data and the texture information to reconstruct the Gaussian splash model to obtain a three-dimensional scene model of the environment where the vehicle is located.

13. A storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the method according to any one of claims 1 to 11 when executed.

14. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 11.

Images