Scene data optimization reconstruction method, system and product

Abstract

The embodiment of the invention provides a scene data optimization reconstruction method, system and product. The method comprises the following steps: carrying out semantic segmentation on image data to extract a static graph; projecting the point cloud of the target laser frame to the image data; searching a nearest feature point around the target point of the point cloud; optimizing the camera attitude according to the distance between the two; projecting the static graph to the point cloud of the target laser frame based on the optimized pose of the camera relative to the laser, and obtaining the color point cloud to construct a static color point cloud map; carrying out downsampling on the static color point cloud map based on the projection times of each point in the point cloud to obtain a merged point of the point cloud; the variance of Gaussian distribution is initialized through the number of projection times of the merging points, the normal vector of Gaussian distribution is initialized through the normal of the merging points, and the initialization result of the merging point area is obtained; and projecting an initialization result to the static graph, updating Gaussian parameters of the merging point region, and obtaining a rendered and reconstructed static color point cloud map. The invention aims to improve the rendering reconstruction effect of scene data.

Description

Technical Field

[0001] The present application relates to the technical field of scene data reconstruction, and in particular to a method, system and product for optimizing and reconstructing scene data. Background Art

[0002] With the development of autonomous driving technology, the collection and use of scene data has become increasingly important. Usually, data is captured by sensors and then directly used in data cleaning and annotation. However, as the technical requirements of autonomous driving technology continue to increase, improving data quality and expanding data coverage through fusion post-processing have also become an important part of data processing. However, currently, only a single sensor is processed to obtain the final target data, and it is impossible to fully utilize multiple sensors to complement each other's information and improve the reliability and stability of target data. Summary of the invention

[0003] In view of this, the embodiments of the present application provide a method, system and product for optimizing and reconstructing scene data, aiming to improve the rendering and reconstruction effect of scene data.

[0004] A first aspect of an embodiment of the present application provides a method for optimizing and reconstructing scene data, the method comprising:

[0005] Performing semantic segmentation on the image data, and extracting static pixels in the image data as a static image;

[0006] Projecting the point cloud data of the target laser frame corresponding to the image data onto the image plane of the image data;

[0007] Searching for feature points in the nearest image feature point matching pair around the target point of the point cloud data;

[0008] Optimize the camera posture according to the distance between the target point and the corresponding feature point;

[0009] Based on the optimized position of the camera relative to the laser, the static image is projected into the point cloud data of the target laser frame to obtain the color point cloud data of the target laser frame and record the number of times each point in the point cloud data is projected;

[0010] Constructing a static color point cloud map through the color point cloud data;

[0011] Based on the recorded number of times each point in the point cloud data is projected, downsampling the static color point cloud map to obtain merged points of the point cloud and the number of projections of the merged points;

[0012] The variance of the Gaussian distribution is initialized by the number of projections of the merged point and the normal vector of the Gaussian distribution is initialized by the normal line of the merged point to obtain the initialization result of the area where each merged point is located;

[0013] The initialization result is projected onto the static image, and the Gaussian parameters of the area where each merging point is located are updated to obtain a rendered and reconstructed static color point cloud map.

[0014] Optionally, the camera pose is optimized based on the distance between the target point and the corresponding feature point, including:

[0015] Determine the depth value deviation and position distance between the feature point and the target point in the camera system;

[0016] Determining whether to retain the target point according to the depth value deviation and the position distance;

[0017] The camera pose is optimized based on the distance between the retained target points and the corresponding feature points.

[0018] Optionally, determining the image data includes:

[0019] Extracting frames from the collected original image data to obtain first image data;

[0020] White balance synchronization processing is performed on the first image data to obtain final image data.

[0021] Optionally, determining a depth value of a feature point in a matching pair of image feature points includes:

[0022] Perform feature point matching on the image data of the previous and next two frames to obtain corresponding image feature point matching pairs;

[0023] Performing depth estimation on the image data of the two frames before and after to obtain a depth map of each image data;

[0024] Calculating the average depth value of the image blocks corresponding to the two feature points in the image feature point matching pair in the respective image data to obtain the corresponding depth value calculation result;

[0025] The depth value calculation result is determined as the depth value of the feature point in the image feature point matching pair.

[0026] Optionally, constructing a static color point cloud map using the color point cloud data includes:

[0027] Extracting a first target point from the color point cloud data of one of the two laser frames, and extracting a color plane from the color point cloud data of the other of the two laser frames;

[0028] constructing a residual equation based on the distance from the first target point to the color plane;

[0029] Optimizing the relative position of the two laser frames based on the residual equation;

[0030] The point cloud coordinates are transformed based on the optimized relative pose to construct a static color point cloud map.

[0031] Optionally, the method further includes:

[0032] When a static color point cloud map has been established, matching the color point cloud data of the laser frame with the static color point cloud map to optimize the laser pose of the laser frame;

[0033] The static color point cloud map is updated based on the color point cloud data corresponding to the optimized laser posture.

[0034] Optionally, the method further includes:

[0035] Performing semantic segmentation on the image data, and extracting dynamic pixels in the image data as a dynamic graph;

[0036] Based on the optimized camera pose, the point cloud data of the laser frame is projected into the corresponding dynamic graph to obtain a dynamic point cloud;

[0037] Clustering the dynamic point cloud into multiple dynamic objects according to image segmentation categories and regions;

[0038] Based on the characteristics of the dynamic objects in the dynamic images, the dynamic objects in the previous and next two frames of dynamic images are associated to construct a motion model of each dynamic object;

[0039] Projecting the dynamic point clouds of the same dynamic object in the two previous and next laser frames into the corresponding image data, and searching for the nearest matching pair of image feature points in the image data around the projected dynamic point clouds;

[0040] According to the searched image feature point matching pairs, establishing the association relationship between the dynamic point clouds of the same dynamic object in the two previous and next laser frames;

[0041] According to the association relationship and the laser sampling interval, constructing the orientation transformation relationship of the dynamic point cloud of the same dynamic object in the two preceding and following laser frames;

[0042] Determining motion information of the same dynamic object according to the orientation transformation relationship;

[0043] According to the timestamp of the laser frame and the motion information of each dynamic object, motion compensation is performed on the dynamic point cloud corresponding to each dynamic object to obtain the corresponding target laser frame;

[0044] Predicting a predicted laser frame of the target laser frame at the next moment according to the motion model, the azimuth transformation relationship and the target laser frame;

[0045] Motion compensation is performed on the dynamic object in the real laser frame at the next moment of the target laser frame according to the laser sampling frequency and the motion information of the dynamic point cloud in the predicted laser frame, so as to obtain a compensated laser frame corresponding to the real laser frame;

[0046] By projecting the corresponding dynamic image onto the compensation laser frame, a color point cloud of the compensation laser frame is obtained and the number of times each point in the compensation laser frame is projected is recorded.

[0047] Optionally, before determining the motion information of the same dynamic object according to the orientation transformation relationship, the method further includes:

[0048] According to the orientation transformation relationship, filtering invalid points in the dynamic point cloud of the same dynamic object;

[0049] Determining the motion information of the same dynamic object according to the orientation transformation relationship includes:

[0050] The motion information of the same dynamic object is determined according to the remaining orientation transformation relationship of the dynamic point cloud of the same dynamic object.

[0051] Optionally, the method further includes:

[0052] Based on the recorded number of times each point in the compensated laser frame is projected, downsampling the color point cloud to obtain a merged point of the point cloud and the number of projections of the merged point;

[0053] The variance of the Gaussian distribution is initialized by the number of projections of the merged point and the normal vector of the Gaussian distribution is initialized by the normal line of the merged point to obtain the initialization result of the area where each merged point is located;

[0054] The initialization result is projected onto the dynamic graph, and the Gaussian parameters of the regions where the merged points are located are updated to obtain a color point cloud of the compensated laser frame reconstructed by rendering.

[0055] Optionally, the method further includes:

[0056] According to the current laser posture, point cloud data of the corresponding range is extracted from the rendered and reconstructed static color point cloud map, and the color point cloud of the compensated laser frame obtained by rendering and reconstruction corresponding to the current laser posture is placed in the extracted point cloud data to form laser perception output.

[0057] Optionally, the method further includes:

[0058] Determine the transmission time error of the collected raw image data according to the optimized camera posture and vehicle speed;

[0059] According to the transmission time error, time compensation is performed on the timestamp of data acquisition.

[0060] A second aspect of an embodiment of the present application provides a system for optimizing and reconstructing scene data, the system comprising:

[0061] A semantic segmentation module, used to perform semantic segmentation on the image data and extract static pixels in the image data as a static image;

[0062] A projection module, used for projecting the point cloud data of the target laser frame corresponding to the image data onto the image plane of the image data;

[0063] A search module, used for searching for a feature point in a nearest image feature point matching pair around a target point in the point cloud data;

[0064] The optimization module is used to optimize the camera posture according to the distance between the target point and the corresponding feature point;

[0065] A color point cloud determination module is used to project the static image into the point cloud data of the target laser frame based on the optimized position of the camera relative to the laser, obtain the color point cloud data of the target laser frame and record the number of times each point in the point cloud data is projected;

[0066] A point cloud map construction module, used to construct a static color point cloud map through the color point cloud data;

[0067] A downsampling module, used to downsample the static color point cloud map based on the recorded number of times each point in the point cloud data is projected, to obtain the merged points of the point cloud and the number of projections of the merged points;

[0068] An initialization module, used to initialize the variance of the Gaussian distribution by the number of projections of the merged point and to initialize the normal vector of the Gaussian distribution by the normal line of the merged point, so as to obtain the initialization result of the area where each merged point is located;

[0069] The rendering and reconstruction module is used to update the Gaussian parameters of the area where each merging point is located by projecting the initialization result onto the static image, so as to obtain a static color point cloud map reconstructed by rendering.

[0070] A third aspect of an embodiment of the present application provides an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and running on the processor, wherein when the computer program is executed by the processor, the steps in a method for optimizing and reconstructing scene data as described in the first aspect of the present application are implemented.

[0071] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in a method for optimizing and reconstructing scene data as described in the first aspect of the present application are implemented.

[0072] The method for optimizing and reconstructing scene data provided by this application has the following advantages:

[0073] The embodiment of the present application provides a method for optimizing and reconstructing scene data. First, semantic segmentation is performed on image data to extract static pixels in the image data as a static image; point cloud data of a target laser frame corresponding to the image data is projected onto an image plane of the image data; the nearest image feature point is searched around the target point in the point cloud data to match the feature points in the pair; the camera posture is optimized according to the distance between the target point and the corresponding feature point; based on the optimized camera posture relative to the laser, the static image is projected onto the point cloud data of the target laser frame to obtain the color point cloud data of the target laser frame and the number of times each point in the point cloud data is projected is recorded; a static color point cloud map is constructed through the color point cloud data; based on the number of times each point in the recorded point cloud data is projected, the static color point cloud map is downsampled to obtain the merged points of the point cloud and the number of projections of the merged points; the variance of the Gaussian distribution is initialized by the number of projections of the merged points and the normal vector of the Gaussian distribution is initialized by the normal of the merged points to obtain the initialization results of the areas where each merged point is located; the Gaussian parameters of the areas where each merged point is located are updated by projecting the initialization results onto the static image to obtain a rendered and reconstructed static color point cloud map. Therefore, the present application segments the static objects in the image data, projects the static image composed of the static objects onto the point cloud data to construct a static color point cloud map, and optimizes the camera posture in the process to make the projection more accurate and thus obtain a more accurate static color point cloud map. For the static color point cloud map obtained in this way, the present application will further update the Gaussian parameters of the static color point cloud map to further perform scene rendering and reconstruction on the obtained static color point cloud map, so as to obtain a final more accurate static color point cloud map that better matches the real scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor.

[0075] Figure 1 A flow chart of a method for optimizing and reconstructing scene data according to an embodiment of the present application;

[0076] Figure 2 This is a flowchart for optimizing dynamic and static color point cloud data in a method for optimizing and reconstructing scene data according to an embodiment of the present application;

[0077] Figure 3 A flowchart of reconstruction of image scene data and laser point cloud scene data in a method for optimizing reconstruction of scene data shown in one embodiment of the present application;

[0078] Figure 4 A schematic diagram of a system for optimizing and reconstructing scene data according to an embodiment of the present application;

[0079] Figure 5 Another schematic diagram of a system for optimizing and reconstructing scene data according to an embodiment of the present application. DETAILED DESCRIPTION

[0080] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0081] refer to Figure 1 , Figure 1 This is a flow chart of a method for optimizing and reconstructing scene data according to an embodiment of the present application. Figure 1 As shown, the method includes:

[0082] Step S1: performing semantic segmentation on the image data, and extracting static pixels in the image data as a static image.

[0083] In this embodiment, each image acquisition device configured in the vehicle acquires image data, performs semantic segmentation processing on the acquired image data, extracts static pixels in the image data, and then determines the extracted static pixels as a static image. The static pixels refer to static pixel points in the image data, such as lanes, lane lines, static objects around lanes, etc.

[0084] Step S2: Projecting the point cloud data of the target laser frame corresponding to the image data onto the image plane of the image data.

[0085] In this embodiment, since each image data collected by each camera participates in the same processing mode in the scene data optimization and reconstruction process, for ease of understanding, the subsequent implementation methods are described using one image data as an example. According to the timestamp of the image data, the laser frame closest to the timestamp in the laser frames collected by the vehicle's laser radar is determined to be the target laser frame. Then, the point cloud data of the target laser frame is projected onto the image plane of the image data.

[0086] Step S3: Searching for feature points in the nearest image feature point matching pair around the target point in the point cloud data.

[0087] In this embodiment, after the point cloud data of the target laser frame is projected onto the image plane of the image data, a feature point belonging to an image feature matching pair closest to the target point is searched around the target point in the point cloud data of the image plane. The target point is any point in the point cloud data.

[0088] Step S4: Optimize the camera posture according to the distance between the target point and the corresponding feature point.

[0089] In this embodiment, after searching for the feature point closest to the target point in the point cloud data of the image plane, the camera posture is optimized based on the distance deviation between each target point and the corresponding feature point to obtain an optimized camera posture.

[0090] Step S5: Based on the optimized position of the camera relative to the laser, the static image is projected into the point cloud data of the target laser frame to obtain the color point cloud data of the target laser frame and record the number of times each point in the point cloud data is projected.

[0091] In this embodiment, after the optimized camera posture is obtained through step S4, based on the posture of the laser radar corresponding to the optimized camera posture relative to the target laser frame, the static image extracted from the image data is projected into the point cloud data of the target laser frame to obtain the color point cloud data of the target laser frame and record the number of times each point in the point cloud data of the target laser frame is projected. The number of projections is determined as follows: the distances between the pixel points of the static image and all the points in the point cloud data of the target laser frame are calculated respectively, and the nearest point is found, and the color of the nearest point found is set to the color of the pixel point. If the point has been colored before, the number of times the point is projected is increased.

[0092] Step S6: constructing a static color point cloud map using the color point cloud data.

[0093] In this embodiment, if Figure 2As shown, through the implementation of the above steps S1 to S5, color point cloud data of laser frames in a large number of different road scenes can be determined, and a static color point cloud map of urban roads can be constructed based on the obtained large amount of color point cloud data.

[0094] Step S7: based on the recorded number of times each point in the point cloud data is projected, downsampling the static color point cloud map is performed to obtain the merged points of the point cloud and the number of projections of the merged points.

[0095] In this embodiment, based on the number of times each point in the color point cloud data of the recorded target laser frame is projected, the point cloud data at the location where the color point cloud data is located in the constructed static color point cloud map is downsampled. During the downsampling process, the color information of each point is checked, and points of the same color are grouped together and merged, points of different colors are retained, and the number of projections for the merged points is accumulated to the merged point, so as to retain the statistical characteristics of the original data after downsampling.

[0096] Step S8: Initialize the variance of the Gaussian distribution by the number of projections of the merged point and initialize the normal vector of the Gaussian distribution by the normal line of the merged point to obtain the initialization result of the area where each merged point is located.

[0097] In this embodiment, the variance of the Gaussian distribution is initialized by the number of projections of the merged point of the color point cloud data, and the normal vector of the Gaussian distribution is initialized by the normal of the merged point to obtain the initialization result of the area (splat) where the merged point is located. Based on the same implementation method, the initialization results of the areas where each merged point is located can be obtained.

[0098] Step S9: by projecting the initialization result onto the static image, the Gaussian parameters of the area where each merging point is located are updated to obtain a rendered and reconstructed static color point cloud map.

[0099] In this embodiment, after obtaining the initialization results of the areas where each merged point is located through step S8, all the initialization results are projected onto the corresponding static image, the photometric error of the static image is calculated, and then the Gaussian parameters of the areas where each merged point is located are updated based on the photometric error to obtain the final static color point cloud map rendered and reconstructed.

[0100] The embodiment of the present application provides a method for optimizing and reconstructing scene data. First, semantic segmentation is performed on image data to extract static pixels in the image data as a static image; point cloud data of a target laser frame corresponding to the image data is projected onto an image plane of the image data; the nearest image feature point is searched around the target point in the point cloud data to match the feature points in the pair; the camera posture is optimized according to the distance between the target point and the corresponding feature point; based on the optimized camera posture relative to the laser, the static image is projected onto the point cloud data of the target laser frame to obtain the color point cloud data of the target laser frame and the number of times each point in the point cloud data is projected is recorded; a static color point cloud map is constructed through the color point cloud data; based on the number of times each point in the recorded point cloud data is projected, the static color point cloud map is downsampled to obtain the merged points of the point cloud and the number of projections of the merged points; the variance of the Gaussian distribution is initialized by the number of projections of the merged points and the normal vector of the Gaussian distribution is initialized by the normal of the merged points to obtain the initialization results of the areas where each merged point is located; the Gaussian parameters of the areas where each merged point is located are updated by projecting the initialization results onto the static image to obtain a rendered and reconstructed static color point cloud map. Therefore, the present application segments the static objects in the image data, projects the static image composed of the static objects onto the point cloud data to construct a static color point cloud map, and optimizes the camera posture in the process to make the projection more accurate and thus obtain a more accurate static color point cloud map. For the static color point cloud map obtained in this way, the present application will further update the Gaussian parameters of the static color point cloud map to further perform scene rendering and reconstruction on the obtained static color point cloud map, so as to obtain a final more accurate static color point cloud map that better matches the real scene.

[0101] In combination with the above embodiments, in one implementation, the present application embodiment further provides a method for optimizing and reconstructing scene data. In the method for optimizing and reconstructing scene data, step S4 may include steps S41 to S43:

[0102] Step S41: Determine the depth value deviation and position distance between the feature point and the target point in the camera system.

[0103] In this embodiment, in order to avoid abnormal deviations between the depth values ​​and / or positions of some points in the point cloud data and the feature points in the image data, which leads to poor optimization effects, the present application will filter out some points in the point cloud data.

[0104] Specifically, after searching for the feature point closest to the target point in the point cloud data of the image plane, the depth value deviation and position distance between the target point and the feature point are determined, where the target point will have a depth value, and the feature point will also have a depth value accordingly, and the difference between the two depth values ​​is the depth value deviation.

[0105] Step S42: Determine whether to retain the target point according to the depth value deviation and the position distance.

[0106] In this embodiment, based on the depth value deviation and position distance between the target point and the feature point determined in step S41, if the depth value deviation is greater than the first threshold and / or the position distance is greater than the second threshold, the target point is deleted; and if the depth value deviation is less than or equal to the first threshold and the position distance is less than or equal to the second threshold, the target point is retained. Through the same implementation method, it can be determined which target points will be deleted and which target points will be retained.

[0107] Step S43: Optimize the camera posture according to the distance between the retained target point and the corresponding feature point.

[0108] In this embodiment, based on the distance between all the retained target points and their corresponding feature points, the camera posture of the camera corresponding to the image data corresponding to the feature point is optimized. Specifically: the distance between all the retained target points and their corresponding feature points is used as the error value, the camera posture of the previous and next two frames of image data is used as the optimization item, and the laser posture is used as a fixed value, and BA optimization is constructed to optimize the camera posture.

[0109] In combination with the above embodiments, in one implementation, the embodiment of the present application further provides a method for optimizing and reconstructing scene data. In the method for optimizing and reconstructing scene data, determining the image data includes: extracting frames from the collected original image data to obtain first image data; and performing white balance synchronization processing on the first image data to obtain final image data.

[0110] In this embodiment, the original image data collected by each image device is firstly extracted to reduce the image frame rate and obtain the first image data. With respect to the obtained first image data, since the first image data includes images from different image acquisition devices, and different image acquisition devices use dynamic white balance, resulting in the image colors collected by different image acquisition devices being inconsistent, the present application performs white balance synchronization processing on the obtained first image data to ensure color consistency and obtain the image data ultimately used for optimizing and reconstructing the scene data. With respect to the collected laser frame, the point cloud data of the laser frame is motion compensated based on the vehicle speed at the time of collection, and the point cloud data of the laser frame used subsequently are all point cloud data after motion compensation based on the vehicle speed.

[0111] In combination with the above embodiments, in one implementation, the embodiment of the present application further provides a method for optimizing and reconstructing scene data. In this method for optimizing and reconstructing scene data, determining the depth value of a feature point in an image feature point matching pair includes: performing feature point matching on the image data of the two frames before and after to obtain a corresponding image feature point matching pair; performing depth estimation on the image data of the two frames before and after to obtain a depth map of each image data; performing average depth value calculation on the image blocks corresponding to the two feature points in the image feature point matching pair in their respective image data to obtain a corresponding depth value calculation result; and determining the depth value calculation result as the depth value of the feature point in the image feature point matching pair.

[0112] In this embodiment, feature point matching is performed on the image data of the two frames before and after to obtain multiple pairs of image feature point matching pairs. Depth estimation is performed on the image data of the two frames before and after to obtain a depth map of each image data. For each feature point, an image block of a preset area is divided with the feature point as the center, and for the image block, the average depth value of the image block is calculated based on the depth map of the image data, thereby obtaining a corresponding average depth value for each feature point, and then for each image feature point matching pair, the two average depth values ​​of the two feature points in the single image feature point matching pair are averaged to obtain a depth value, which is the depth value of the single image feature point matching pair and also the depth value of the two feature points in the single image feature point matching pair.

[0113] In combination with the above embodiments, in one implementation, the present application embodiment further provides a method for optimizing and reconstructing scene data. In the method for optimizing and reconstructing scene data, step S6 may include steps S61 to S64:

[0114] Step S61: extracting a first target point from the color point cloud data of one of the two laser frames, and extracting a color plane from the color point cloud data of the other of the two laser frames.

[0115] In this embodiment, a first target point is extracted from the color point cloud data of one of the two laser frames, and then a second target point at the same position as the first target point is located in the color point cloud data of the other of the two laser frames, and then a color plane with a similar color is extracted around the second target point. The first target point can be any point in the color point cloud data.

[0116] Step S62: constructing a residual equation based on the distance from the first target point to the color plane.

[0117] In this embodiment, a plurality of first target points and color planes corresponding to the first target points are extracted, and a residual equation is constructed based on the distances from the extracted first target points to the corresponding color planes.

[0118] Step S63: Optimizing the relative position of the two laser frames based on the residual equation.

[0119] In this embodiment, the relative posture of the two laser frames is optimized based on the constructed residual equation to obtain the optimized relative posture.

[0120] Step S64: transform the point cloud coordinates based on the optimized relative posture to construct a static color point cloud map.

[0121] In this embodiment, the point cloud coordinates of the color point cloud data in the latter laser frame of the two laser frames are converted based on the optimized relative posture to optimize the color point cloud data in the latter laser frame, and then the optimized color point cloud data are used to construct a static color point cloud map of urban roads. It should be understood that a large amount of optimized color point cloud data of different locations and scenes are required when constructing the static color point cloud map of urban roads here.

[0122] In combination with the above embodiments, in one implementation, the present application embodiment further provides a method for optimizing and reconstructing scene data. In the method for optimizing and reconstructing scene data, the method further includes steps S65 to S66:

[0123] Step S65: When a static color point cloud map has been established, the color point cloud data of the laser frame is matched with the static color point cloud map to optimize the laser pose of the laser frame.

[0124] In this embodiment, when a static color point cloud map has been established, the color point cloud data of the obtained laser frame is matched with the static color point cloud map to determine the location where the color point cloud data is collected, and then the laser posture of the laser frame is optimized based on the deviation between the color point cloud data in the matched static color point cloud map and the color point cloud data of the laser frame.

[0125] Step S66: updating the static color point cloud map based on the color point cloud data corresponding to the optimized laser posture.

[0126] In this embodiment, the color point cloud data of the laser frame is adjusted based on the optimized laser posture, and then the static color point cloud map is updated with the adjusted color point cloud data.

[0127] In combination with the above embodiments, in one implementation, the present application embodiment further provides a method for optimizing and reconstructing scene data. In the method for optimizing and reconstructing scene data, the method further includes steps S01 to S012:

[0128] Step S01: performing semantic segmentation on image data, and extracting dynamic pixels in the image data as a dynamic image.

[0129] In this embodiment, each image acquisition device configured in the vehicle acquires image data, extracts dynamic pixels from the image data through semantic segmentation, and then determines a dynamic graph based on the extracted dynamic pixels. The dynamic pixels refer to dynamic pixels that are constantly moving in the image data, such as other vehicles. In this embodiment, since the subsequent processing implementation methods of each dynamic graph are the same, a dynamic graph is used as an example for explanation.

[0130] Step S02: Based on the optimized camera posture, the point cloud data of the laser frame is projected into the corresponding dynamic map to obtain a dynamic point cloud.

[0131] In this embodiment, based on the camera posture optimized in step S4, the point cloud data of the laser frame closest to the timestamp of the extracted dynamic image is projected into the dynamic image to obtain the dynamic point cloud of the laser frame.

[0132] Step S03: clustering the dynamic point cloud into multiple dynamic objects according to the image segmentation category and region.

[0133] In this embodiment, the various moving dynamic objects in the dynamic image can be segmented out according to the image segmentation category and area of ​​the dynamic image, and the point cloud data of the laser frame is projected onto the dynamic image, and the dynamic point cloud of the laser frame can be clustered into various dynamic objects accordingly.

[0134] Step S04: Based on the features of the dynamic objects in the dynamic image, the dynamic objects in the two frames of the dynamic image are associated to construct a motion model of each dynamic object.

[0135] In this embodiment, based on the characteristics of the dynamic objects in each dynamic image, the dynamic objects in the dynamic image are associated with the dynamic objects in the dynamic image of the next frame, and based on the association, a motion model of each dynamic object from the dynamic image to the next frame is established.

[0136] Step S05: projecting the dynamic point clouds of the same dynamic object in the two preceding and succeeding laser frames into the corresponding image data, and searching for the nearest matching pair of image feature points in the image data around the projected dynamic point clouds.

[0137] In this embodiment, the dynamic point clouds of the same dynamic object in the dynamic image in the two laser frames closest to the dynamic object are projected into the corresponding image data. Specifically, the dynamic point cloud in the first laser frame of the two laser frames is projected into the image data to which the dynamic image belongs, and the dynamic point cloud in the second laser frame of the two laser frames is projected into the image data to which the dynamic image belongs at the next moment of the dynamic image. After the projection is completed, the image feature point matching pairs closest to the dynamic point cloud are searched in the two image data respectively.

[0138] Step S06: establishing an association relationship between the dynamic point clouds of the same dynamic object in the two preceding and following laser frames based on the searched image feature point matching pairs.

[0139] In this embodiment, based on the searched image feature point matching pairs, an association relationship between the dynamic point clouds of the same dynamic object in the last two laser frames in step S05 is established.

[0140] For example, dynamic point cloud a is a dynamic point cloud corresponding to a dynamic object in the previous laser frame of the last two laser frames, and dynamic point cloud b is a dynamic point cloud corresponding to a dynamic object in the last laser frame of the last two laser frames. Dynamic point cloud a will be projected into image data A, and dynamic point cloud b will be projected into image data B. Image data A and image data B construct image feature point matching pairs. When searching for the nearest feature point near dynamic point cloud a in image data A, feature point x1 is searched. When searching for the nearest feature point near dynamic point cloud b in image data B, feature point x2 is searched. Feature point x1 and feature point x2 are two feature points in the same image feature matching pair. Therefore, it is determined that dynamic point cloud a and dynamic point cloud b belong to the dynamic point clouds of the same dynamic object. Therefore, the association relationship of the same object in the last two laser frames is established as the association between dynamic point cloud a and dynamic point cloud b.

[0141] Step S07: constructing the orientation transformation relationship of the dynamic point cloud of the same dynamic object in the two preceding and succeeding laser frames according to the association relationship and the laser sampling interval.

[0142] In this embodiment, based on the association relationship between the same dynamic object in the last two laser frames in step S5 and the sampling interval between two adjacent laser frames, the orientation transformation relationship of the dynamic point cloud of the same dynamic object in the last two laser frames is constructed by the ICP (Iterative Closest Point) algorithm.

[0143] Step S08: determining the motion information of the same dynamic object according to the orientation transformation relationship.

[0144] In this embodiment, based on the azimuth transformation relationship of the dynamic point cloud of the same dynamic object in the last two laser frames, the motion information of the same dynamic object is calculated by combining the translation vector and rotation matrix in the azimuth transformation relationship with the time interval of the two laser frames, and the motion information includes at least the average velocity vector and the average angular velocity vector.

[0145] Step S09: According to the timestamp of the laser frame and the motion information of each dynamic object, motion compensation is performed on the dynamic point cloud corresponding to each dynamic object to obtain the corresponding target laser frame.

[0146] In this embodiment, after the motion information of each dynamic object is determined, motion compensation is performed on the dynamic point cloud of each dynamic object in the latter laser frame based on the motion information of each dynamic object and the timestamp of the latter laser frame in the latter two laser frames to obtain the motion compensated target laser frame corresponding to the latter laser frame.

[0147] Step S010: predicting a predicted laser frame at the next moment of the target laser frame according to the motion model, the orientation transformation relationship and the target laser frame.

[0148] In this embodiment, based on the orientation transformation relationship constructed in step S07 and the target laser frame after motion compensation in step S09, the laser frame at the next moment of the target laser frame is predicted by the motion model constructed in step S04 to obtain a corresponding predicted laser frame.

[0149] Step S011: Motion compensation is performed on the dynamic object in the real laser frame at the next moment of the target laser frame according to the laser sampling frequency and the motion information of the dynamic point cloud in the predicted laser frame to obtain a compensated laser frame corresponding to the real laser frame.

[0150] In this embodiment, for the predicted laser frame determined by step S010, based on the motion information of the dynamic point cloud in the predicted laser frame and the laser sampling frequency, motion compensation is performed on the dynamic point cloud of the dynamic object in the real laser frame at the next moment of the target laser frame to obtain a compensated laser frame corresponding to the real laser frame.

[0151] Step S012: projecting the corresponding dynamic image onto the compensated laser frame to obtain a color point cloud of the compensated laser frame and recording the number of times each point in the compensated laser frame is projected.

[0152] In this embodiment, if Figure 2 As shown, after obtaining the compensated laser frame corresponding to the real laser frame through step S011, the dynamic image with the closest timestamp to the compensated laser frame is projected onto the compensated laser frame to obtain the color point cloud of the compensated laser frame and record the number of times each point in the compensated laser frame is projected.

[0153] In this embodiment, before projecting the dynamic image whose timestamp is closest to the compensated laser frame to the compensated laser frame, the image position of the dynamic object in the dynamic image can be compensated and the distortion can be corrected through the rolling exposure model, and then the compensated dynamic image is projected to the compensated laser frame to obtain the color point cloud of the compensated laser frame and record the number of times each point in the compensated laser frame is projected.

[0154] In combination with the above embodiments, in one implementation, the present application embodiment further provides a method for optimizing and reconstructing scene data. In the method for optimizing and reconstructing scene data, before step S08, the method further includes: filtering invalid points in the dynamic point cloud of the same dynamic object according to the orientation transformation relationship.

[0155] In this embodiment, after the orientation transformation relationship of the same dynamic object is determined through step S08, the points in the dynamic point cloud of the same dynamic object whose orientation transformation exceeds the set threshold of the orientation transformation relationship are determined as abnormal invalid points, and such invalid points are filtered out to avoid excessive errors in the subsequent determination of the dynamic object motion information.

[0156] In the present application, when the method further includes S081, step S08 includes: determining the motion information of the same dynamic object according to the remaining orientation transformation relationship of the dynamic point cloud of the same dynamic object.

[0157] In this embodiment, based on the orientation transformation relationship of the remaining dynamic point clouds of the same dynamic object, the motion information of the same dynamic object is calculated by combining the translation vector and the rotation matrix in the orientation transformation relationship with the time interval between two laser frames.

[0158] In combination with the above embodiments, in one implementation, the present application embodiment further provides a method for optimizing and reconstructing scene data. In the method for optimizing and reconstructing scene data, the method further includes steps S013 to S015:

[0159] Step S013: based on the recorded number of times each point in the compensated laser frame is projected, downsampling the color point cloud is performed to obtain the merged point of the point cloud and the number of projections of the merged point.

[0160] In this embodiment, based on the number of times each point in the color point cloud data of the compensated laser frame recorded in step S012 is projected, the color point cloud data of the compensated laser frame is downsampled. During the downsampling process, the points of the same color are grouped together and merged by checking the color information of each point, and points of different colors are retained. The number of projections for the merged points is accumulated to the merged points, so as to retain the statistical characteristics of the original data after downsampling.

[0161] Step S014: Initialize the variance of the Gaussian distribution by the number of projections of the merged point and initialize the normal vector of the Gaussian distribution by the normal line of the merged point to obtain the initialization result of the area where each merged point is located.

[0162] In this embodiment, the variance of the Gaussian distribution is initialized by the number of projections of the merged point of the color point cloud data in step S013, and the normal vector of the Gaussian distribution is initialized by the normal of the merged point in step S013 to obtain the initialization result of the area (splat) where the merged point is located. Based on the same implementation method, the initialization results of the areas where each merged point is located can be obtained.

[0163] Step S015: by projecting the initialization result onto the dynamic graph, the Gaussian parameters of the area where each merging point is located are updated to obtain a color point cloud of the compensated laser frame reconstructed by rendering.

[0164] In this embodiment, after obtaining the initialization results of the areas where each merged point is located through step S014, all the initialization results are projected onto the corresponding dynamic image, the photometric error of the dynamic image is calculated, and then the Gaussian parameters of the areas where each merged point is located are updated based on the photometric error to obtain the color point cloud data of the final compensated laser frame rendered and reconstructed.

[0165] In combination with the above embodiments, in one implementation, the embodiment of the present application also provides a method for optimizing and reconstructing scene data. In the method for optimizing and reconstructing scene data, the method further includes: extracting point cloud data of a corresponding range from a static color point cloud map reconstructed by rendering according to the current laser posture, and placing the color point cloud of the compensated laser frame obtained by rendering and reconstruction corresponding to the current laser posture into the extracted point cloud data to form a laser perception output.

[0166] In this embodiment, if Figure 3 As shown, based on the current laser posture, point cloud data within the range corresponding to the current laser posture is extracted from the static color point cloud map rendered in step S9, and then the color point cloud data of the compensated laser frame obtained by rendering and reconstruction corresponding to the current laser posture is placed in the extracted point cloud data to form the perception output of the laser radar.

[0167] In this embodiment, when outputting the point cloud data collected at a certain speed, the motion error at the current self-vehicle speed is superimposed on the static point cloud data within the range according to the scanning time. For the moving point, the motion error generated by its speed and the motion error of the self-vehicle are added. When outputting the image data, the static color point cloud map in the middle of the rendering is used to project the spalt to the image plane in combination with the camera's posture, and the superimposed rendering is performed. The dynamic object is projected to the image plane in the same way in combination with the motion trajectory information for rendering and reconstruction. When outputting the image data collected at a certain speed, the static points to be rendered to the image are sequentially superimposed with motion offsets in order from top to bottom, and then projected to the image for rendering. For the moving point, the motion offset generated by its speed and the motion offset of the self-vehicle need to be added, and then projected to the image for rendering. When outputting the image data collected in different weather and lighting conditions, a color mask is superimposed on the rendered image to simulate the image in the scene. When outputting the image data of different types of cameras, different camera models are used to calculate the projection coordinates of the rendering point to the image, and then rendering is performed. Every time new collected data is received, the data is generated into dynamic and static scenes through the computing platform and stored in the computing platform together with other data. When it is needed, the user designs the demand script, configures the rendering module according to the demand script, and renders the data required by the demander from the dynamic and static scenes on the computing platform. Then, the rendered data and other collected data are synchronously processed through the platform, and then formed into a data set and fed back to the data user.

[0168] In combination with the above embodiments, in one implementation, the present application embodiment further provides a method for optimizing and reconstructing scene data. In the method for optimizing and reconstructing scene data, the method further includes:

[0169] Step S001: Determine the transmission time error of the collected original image data according to the optimized camera posture and vehicle speed.

[0170] In this embodiment, based on the optimized camera posture and the current vehicle speed, the transmission time error of the collected original head portrait mutual data is determined.

[0171] Step S002: Perform time compensation on the timestamp of data collection according to the transmission time error.

[0172] In this embodiment, based on the determined transmission time error, time compensation is performed on the timestamp of the data set for data collection, so that the collection timestamp of the collected data is more accurate.

[0173] Based on the same inventive concept, an embodiment of the present application provides a system for optimizing and reconstructing scene data, such as Figure 4 As shown, the system 400 includes:

[0174] The semantic segmentation module 401 is used to perform semantic segmentation on the image data and extract static pixels in the image data as a static image;

[0175] A projection module 402 is used to project the point cloud data of the target laser frame corresponding to the image data onto an image plane of the image data;

[0176] A search module 403, configured to search for a feature point in a nearest matching pair of image feature points around a target point in the point cloud data;

[0177] An optimization module 404, configured to optimize the camera posture according to the distance between the target point and the corresponding feature point;

[0178] A color point cloud determination module 405 is used to project the static image into the point cloud data of the target laser frame based on the optimized position of the camera relative to the laser, obtain the color point cloud data of the target laser frame and record the number of times each point in the point cloud data is projected;

[0179] A point cloud map construction module 406 is used to construct a static color point cloud map using the color point cloud data;

[0180] A downsampling module 407 is used to downsample the static color point cloud map based on the recorded number of times each point in the point cloud data is projected, to obtain the merged points of the point cloud and the number of projections of the merged points;

[0181] Initialization module 408, used to initialize the variance of Gaussian distribution by the number of projections of the merged point and initialize the normal vector of Gaussian distribution by the normal line of the merged point, so as to obtain the initialization result of the area where each merged point is located;

[0182] The rendering and reconstruction module 409 is used to update the Gaussian parameters of the area where each merging point is located by projecting the initialization result onto the static image, so as to obtain a rendered and reconstructed static color point cloud map.

[0183] Optionally, the optimization module 404 includes:

[0184] A deviation determination module, used to determine the depth value deviation and position distance between the feature point and the target point in the camera system;

[0185] A reserved point determination module, used to determine whether to retain the target point according to the depth value deviation and the position distance;

[0186] The optimization submodule is used to optimize the camera posture according to the distance between the retained target points and the corresponding feature points.

[0187] Optionally, the system 400 further includes an image data determination module, which is used to determine the image data; the image data determination module includes:

[0188] A frame extraction module, used for extracting frames from the collected original image data to obtain first image data;

[0189] The white balance synchronization module is used to perform white balance synchronization processing on the first image data to obtain final image data.

[0190] Optionally, the system 400 further includes a depth value determination module, which is used to determine the depth value of the feature point in the image feature point matching pair; the depth value determination module includes:

[0191] A feature matching module is used to match feature points of image data of two frames to obtain corresponding image feature point matching pairs;

[0192] A depth estimation module, used to perform depth estimation on the image data of the two frames before and after to obtain a depth map of each image data;

[0193] A depth value calculation module is used to calculate the average depth value of the image blocks corresponding to the two feature points in the image feature point matching pair in their respective image data to obtain the corresponding depth value calculation result;

[0194] The depth value determination submodule is used to determine the depth value calculation result as the depth value of the feature point in the image feature point matching pair.

[0195] Optionally, the point cloud map building module 406 includes:

[0196] A data extraction module, used to extract a first target point from the color point cloud data of one of the two laser frames, and to extract a color plane from the color point cloud data of the other of the two laser frames;

[0197] A residual construction module, used for constructing a residual equation based on the distance from the first target point to the color plane;

[0198] A relative posture optimization module, used for optimizing the relative posture of the two laser frames before and after based on the residual equation;

[0199] The point cloud map construction submodule is used to transform the point cloud coordinates based on the optimized relative pose and construct a static color point cloud map.

[0200] Optionally, the system 400 further includes:

[0201] A matching module, for matching the color point cloud data of the laser frame with the static color point cloud map when a static color point cloud map has been established, so as to optimize the laser pose of the laser frame;

[0202] An updating module is used to update the static color point cloud map based on the color point cloud data corresponding to the optimized laser posture.

[0203] Optionally, the system 400 further includes:

[0204] A first semantic segmentation module, used to perform semantic segmentation on the image data, and extract dynamic pixels in the image data as a dynamic image;

[0205] A first projection module is used to project the point cloud data of the laser frame into the corresponding dynamic map based on the optimized camera posture to obtain a dynamic point cloud;

[0206] A clustering module, used for clustering the dynamic point cloud into multiple dynamic objects according to image segmentation categories and regions;

[0207] A motion model building module, used to associate the dynamic objects in the previous and next two frames of dynamic images based on the features of the dynamic objects in the dynamic images, and to build a motion model for each dynamic object;

[0208] A search module is used to project the dynamic point clouds of the same dynamic object in the two previous and next laser frames into the corresponding image data, and search for the nearest matching pair of image feature points in the image data around the projected dynamic point clouds;

[0209] An association relationship determination module, used to establish an association relationship between the dynamic point clouds of the same dynamic object in two preceding and following laser frames according to the searched image feature point matching pairs;

[0210] An azimuth transformation relationship determination module is used to construct an azimuth transformation relationship of the dynamic point cloud of the same dynamic object in the two preceding and succeeding laser frames according to the association relationship and the laser sampling interval;

[0211] A motion information determination module, used to determine the motion information of the same dynamic object according to the orientation transformation relationship;

[0212] A first motion compensation module is used to perform motion compensation on the dynamic point cloud corresponding to each dynamic object according to the timestamp of the laser frame and the motion information of each dynamic object, so as to obtain a corresponding target laser frame;

[0213] A prediction module, used for predicting a predicted laser frame of the target laser frame at the next moment according to the motion model, the azimuth transformation relationship and the target laser frame;

[0214] A second motion compensation module is used to perform motion compensation on the dynamic object in the real laser frame at the next moment of the target laser frame by using the laser sampling frequency and the motion information of the dynamic point cloud in the predicted laser frame, so as to obtain a compensated laser frame corresponding to the real laser frame;

[0215] The second projection module is used to obtain a color point cloud of the compensation laser frame and record the number of times each point in the compensation laser frame is projected by projecting the corresponding dynamic image onto the compensation laser frame.

[0216] Optionally, the system 400 further includes:

[0217] A filtering processing module, used for filtering invalid points in the dynamic point cloud of the same dynamic object according to the orientation transformation relationship;

[0218] The motion information determination module is used to determine the motion information of the same dynamic object according to the remaining orientation transformation relationship of the dynamic point cloud of the same dynamic object.

[0219] Optionally, the system 400 further includes:

[0220] A first downsampling module is used to downsample the color point cloud based on the recorded number of times each point in the compensated laser frame is projected, so as to obtain a merged point of the point cloud and the number of projections of the merged point;

[0221] A first initialization module is used to initialize the variance of the Gaussian distribution by the number of projections of the merged point and to initialize the normal vector of the Gaussian distribution by the normal line of the merged point, so as to obtain the initialization result of the area where each merged point is located;

[0222] The first rendering and reconstruction module is used to update the Gaussian parameters of the area where each merging point is located by projecting the initialization result onto the dynamic image, so as to obtain the color point cloud of the compensated laser frame reconstructed by rendering.

[0223] Optionally, the system 400 further includes:

[0224] The rendering output module is used to extract the point cloud data of the corresponding range from the rendered and reconstructed static color point cloud map according to the current laser posture, and place the color point cloud of the compensated laser frame obtained by the rendering and reconstruction corresponding to the current laser posture into the extracted point cloud data to form laser perception output.

[0225] Optionally, the system 400 further includes:

[0226] A transmission time error determination module, used to determine the transmission time error of the collected original image data according to the optimized camera posture and vehicle speed;

[0227] The time compensation module is used to perform time compensation on the timestamp of data collection according to the transmission time error.

[0228] In this embodiment, if Figure 5 As shown, a system for optimizing and reconstructing scene data provided by the present application includes three parts from the functional level, namely, an acquisition platform, a business platform and a computing platform. The acquisition platform is used to acquire scene data, while the computing platform is used to optimize, reconstruct and store the acquired scene data. The business platform is used to configure the needs, and based on the demand configuration, the optimized and reconstructed scene user data is called and rendered in the computing platform to form a data packet and returned to the data user of the business platform for use.

[0229] Based on the same inventive concept, an embodiment of the present application provides an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and running on the processor. When the computer program is executed by the processor, it implements the steps in a method for optimizing and reconstructing scene data as described in the first aspect of the present application.

[0230] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in a method for optimizing and reconstructing scene data as described in the first aspect of the present application are implemented.

[0231] As for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0232] It should be noted that, for the method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the described order of actions, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.

[0233] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0234] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the embodiments of the present application may adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the embodiments of the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0235] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0236] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0237] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0238] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0239] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0240] The above is a detailed introduction to the method, system and product for optimizing and reconstructing scene data provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for optimizing and reconstructing scene data, It is characterized in that The method comprises: Performing semantic segmentation on the image data, and extracting static pixels in the image data as a static image; Projecting the point cloud data of the target laser frame corresponding to the image data onto the image plane of the image data; Searching for feature points in the nearest image feature point matching pair around the target point of the point cloud data; Optimize the camera posture according to the distance between the target point and the corresponding feature point; Based on the optimized position of the camera relative to the laser, the static image is projected into the point cloud data of the target laser frame to obtain the color point cloud data of the target laser frame and record the number of times each point in the point cloud data is projected; Constructing a static color point cloud map through the color point cloud data; Based on the recorded number of times each point in the point cloud data is projected, downsampling the static color point cloud map to obtain merged points of the point cloud and the number of projections of the merged points; The variance of the Gaussian distribution is initialized by the number of projections of the merged point and the normal vector of the Gaussian distribution is initialized by the normal line of the merged point to obtain the initialization result of the area where each merged point is located; The initialization result is projected onto the static image, and the Gaussian parameters of the area where each merging point is located are updated to obtain a rendered and reconstructed static color point cloud map.

2. A method for optimizing and reconstructing scene data according to claim 1, It is characterized in that According to the distance between the target point and the corresponding feature point, the camera posture is optimized, including: Determine the depth value deviation and position distance between the feature point and the target point in the camera system; Determining whether to retain the target point according to the depth value deviation and the position distance; The camera pose is optimized based on the distance between the retained target points and the corresponding feature points.

3. The method for optimizing and reconstructing scene data according to claim 1, It is characterized in that Determining the image data includes: Extracting frames from the collected original image data to obtain first image data; White balance synchronization processing is performed on the first image data to obtain final image data.

4. The method for optimizing and reconstructing scene data according to claim 1, It is characterized in that Determine the depth value of the feature point in the image feature point matching pair, including: Perform feature point matching on the image data of the previous and next two frames to obtain corresponding image feature point matching pairs; Performing depth estimation on the image data of the two frames before and after to obtain a depth map of each image data; Calculating the average depth value of the image blocks corresponding to the two feature points in the image feature point matching pair in the respective image data to obtain the corresponding depth value calculation result; The depth value calculation result is determined as the depth value of the feature point in the image feature point matching pair.

5. The method for optimizing and reconstructing scene data according to claim 1, It is characterized in that Using the color point cloud data, a static color point cloud map is constructed, including: Extracting a first target point from the color point cloud data of one of the two laser frames, and extracting a color plane from the color point cloud data of the other of the two laser frames; constructing a residual equation based on the distance from the first target point to the color plane; Optimizing the relative position of the two laser frames based on the residual equation; The point cloud coordinates are transformed based on the optimized relative pose to construct a static color point cloud map.

6. The method for optimizing and reconstructing scene data according to claim 1, It is characterized in that The method further comprises: When a static color point cloud map has been established, matching the color point cloud data of the laser frame with the static color point cloud map to optimize the laser pose of the laser frame; The static color point cloud map is updated based on the color point cloud data corresponding to the optimized laser posture.

7. The method for optimizing and reconstructing scene data according to claim 1, It is characterized in that The method further comprises: Performing semantic segmentation on the image data, and extracting dynamic pixels in the image data as a dynamic graph; Based on the optimized camera pose, the point cloud data of the laser frame is projected into the corresponding dynamic graph to obtain a dynamic point cloud; Clustering the dynamic point cloud into multiple dynamic objects according to image segmentation categories and regions; Based on the characteristics of the dynamic objects in the dynamic images, the dynamic objects in the previous and next two frames of dynamic images are associated to construct a motion model of each dynamic object; Projecting the dynamic point clouds of the same dynamic object in the two previous and next laser frames into the corresponding image data, and searching for the nearest matching pair of image feature points in the image data around the projected dynamic point clouds; According to the searched image feature point matching pairs, establishing the association relationship between the dynamic point clouds of the same dynamic object in the two previous and next laser frames; According to the association relationship and the laser sampling interval, constructing the orientation transformation relationship of the dynamic point cloud of the same dynamic object in the two preceding and following laser frames; Determining motion information of the same dynamic object according to the orientation transformation relationship; According to the timestamp of the laser frame and the motion information of each dynamic object, motion compensation is performed on the dynamic point cloud corresponding to each dynamic object to obtain the corresponding target laser frame; Predicting a predicted laser frame of the target laser frame at the next moment according to the motion model, the azimuth transformation relationship and the target laser frame; Motion compensation is performed on the dynamic object in the real laser frame at the next moment of the target laser frame according to the laser sampling frequency and the motion information of the dynamic point cloud in the predicted laser frame, so as to obtain a compensated laser frame corresponding to the real laser frame; By projecting the corresponding dynamic image onto the compensation laser frame, a color point cloud of the compensation laser frame is obtained and the number of times each point in the compensation laser frame is projected is recorded.

8. The method for optimizing and reconstructing scene data according to claim 7, It is characterized in that Before determining the motion information of the same dynamic object according to the orientation transformation relationship, the method further includes: According to the orientation transformation relationship, filtering invalid points in the dynamic point cloud of the same dynamic object; Determining the motion information of the same dynamic object according to the orientation transformation relationship includes: The motion information of the same dynamic object is determined according to the remaining orientation transformation relationship of the dynamic point cloud of the same dynamic object.

9. The method for optimizing and reconstructing scene data according to claim 7, It is characterized in that The method further comprises: Based on the recorded number of times each point in the compensated laser frame is projected, downsampling the color point cloud to obtain a merged point of the point cloud and the number of projections of the merged point; The variance of the Gaussian distribution is initialized by the number of projections of the merged point and the normal vector of the Gaussian distribution is initialized by the normal line of the merged point to obtain the initialization result of the area where each merged point is located; The initialization result is projected onto the dynamic graph, and the Gaussian parameters of the regions where the merged points are located are updated to obtain a color point cloud of the compensated laser frame reconstructed by rendering.

10. A method for optimizing and reconstructing scene data according to claim 9, It is characterized in that The method further comprises: According to the current laser posture, point cloud data of the corresponding range is extracted from the rendered and reconstructed static color point cloud map, and the color point cloud of the compensated laser frame obtained by rendering and reconstruction corresponding to the current laser posture is placed in the extracted point cloud data to form laser perception output.

11. The method for optimizing and reconstructing scene data according to claim 1, It is characterized in that The method further comprises: Determine the transmission time error of the collected raw image data according to the optimized camera posture and vehicle speed; According to the transmission time error, time compensation is performed on the timestamp of data acquisition.

12. A system for optimizing and reconstructing scene data, It is characterized in that The system comprises: A semantic segmentation module, used to perform semantic segmentation on the image data and extract static pixels in the image data as a static image; A projection module, used for projecting the point cloud data of the target laser frame corresponding to the image data onto the image plane of the image data; A search module, used for searching for a feature point in a nearest image feature point matching pair around a target point in the point cloud data; The optimization module is used to optimize the camera posture according to the distance between the target point and the corresponding feature point; A color point cloud determination module is used to project the static image into the point cloud data of the target laser frame based on the optimized position of the camera relative to the laser, obtain the color point cloud data of the target laser frame and record the number of times each point in the point cloud data is projected; A point cloud map construction module, used to construct a static color point cloud map through the color point cloud data; A downsampling module, used to downsample the static color point cloud map based on the recorded number of times each point in the point cloud data is projected, to obtain the merged points of the point cloud and the number of projections of the merged points; An initialization module, used to initialize the variance of the Gaussian distribution by the number of projections of the merged point and to initialize the normal vector of the Gaussian distribution by the normal line of the merged point, so as to obtain the initialization result of the area where each merged point is located; The rendering and reconstruction module is used to update the Gaussian parameters of the area where each merging point is located by projecting the initialization result onto the static image, so as to obtain a static color point cloud map reconstructed by rendering.

13. An electronic device, It is characterized in that include: A processor, a memory, and a computer program stored in the memory and running on the processor, wherein the computer program, when executed by the processor, implements the steps in a method for optimizing and reconstructing scene data as described in any one of claims 1 to 11.

14. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps in a method for optimizing and reconstructing scene data as described in any one of claims 1 to 11.

Images