Data fusion method and device and computer equipment

Abstract

The invention is suitable for the technical field of intelligent transportation, and provides a data fusion method and device and computer equipment, and the method comprises the steps: obtaining initial color point cloud data; wherein the initial color point cloud data is obtained based on fusion of point cloud data and image data; performing semantic segmentation on the initial color point cloud data to obtain a color point cloud data segmentation result; performing semantic segmentation on the image data to obtain an image data segmentation result; and correcting the initial color point cloud data based on the color point cloud data segmentation result and the image data segmentation result to obtain target color point cloud data. Therefore, the completeness and accuracy of the fused data are ensured.

Description

Technical Field

[0001] This application belongs to the field of intelligent transportation technology, and particularly relates to a data fusion method, device, and computer device. Background Art

[0002] A colored point cloud is composed of point cloud data in three-dimensional space and corresponding color information (usually from an RGB image). Colored point clouds have important application prospects in fields such as environmental perception, road sign recognition, building modeling, and urban planning. For example, in environmental perception, colored point clouds can help vehicles more accurately identify targets such as traffic signs, pedestrians, and vehicles.

[0003] However, since the RGB image and point cloud data are collected by different sensors respectively, there are problems such as inconsistent coordinate systems, asynchronous data collection, and different data accuracies, which lead to phenomena such as information loss and inaccurate color assignment during the data fusion process. Summary of the Invention

[0004] Embodiments of this application provide a data fusion method, device, and computer device, which can solve the technical problems in the prior art that due to the RGB image and point cloud data being collected by different sensors respectively, there are problems such as inconsistent coordinate systems, asynchronous data collection, and different data accuracies, resulting in information loss and inaccurate color assignment during the data fusion process.

[0005] In a first aspect, embodiments of this application provide a data fusion method, including: Obtain initial colored point cloud data; wherein, the initial colored point cloud data is obtained by fusing point cloud data and image data; Perform semantic segmentation on the initial colored point cloud data to obtain a segmentation result of the colored point cloud data; Perform semantic segmentation on the image data to obtain a segmentation result of the image data; Correct the initial colored point cloud data based on the segmentation result of the colored point cloud data and the segmentation result of the image data to obtain target colored point cloud data.

[0006] In a possible implementation manner of the first aspect, the obtaining of the initial colored point cloud data includes: Obtain the point cloud data and the image data; Perform a fusion process on the point cloud data and the image data to generate the initial colored point cloud data; wherein, the fusion process includes data fusion and / or feature fusion.

[0007] In a possible implementation manner of the first aspect, the performing of a fusion process on the point cloud data and the image data to generate the initial colored point cloud data includes: Through sensor calibration, align the coordinate systems of the point cloud data and the image data; Map the pixel values in the image data to each point cloud point to generate the initial colored point cloud data; Or, Extract features from the point cloud data to obtain point cloud features; Extract features from the image data to obtain image features; Fuse the point cloud features and the image features to generate a bird's-eye view; Map the pixel values in the image features to the point cloud points in the bird's-eye view to generate the initial colored point cloud data.

[0008] In a possible implementation manner of the first aspect, correcting the initial colored point cloud data based on the colored point cloud data segmentation result and the image data segmentation result to obtain the target colored point cloud data includes: Based on the colored point cloud data segmentation result and the image data segmentation result, determine the semantic missing data; Fill the semantic missing data into the initial colored point cloud data to obtain the target colored point cloud data.

[0009] In a possible implementation manner of the first aspect, the method further includes: Based on the colored point cloud data segmentation result and the image data segmentation result, determine the color deviation data; Replace the color deviation data into the initial colored point cloud data to obtain the target colored point cloud data.

[0010] In a possible implementation manner of the first aspect, the method further includes: Fuse the target colored point cloud data, the colored point cloud data segmentation result, the image data for semantic segmentation, and the geographic information data to generate a map.

[0011] In a second aspect, an embodiment of the present application provides a data fusion device, including: An acquisition module, configured to acquire initial colored point cloud data; wherein, the initial colored point cloud data is obtained by fusing point cloud data and image data; A first semantic segmentation module, configured to perform semantic segmentation on the initial colored point cloud data to obtain a colored point cloud data segmentation result; A second semantic segmentation module, configured to perform semantic segmentation on the image data to obtain an image data segmentation result; A calibration module, configured to calibrate the initial color point cloud data based on the color point cloud data segmentation result and the image data segmentation result, so as to obtain target color point cloud data.

[0012] In a third aspect, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the data fusion method according to any one of the above first aspects is implemented.

[0013] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the data fusion method according to any one of the above first aspects is implemented.

[0014] In a fifth aspect, an embodiment of the present application provides a computer program product, which when running on a computer device causes the computer device to execute the data fusion method according to any one of the above first aspects.

[0015] The beneficial effects of the embodiments of the present application compared with the prior art are as follows: obtaining initial color point cloud data, where the initial color point cloud data is obtained by fusing point cloud data and image data; performing semantic segmentation on the initial color point cloud data to obtain a color point cloud data segmentation result; performing semantic segmentation on the image data to obtain an image data segmentation result; calibrating the initial color point cloud data based on the color point cloud data segmentation result and the image data segmentation result to obtain target color point cloud data. Among them, by obtaining the initial color point cloud that fuses point cloud and image data, performing semantic segmentation on it and the original image data respectively, and then calibrating the initial color point cloud data according to the segmentation results, the finally obtained target color point cloud data is more accurate in color, and the integrity, consistency, and reliability of the data are greatly improved.

[0016] It can be understood that the beneficial effects of the above second to fifth aspects can refer to the relevant descriptions in the above first aspect, and will not be elaborated here. Description of the Drawings

[0017] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for use in the embodiments or the description of the prior art. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0018] Figure 1 It is a schematic flowchart of the data fusion method provided by an embodiment of the present application; Figure 2 is a schematic structural diagram of the data fusion device provided by an embodiment of the present application; Figure 3 is a schematic structural diagram of the computer device provided by an embodiment of the present application. Detailed implementation manners

[0019] In the following description, for the purpose of illustration rather than limitation, specific details such as specific system architectures, technologies, etc. are presented to thoroughly understand the embodiments of the present application. However, those skilled in the art should clearly understand that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from interfering with the description of the present application.

[0020] It should be understood that when used in the specification and claims of the present application, the term "comprising" indicates the presence of the described features, wholes, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0021] It should also be understood that the term "and / or" as used in the specification and claims of the present application refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0022] As used in the specification and claims of the present application, the term "if" can be interpreted as "when", "once", "in response to determining", or "in response to detecting" depending on the context. Similarly, the phrase "if determined" or "if detected [the described condition or event]" can be interpreted as meaning "once determined", "in response to determining", "once detected [the described condition or event]", or "in response to detecting [the described condition or event]" depending on the context.

[0023] In addition, in the description of the specification and claims of the present application, the terms "first", "second", "third", etc. are only used for distinguishing descriptions and cannot be understood as indicating or implying relative importance.

[0024] References to "one embodiment" or "some embodiments" in the description of this application mean that specific features, structures, or characteristics described in connection with that embodiment are included in one or more embodiments of this application. Thus, statements such as "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments", etc. that appear in different places in this specification do not necessarily all refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized. The terms "comprising", "including", "having" and their variants all mean "including but not limited to", unless otherwise specifically emphasized.

[0025] In the field of autonomous driving technology, as a key infrastructure for vehicle environmental perception, high-precision maps need to provide geospatial data containing precise three-dimensional coordinates and semantic information. Among them, point cloud data, as an important carrier of three-dimensional geographic information, can provide rich spatial structure information and has become the core data source for high-precision map construction.

[0026] However, point cloud data only contains geometric information and lacks color information, which limits its use effect in some application scenarios. For example, in tasks such as environmental modeling, object recognition, and scene understanding, color information, as important auxiliary information, helps to distinguish different types of objects and surface features.

[0027] RGB images, as a common type of two-dimensional image data, can provide rich color information. By fusing RGB images with point cloud data and assigning RGB values to the point cloud, colored point clouds can be generated, thereby increasing color information while retaining the three-dimensional spatial information of the point cloud, and improving the expression ability and application value of high-precision maps.

[0028] Colored point clouds have important application prospects in fields such as environmental perception, road sign recognition, building modeling, and urban planning. For example, in environmental perception, colored point clouds can help vehicles more accurately identify targets such as traffic signs, pedestrians, and vehicles.

[0029] However, since RGB images and point cloud data are collected by different sensors respectively, there are problems such as inconsistent coordinate systems, asynchronous data collection, and differences in data accuracy, which lead to phenomena such as information loss and inaccurate color assignment during the data fusion process. In addition, the resolution of RGB images and the density of point cloud data may also not match, which will also cause error accumulation and information loss during the data fusion process.

[0030] Therefore, how to effectively assign RGB values accurately to point cloud data to generate high-quality colored point clouds is a difficult point in current high-precision map generation.

[0031] To solve the above problems,Figure 1 The figure shows a schematic flowchart of a data fusion method provided by this application.

[0032] S101. Obtain initial color point cloud data; wherein, the initial color point cloud data is obtained by fusing point cloud data and image data.

[0033] Wherein, the point cloud data is a set composed of a group of points distributed in three-dimensional space, and each point contains its coordinates in space.

[0034] In the embodiments of this application, point cloud data can be collected by lidar, depth cameras, etc. Among them, lidar is one of the main sources of point cloud data. It obtains the three-dimensional coordinates of an object by emitting laser pulses and measuring the reflection time or phase difference. Lidar can quickly generate high-precision point cloud data and is widely used in autonomous driving, terrain mapping, three-dimensional modeling, etc. In autonomous driving, point cloud data is used for environmental perception, obstacle detection, and high-precision map construction. The point cloud generated by lidar can provide real-time three-dimensional information of the vehicle's surrounding environment and help the vehicle make decisions.

[0035] Wherein, the image data is two-dimensional data captured by a camera, containing rich color information (usually represented by RGB values), and can provide visual details and semantic information for a high-precision map. The image data can be obtained by various camera devices, including ordinary RGB cameras, panoramic cameras, etc. These image data are used in high-precision map construction to enhance environmental perception, assist in the semantic understanding of point cloud data, and improve the visualization effect of the map.

[0036] Wherein, the color point cloud data is a data form that combines point cloud data in three-dimensional space with color information. This data not only contains the position of each point in space but also contains the color information associated with each point.

[0037] Wherein, the above-mentioned initial color point cloud data refers to the preliminary result obtained in the process of fusing point cloud data and image data, which contains spatial position information and color information. There may be some problems with the initial color point cloud data, such as inaccurate color assignment, information loss, or data inconsistency, etc. To improve the data quality, it is usually necessary to further process the initial color point cloud data. For the specific processing process, refer to the following text.

[0038] Wherein, there are various ways to obtain the initial color point cloud data. It is possible to directly obtain the initially fused initial color point cloud data; it is also possible to separately obtain the point cloud data and the image data and then perform the fusion process.

[0039] S102. Perform semantic segmentation on the initial color point cloud data to obtain the segmentation result of the color point cloud data.

[0040] Among them, semantic segmentation is a computer vision task that aims to classify each pixel in an image into one of the predefined categories. In the embodiments of the present application, it refers to a process of classifying each point in the point cloud according to the object category to which it belongs. This classification not only involves spatial position but also color information, deeply mining the spatial and semantic features in the point cloud data, accurately classifying each point in the point cloud according to the object category to which it belongs, and assigning a semantic label to each point, that is, understanding what type of object each point represents in the real world, such as traffic participants, obstacles, trees, etc.

[0041] Among them, the segmentation result of the colored point cloud data refers to the output obtained after applying semantic segmentation technology to the initial colored point cloud data. This result classifies and labels the points in the colored point cloud according to the object categories or scene elements to which they belong. Among them, the segmentation result of the colored point cloud data includes but is not limited to semantic labels (also called category labels) such as traffic participants, obstacles, trees, etc.

[0042] Among them, the segmentation result of the colored point cloud data has the following characteristics: Pixel-level classification: Each point cloud point is assigned one or more semantic labels, which indicate the object type to which the point belongs, such as pedestrians, vehicles, buildings, roads, vegetation, etc.

[0043] Rich semantic information: The segmentation result not only contains the spatial position and color information of the points but also contains semantic labels, making the point cloud data richer and more useful.

[0044] Among them, deep learning-based point cloud segmentation algorithms (such as PointNet, PointCNN, etc.) can be used to perform semantic segmentation on the colored point cloud data.

[0045] S103. Perform semantic segmentation on the image data to obtain an image data segmentation result.

[0046] Among them, each pixel in the image data is classified, and a semantic label is assigned to each pixel, such as traffic participants, roads, lane lines, road surface markers, etc. In this way, the semantic information of the image data is fully mined, providing strong support for subsequent fusion and calibration with the point cloud data.

[0047] Among them, deep learning-based image segmentation algorithms (such as U-Net, Mask R-CNN, etc.) can be used to perform semantic segmentation on the image data.

[0048] S104. Calibrate the initial colored point cloud data based on the segmentation result of the colored point cloud data and the segmentation result of the image data to obtain the target colored point cloud data.

[0049] In the embodiments of the present application, the inconsistency between the colored point cloud data segmentation result and the image data segmentation result can be identified and resolved by comparing the semantic labels therein. Based on the comparison result, the semantic labels in the colored point cloud data are adjusted to match the semantic labels in the image data. After the above correction steps, the final target colored point cloud data is generated. The generated target colored point cloud data not only contains accurate three-dimensional structure information, but also contains rich and accurate color information.

[0050] In the embodiments of the present application, by obtaining the initial colored point cloud that fuses the point cloud and image data, performing semantic segmentation on it and the original image data respectively, and then correcting the initial colored point cloud data according to the segmentation results, the finally obtained target colored point cloud data is more accurate in color, and the integrity, consistency and reliability of the data are greatly improved.

[0051] In an alternative embodiment, S101 obtaining the initial colored point cloud data includes: Obtaining the point cloud data and the image data; performing a fusion process on the point cloud data and the image data to generate the initial colored point cloud data; wherein the fusion process includes data fusion, and / or, feature fusion.

[0052] For the relevant descriptions of the point cloud data and the image data, refer to the relevant descriptions in the previous embodiment, which will not be elaborated here.

[0053] Optionally, after obtaining the point cloud data and the image data, preprocessing of the point cloud data and the image data is also required. The preprocessing of the point cloud data includes but is not limited to: operations such as denoising, filtering, and duplicate removal. These operations help to eliminate the noise and redundant information in the point cloud data, thereby providing clearer data for subsequent analysis and processing. The preprocessing of the image data includes but is not limited to: operations such as de-distortion, color correction, and contrast enhancement. These operations help to eliminate the influence caused by camera lens distortion or changes in lighting conditions, etc., making the image data more accurate and reliable.

[0054] Among them, after obtaining the preprocessed point cloud data and image data, a fusion process is performed on the preprocessed point cloud data and image data to generate the initial colored point cloud data.

[0055] Optionally, before data acquisition, the fixedly installed lidar and camera are calibrated to ensure the accuracy and consistency of the sensor data. Specifically: The Zhang-Zhengyou calibration method or a calibration method based on deep learning can be used to obtain the internal parameters of the camera (such as focal length, principal point coordinates, etc.). Alternatively, the external parameters (relative position and attitude) between the lidar and the camera can be calibrated through the manual selection of control points calibration method or the checkerboard calibration method, establishing the spatial correspondence between the point cloud data and the image data to ensure that the point cloud can be accurately projected onto the image plane.

[0056] Among them, the vehicle equipped with sensors travels in the scene at a low speed (such as within 40 km / h), collecting sensor data such as lidar and camera. The time stamp of the lidar can be used as a reference to find the data frames of other sensors within half a sampling period before and after. The frame with the closest time is selected for synchronization to avoid excessive time intervals between different sensor data, which may lead to difficult alignment.

[0057] In the embodiments of the present application, data-level fusion methods and feature-level fusion methods can be adopted.

[0058] Among them, the data-level fusion method includes: through sensor calibration, aligning the coordinate systems of the point cloud data and the image data; mapping the pixel values in the image data to each point cloud point to generate the initial colored point cloud data.

[0059] Specifically, through sensor calibration, the point cloud data and the image data are aligned to the same coordinate system. Then, the color information (RGB values) in the image data is mapped to each point of the point cloud. For example, according to the projection position of the point cloud point in the image, the corresponding RGB value is directly assigned to it. Each point cloud point not only contains the spatial position but also has the color information attached, forming a colored point cloud.

[0060] Among them, the feature-level fusion method includes: extracting features from the point cloud data to obtain point cloud features; extracting features from the image data to obtain image features; fusing the point cloud features and the image features to generate a bird's-eye view; mapping the pixel values in the image features to the point cloud points in the bird's-eye view to generate the initial colored point cloud data.

[0061] Specifically, a point cloud detection model based on points, such as PointNet++, and a point cloud detection model based on voxels, such as VoxelNet, are used to process point cloud data and extract point cloud features such as spatial information. A convolutional neural network (CNN) or the like is used to extract image features from image data. The point cloud features and the image features are fused to generate a bird's eye view (BEV). Then, the pixel values in the image features are mapped into the generated bird's eye view, and finally a bird's eye view with color information is obtained. Through the above color assignment, each point cloud point not only retains its spatial position information but also obtains color information. It can more intuitively represent the geometric and visual features of the environment or object. This fusion method can fully combine the spatial structure advantages of point cloud data and the color and semantic advantages of image data, providing a better data basis for subsequent processing.

[0062] In an optional embodiment, S104 corrects the initial colored point cloud data based on the colored point cloud data segmentation result and the image data segmentation result to obtain target colored point cloud data, including: Based on the colored point cloud data segmentation result and the image data segmentation result, determine the semantically missing data; fill the semantically missing data into the initial colored point cloud data to obtain the target colored point cloud data.

[0063] Based on the colored point cloud data segmentation result and the image data segmentation result, determine the color deviation data; replace the color deviation data in the initial colored point cloud data to obtain the target colored point cloud data.

[0064] In the embodiments of the present application, during the colored point cloud data segmentation process, there may be some cases where point cloud data is not correctly classified or labeled. These unlabeled or mislabeled data are called semantically missing data. By comparing the colored point cloud data segmentation result and the image data segmentation result, these data points lacking semantic information are identified. Then, according to the surrounding data points with known semantic labels or the corresponding regions in the image segmentation result, appropriate semantic labels are assigned to these missing data points. This helps to improve the integrity and accuracy of the colored point cloud data.

[0065] In addition, in the initial colored point cloud data, due to sensor noise, changes in lighting conditions, or errors in the data fusion process, the color information of some data points may deviate from the actual situation. By comparing the color information in the image data segmentation result, these data points with color deviations are identified. Then, more accurate color information is obtained from the image data and used to replace or correct the deviated colors in the initial colored point cloud data. This helps to improve the color accuracy and consistency of the colored point cloud data.

[0066] In the embodiments of the present application, on the one hand, semantic missing data is determined, that is, the data points that are not correctly classified or labeled during the segmentation process of the colored point cloud data. By referring to the data points with known semantic labels around or the corresponding regions in the image segmentation results, appropriate semantic labels are assigned to these missing data points to fill the semantic gaps and enhance the integrity and accuracy of the colored point cloud data. On the other hand, color deviation data is determined. Due to sensor noise, changes in lighting conditions, or errors in the data fusion process, the color information of some data points in the initial colored point cloud data may not match the actual situation. By comparing the color information in the image data segmentation results, these color-deviated data points are accurately identified, and then more accurate color information is obtained from the image data to replace or correct the deviated colors in the initial colored point cloud data, improving the color accuracy and consistency of the colored point cloud data.

[0067] In the embodiments of the present application, through the above two operations, the target colored point cloud data has been significantly improved in terms of semantic information and color information. The generated target colored point cloud data not only retains the precise three-dimensional structural information but also has rich and accurate color information, and the data quality has been significantly improved. Thereby providing higher-quality and more reliable data support for subsequent applications (such as autonomous driving, robot navigation, 3D reconstruction, etc.).

[0068] In an alternative embodiment, in the actual scenario, the occlusion phenomenon between objects is widespread, which may lead to partial loss of point cloud data and image data, affecting the accuracy of semantic segmentation and correction results. The present application can combine multi-view data acquisition technology, use multiple sensors to collect data from different angles, and reduce the impact of occlusion through data complementarity. At the same time, based on the occlusion reasoning algorithm, according to the surrounding visible data and the prior knowledge of objects, the data characteristics and semantic information of the occluded part are inferred to improve the integrity and accuracy of the data.

[0069] For example, the occlusion reasoning algorithm based on the geometric model: uses the geometric shape and spatial position relationship of the object to infer the data of the occluded part. Taking the autonomous driving scenario as an example, when part of the vehicle is occluded by other objects, the algorithm first identifies the geometric shape of the unoccluded part of the vehicle according to the known point cloud data and image data. For example, by fitting the three-dimensional point cloud data, the approximate contour model of the vehicle can be obtained, which may be a cuboid or a combination of more complex polygons. Then, based on the position and attitude information of the vehicle in space and the position relationship of the surrounding environmental objects, such as the positions of the adjacent vehicles and buildings, according to geometric principles, such as the geometric rules of light propagation and object occlusion, the possible shape and position of the occluded part are inferred.

[0070] For example, an occlusion inference algorithm based on deep learning: leveraging the powerful feature learning ability of deep learning to handle occlusion problems. Taking the convolutional neural network (CNN) as an example, a large amount of point cloud data and image data containing occlusion situations are used for training. During the training process, the network learns to extract rich features from the visible data, including various features such as the texture, shape, and spatial position of objects. When encountering a new occlusion scenario, based on the learned features, predictions are made for the data of the occluded part. By analyzing the features of the surrounding visible pixels, an attempt is made to restore the pixel values and semantic information of the occluded area.

[0071] In an alternative embodiment, the method further includes: fusing the target color point cloud data, the color point cloud data segmentation result, the image data after semantic segmentation, and the geographic information data to generate a map.

[0072] In the embodiments of the present application, the target color point cloud data, with its precise three-dimensional spatial coordinates and rich and accurate color information, provides an intuitive and detailed basic spatial framework and visual feature expression for map construction. The color point cloud data segmentation result clarifies the object category corresponding to each point cloud, enabling the map to have semantic-level understanding. For example, different types of objects such as roads, buildings, and vegetation can be directly distinguished on the map. After semantic segmentation of the image data, its rich visual details and semantic labels further improve the description of the environment. For example, the accurate recognition of road signs, traffic lights, etc. in the image can be more accurately marked on the map, adding more practical information to the map. Geographic information data (such as GPS coordinates, map boundaries, road networks, etc.) provides an accurate spatial reference and geographical layout for the high-precision map, enabling the high-precision map to accurately reflect the geographical locations and features in the real world. When fusing the target color point cloud data, the color point cloud data segmentation result, the semantic segmentation result of the image data, and the geographic information data to generate a map, each type of data plays its own advantages. During the fusion process, after unifying the coordinate systems and merging and optimizing the same category elements, the generated map not only has a high-precision three-dimensional terrain display but also includes detailed ground object semantic annotations and rich visual appearance information.

[0073] Among them, during the fusion process, it is necessary to unify the coordinate systems of various types of data first to ensure the precise matching of data from different data sources. Then, the spatial positions in the target color point cloud data are corresponded to the geodetic coordinates of the geographic information data. At the same time, according to the semantic labels in the color point cloud data segmentation result and the image data semantic segmentation result, the elements of the same category are merged and optimized. For example, the roads identified in the point cloud data are integrated with the road areas segmented in the image data to supplement detailed information. The finally generated map not only has a high-precision three-dimensional terrain display but also includes detailed ground object semantic annotations and rich visual appearance information, and can be widely applied to multiple fields such as autonomous driving and urban planning, providing comprehensive and accurate data support for relevant decision-making and applications.

[0074] It should be understood that the magnitudes of the sequence numbers of the steps in the above embodiments do not mean the order of execution. The execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0075] Corresponding to the data fusion method described in the above embodiments, Figure 2 The structural block diagram of the data fusion device provided by the embodiments of the present application is shown. For the sake of convenience of description, only the parts related to the embodiments of the present application are shown.

[0076] Referring to Figure 2 , the data fusion device includes: An acquisition module, configured to acquire initial color point cloud data; wherein, the initial color point cloud data is obtained by fusing point cloud data and image data; A first semantic segmentation module, configured to perform semantic segmentation on the initial color point cloud data to obtain a color point cloud data segmentation result; A second semantic segmentation module, configured to perform semantic segmentation on the image data to obtain an image data segmentation result; A calibration module, configured to calibrate the initial color point cloud data based on the color point cloud data segmentation result and the image data segmentation result to obtain target color point cloud data.

[0077] In a possible implementation manner, the acquisition module is configured to: Acquire the point cloud data and the image data; Perform a fusion process on the point cloud data and the image data to generate the initial color point cloud data; wherein, the fusion process includes data fusion and / or feature fusion.

[0078] In a possible implementation manner, the acquisition module is configured to: Align the coordinate systems of the point cloud data and the image data through sensor calibration; Map the pixel values in the image data to each point cloud point to generate the initial colored point cloud data; Or, Extract features from the point cloud data to obtain point cloud features; Extract features from the image data to obtain image features; Fuse the point cloud features and the image features to generate a bird's-eye view; Map the pixel values in the image features to the point cloud points in the bird's-eye view to generate the initial colored point cloud data.

[0079] In a possible implementation manner, a correction module is configured to: Determine semantic missing data based on the colored point cloud data segmentation result and the image data segmentation result; Fill the semantic missing data into the initial colored point cloud data to obtain the target colored point cloud data.

[0080] In a possible implementation manner, a correction module is configured to: Determine color deviation data based on the colored point cloud data segmentation result and the image data segmentation result; Replace the color deviation data into the initial colored point cloud data to obtain the target colored point cloud data.

[0081] In a possible implementation manner, the data fusion device further includes a map generation module configured to: Fuse the target colored point cloud data, the colored point cloud data segmentation result, the image data for semantic segmentation, and the geographic information data to generate a map.

[0082] It should be noted that for the information interaction, execution process, etc. between the above modules, since they are based on the same concept as the method embodiment of the present application, their specific functions and the technical effects brought are specifically described in the method embodiment part, and will not be elaborated here.

[0083] Those skilled in the art can clearly understand that, for the convenience and conciseness of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of the functional units and modules are only for the convenience of mutual distinction and do not limit the protection scope of this application. The specific working process of the units and modules in the above system can refer to the corresponding process in the foregoing method embodiment and will not be elaborated here.

[0084] An embodiment of this application also provides a computer device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor. When the processor executes the computer program, the steps in any of the foregoing method embodiments are implemented.

[0085] An embodiment of this application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the foregoing method embodiments can be implemented.

[0086] An embodiment of this application provides a computer program product. When the computer program product runs on a computer device, the computer device is enabled to implement the steps in the foregoing method embodiments.

[0087] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, to implement all or part of the processes in the above method embodiments of this application, a computer program can be used to instruct relevant hardware to complete. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form, etc. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the photographing device / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal, and software distribution medium. For example, USB flash drive, mobile hard disk, magnetic disk or optical disc, etc. In some jurisdictions, according to legislation and patent practice, the computer-readable medium cannot be electrical carrier signals and telecommunication signals.

[0088] In the above embodiments, the descriptions of the respective embodiments have their own focuses. For the parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0089] Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0090] In the embodiments provided in this application, it should be understood that the disclosed device / computer device and method can be implemented in other ways. For example, the device / computer device embodiments described above are only illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces. The indirect coupling or communication connection of the device or unit can be in an electrical, mechanical or other form.

[0091] The unit described as a separation component may or may not be physically separated. The component shown as a unit may or may not be a physical unit, that is, it may be located in one place or distributed across multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0092] The above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit them. Although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features. And these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

[0093] Figure 3 The structural schematic diagram of a computer device provided by an embodiment of the present application is as follows Figure 3 As shown, the computer device of this embodiment includes: at least one processor 20 ( Figure 3 only one is shown in the figure), a memory 21, and a computer program 22 stored in the memory 21 and executable on the at least one processor 20. When the processor 20 executes the computer program 22, the steps in any of the above-described data fusion method embodiments are implemented.

[0094] The computer device may include, but is not limited to, a processor 20 and a memory 21. Those skilled in the art can understand that Figure 3 this is only an example of a computer device and does not constitute a limitation on the computer device. It may include more or fewer components than shown in the figure, or combine some components, or different components. For example, it may also include input / output devices, network access devices, etc.

[0095] The so-called processor 20 may be a central processing unit (CPU). The processor 20 may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0096] In some embodiments, the memory 21 may be an internal storage unit of the computer device, such as the hard disk or memory of the computer device. In some other embodiments, the memory 21 may also be an external storage device of the computer device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the computer device. Further, the memory 21 may also include both the internal storage unit and the external storage device of the computer device. The memory 21 is used to store an operating system, application programs, a BootLoader, data, and other programs, such as the program code of the computer program. The memory 21 may also be used to temporarily store data that has been output or is to be output.

[0097] In each embodiment of the present application, the relevant user personal information that may be involved is all processed in strict accordance with the requirements of laws and regulations, following the principles of legality, legitimacy, and necessity, and for reasonable purposes based on business scenarios, for the personal information actively provided by the user during the use of the product / service or generated due to the use of the product / service, as well as the personal information obtained with the authorization of the user.

[0098] The user personal information processed by the applicant may vary depending on the specific product / service scenario, and it is subject to the specific scenario of the user's use of the product / service. It may involve the user's account information, device information, driving information, vehicle information, or other relevant information. The applicant will treat the user's personal information and its processing with a high degree of diligence.

[0099] The applicant attaches great importance to the security of user personal information and has taken security protection measures that meet industry standards and are reasonable and feasible to protect the user's information and prevent the personal information from being accessed, publicly disclosed, used, modified, damaged, or lost without authorization.

Claims

1. A data fusion method, characterized in that: include: Acquire initial color point cloud data; wherein the initial color point cloud data is obtained based on the fusion of point cloud data and image data; Performing semantic segmentation on the initial color point cloud data to obtain a color point cloud data segmentation result; Performing semantic segmentation on the image data to obtain an image data segmentation result; The initial color point cloud data is corrected based on the color point cloud data segmentation result and the image data segmentation result to obtain target color point cloud data.

2. The data fusion method according to claim 1, characterized in that: The obtaining of initial color point cloud data comprises: Acquire the point cloud data and the image data; The point cloud data and the image data are fused to generate the initial color point cloud data; wherein the fusion process includes data fusion and / or feature fusion.

3. The data fusion method according to claim 2, characterized in that: The step of fusing the point cloud data and the image data to generate the initial color point cloud data includes: Aligning the point cloud data and the image data in coordinate systems through sensor calibration; Mapping the pixel value in the image data to each point cloud point to generate the initial color point cloud data; or, Performing feature extraction on the point cloud data to obtain point cloud features; Performing feature extraction on the image data to obtain image features; Fusing the point cloud features with the image features to generate a bird's-eye view; The pixel values ​​in the image features are mapped to the point cloud points in the bird's-eye view to generate the initial color point cloud data.

4. The data fusion method according to any one of claims 1 to 3, characterized in that: The correcting the initial color point cloud data based on the color point cloud data segmentation result and the image data segmentation result to obtain target color point cloud data includes: Determining semantic missing data based on the color point cloud data segmentation result and the image data segmentation result; The semantic missing data is filled into the initial color point cloud data to obtain the target color point cloud data.

5. The data fusion method according to claim 4, characterized in that: The method further comprises: Determining color deviation data based on the color point cloud data segmentation result and the image data segmentation result; The color deviation data is replaced with the initial color point cloud data to obtain the target color point cloud data.

6. The data fusion method according to claim 5, characterized in that: The method further comprises: The target color point cloud data, the color point cloud data segmentation result, the image data are semantically segmented and the geographic information data are integrated to generate a map.

7. A data fusion device, characterized in that: include: An acquisition module, used for acquiring initial color point cloud data; wherein the initial color point cloud data is obtained based on the fusion of point cloud data and image data; A first semantic segmentation module, used to perform semantic segmentation on the initial color point cloud data to obtain a color point cloud data segmentation result; A second semantic segmentation module is used to perform semantic segmentation on the image data to obtain an image data segmentation result; The correction module is used to correct the initial color point cloud data based on the color point cloud data segmentation result and the image data segmentation result to obtain target color point cloud data.

8. A computer device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that When the computer program product is executed on a computer device, the computer device is caused to execute the method according to any one of claims 1 to 6.

Images