An image enhancement method and apparatus

Abstract

The application discloses an image enhancement method and device, and relates to the technical field of measurement data processing. The method comprises the following steps: acquiring point cloud data and a panoramic image set in road measurement information; fusing the point cloud data and the panoramic image set to obtain a fusion result, wherein the fusion result comprises a corresponding relationship between data points in the point cloud data and pixel points in the panoramic image set; determining first attribute data of the data points; and performing image enhancement processing on the panoramic image set based on the corresponding relationship and the first attribute data of the data points to obtain a target panoramic image set. The point cloud data, which is also road measurement information, is used to perform image enhancement processing on the panoramic image set, so that the image quality of the panoramic image obtained through road measurement can be improved, and road scene element information can be highlighted better.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of measurement data processing, and particularly relates to an image enhancement method and device. BACKGROUND

[0002] With the upgrading of hardware and software devices, mobile measurement systems have developed rapidly, are widely used in geospatial data production and updating, and gradually become the data basis for high-precision maps of unmanned vehicles. At present, a vehicle-mounted mobile measurement system (MMS) integrates a global satellite positioning system, an inertial navigation system, a laser radar scanning system, an image acquisition system, a high-precision time synchronization system, and a vehicle odometer and other sensors.

[0003] Among them, a panoramic image is a kind of super-wide field of view expression, which contains more intuitive and more complete scene information than a normal single image. In the vehicle-mounted mobile measurement system, panoramic image acquisition equipment is installed, so that the advantages of panoramic images are more deeply reflected in the fields of virtual scene construction and high-precision maps of unmanned vehicles. However, due to the influence of weather, surrounding environment and other factors, the panoramic images taken by the panoramic camera will have problems such as low brightness and unclear elements, thereby resulting in low information effectiveness and utilization rate of the panoramic images. SUMMARY

[0004] In order to improve the image quality and element performance of the panoramic image, the present application provides an image enhancement method and device. The technical solution is as follows:

[0005] In a first aspect, the present application provides an image enhancement method, which comprises:

[0006] obtaining point cloud data in road measurement information and a panoramic image set;

[0007] fusing the point cloud data and the panoramic image set to obtain a fusion result, the fusion result containing a corresponding relationship between data points in the point cloud data and pixel points in the panoramic image set;

[0008] determining first attribute data of the data points;

[0009] performing image enhancement processing on the panoramic image set based on the corresponding relationship and the first attribute data of the data points to obtain a target panoramic image set.

[0010] Optionally, the method further comprises:

[0011] determining second attribute data of the pixel points, the second attribute data including color data;

[0012] Determining the color data of the data point based on the corresponding relationship and the color data of the pixel point;

[0013] The target point cloud data is generated according to the color data of the data points.

[0014] Optionally, the panoramic image set includes at least one panoramic image frame, and fusing the point cloud data with the panoramic image set to obtain a fusion result includes:

[0015] Traversing data points in the point cloud data, and determining an associated panoramic image matching the data points from the at least one panoramic image frame;

[0016] Mapping the data points to the coordinate system of the associated panoramic image based on a coordinate system conversion relationship to obtain position data of the data points in the coordinate system;

[0017] determining a pixel point corresponding to the data point in the associated panoramic image according to the position data of the data point in the coordinate system;

[0018] A fusion result is obtained according to the panoramic image set, the position data of the data point, and the pixel points corresponding to the data point.

[0019] Optionally, traversing the data points in the point cloud data and determining an associated panoramic image matching the data points from the at least one panoramic image frame includes:

[0020] Acquiring position data of a panoramic camera; the panoramic image set is obtained by photographing the panoramic camera;

[0021] Determine the distance between the data point and each frame of the panoramic image based on the pose data;

[0022] An associated panoramic image matching the data point is determined from the at least one panoramic image frame according to the distance between the data point and each panoramic image frame.

[0023] Optionally, the first attribute data includes distance data; the distance data indicates the distance between the data point and the center of the coordinate system in which the panoramic image set is located; and performing enhancement processing on the panoramic image set based on the corresponding relationship and the first attribute data of the data point to obtain a target panoramic image set includes:

[0024] Determining depth data of the pixel point based on the corresponding relationship and the distance data of the data point;

[0025] Image enhancement processing is performed on the panoramic image set according to the depth data of the pixel points to obtain the target panoramic image set.

[0026] Optionally, the first attribute data comprises intensity data; the intensity data indicates a reflection intensity of the laser at the data point; and the image enhancement processing of the panoramic image set based on the correspondence and the first attribute data of the data point to obtain the target panoramic image set further comprises:

[0027] constructing a point cloud intensity enhancement model;

[0028] inputting the intensity data of the data point into the point cloud intensity enhancement model to perform intensity enhancement processing to obtain target intensity data of the data point;

[0029] updating the color data of the pixel point based on the correspondence and the target intensity data of the data point;

[0030] performing enhancement processing on the panoramic image set according to the updated color data of the pixel point to obtain the target panoramic image set.

[0031] Optionally, the method further comprises:

[0032] determining boundary data points in the point cloud data according to a collection track of the road measurement information;

[0033] mapping the boundary data points to a first associated panoramic image in the panoramic image set to determine first pixel points corresponding to the boundary data points in the first associated panoramic image;

[0034] mapping the boundary data points to a second associated panoramic image in the panoramic image set to determine second pixel points corresponding to the boundary data points in the second associated panoramic image; the first associated panoramic image and the second associated panoramic image are adjacent frame images in the panoramic image set;

[0035] in a case where a similarity between the first pixel point and the second pixel point does not satisfy a preset condition, determining color data of the boundary data points according to color data of the first pixel point and color data of the second pixel point.

[0036] Optionally, the determining of the color data of the data point based on the correspondence and the color data of the pixel point comprises:

[0037] determining first abnormal data points in the point cloud data according to the correspondence and the color data of the pixel point; color data of a pixel point corresponding to the first abnormal data point is in a missing state;

[0038] determine a first candidate panoramic image according to the associated panoramic image in the panoramic image set that matches the first abnormal data point; the first candidate panoramic image and the associated panoramic image are adjacent frame images in the panoramic image set;

[0039] determine a first candidate pixel point in the first candidate panoramic image that corresponds to the first abnormal data point;

[0040] in a case where color data of the first candidate pixel point is not in a missing state, determine color data of the first abnormal data point according to the color data of the first candidate pixel point.

[0041] Optionally, the determining the color data of the data point based on the correspondence and the color data of the pixel point further includes:

[0042] perform image segmentation and recognition on the panoramic image set to determine category data of the pixel point;

[0043] determine a second abnormal data point in the point cloud data according to the correspondence and the category data of the pixel point; category data of a pixel point corresponding to the second abnormal data point satisfies a preset limitation condition;

[0044] determine a second candidate panoramic image according to the associated panoramic image in the panoramic image set that matches the second abnormal data point; the second candidate panoramic image and the associated panoramic image are adjacent frame images in the panoramic image set;

[0045] determine a second candidate pixel point in the second candidate panoramic image that corresponds to the second abnormal data point;

[0046] in a case where the category data of the second candidate pixel point does not satisfy the preset limitation condition, determine color data of the second abnormal data point according to color data of the second candidate pixel point.

[0047] Optionally, the method further includes:

[0048] construct a ground triangular grid according to a collection track of the road measurement information;

[0049] determine a distance from the second abnormal data point to the ground triangular grid;

[0050] in a case where the distance indicates that the second abnormal data point is a ground data point, determine the second candidate panoramic image, the second candidate panoramic image being a previous frame panoramic image of the associated panoramic image;

[0051] In a case that the distance indicates that the second abnormal data point is not a ground data point, a second candidate panoramic image is determined, the second candidate panoramic image being a next frame panoramic image of the associated panoramic image.

[0052] In a second aspect, the present application provides an image enhancement device, the device comprising:

[0053] An acquisition module is configured to acquire point cloud data and a panoramic image set in road measurement information.

[0054] A fusion module is configured to fuse the point cloud data and the panoramic image set to obtain a fusion result, the fusion result comprising a correspondence between data points in the point cloud data and pixel points in the panoramic image set.

[0055] A first attribute information determination module is configured to determine first attribute data of the data points.

[0056] A first optimization module is configured to perform image enhancement processing on the panoramic image set based on the correspondence and the first attribute data of the data points to obtain a target panoramic image set.

[0057] The image enhancement method and device provided by the present application have the following technical effects:

[0058] According to the present application, the point cloud data and the panoramic image set in the road measurement information are fused to obtain a fusion result, and the fusion result comprises a correspondence between data points in the point cloud data and pixel points in the panoramic image set. Then, the attribute data associated with the pixel points in the panoramic image set is enhanced based on the correspondence and the first attribute data of the data points, that is, the image enhancement processing is performed on the panoramic image set to obtain a target panoramic image set. The target panoramic image set obtained in this way can be enhanced in terms of image depth and pixel color, and the image quality of the panoramic image obtained in the road measurement is improved, and the expressiveness of the road scene elements is improved, so that effective and reliable panoramic image data can be provided for virtual scene construction, high-precision map and other applications.

[0059] Additional aspects and advantages of the present application will be made apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0060] In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments or the prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0061] Figure 1 is a schematic diagram of an implementation environment of an image enhancement method provided by an embodiment of the present application;

[0062] Figure 2 is a schematic diagram of a flow of an image enhancement method provided by an embodiment of the present application;

[0063] Figure 3 is a schematic diagram of grid division of a point cloud provided by an embodiment of the present application;

[0064] Figure 4 is a schematic diagram of a flow of fusion of point cloud data and a panoramic image set provided by an embodiment of the present application;

[0065] Figure 5 is a schematic diagram of multiple coordinate systems provided by an embodiment of the present application;

[0066] Figure 6 is a schematic diagram of data point coordinates in a pixel coordinate system provided by an embodiment of the present application;

[0067] Figure 7 is a schematic diagram of a flow of determination of an associated panoramic image matched with a data point provided by an embodiment of the present application;

[0068] Figure 8 is a schematic diagram of a flow of enhancement of depth data of a panoramic image provided by an embodiment of the present application;

[0069] Figure 9 is a schematic diagram of a flow of enhancement of color data of a panoramic image provided by an embodiment of the present application;

[0070] Figure 10 is a schematic diagram of a flow of point cloud coloring provided by an embodiment of the present application;

[0071] Figure 11 is a schematic diagram of a flow of optimization of point cloud coloring provided by an embodiment of the present application;

[0072] Figure 12 is a schematic diagram of data point neighborhood search provided by an embodiment of the present application;

[0073] Figure 13 is a schematic diagram of a flow of optimization of point cloud coloring in a data missing case provided by an embodiment of the present application;

[0074] Figure 14 is a schematic diagram of a flow of optimization of point cloud coloring in a data interference case provided by an embodiment of the present application;

[0075] Figure 15is another process schematic diagram for optimizing point cloud coloring in a data interference case provided by an embodiment of the present application;

[0076] Figure 16 is a schematic diagram of a ground triangular grid provided by an embodiment of the present application;

[0077] Figure 17 is a specific fusion and optimization process schematic diagram provided by an embodiment of the present application;

[0078] Figure 18 is a schematic diagram of an image enhancement device provided by an embodiment of the present application;

[0079] Figure 19 is another schematic diagram of an image enhancement device provided by an embodiment of the present application;

[0080] Figure 20 is a hardware structure schematic diagram of a device for implementing an image enhancement method provided by an embodiment of the present application. DETAILED DESCRIPTION

[0081] In order to improve the image quality and element expressiveness of a panoramic image, an image enhancement method and device are provided by an embodiment of the present application. The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work are within the scope of protection of the present application. Examples of the described embodiments are shown in the drawings, wherein the same or similar reference signs represent the same or similar elements or elements with the same or similar functions throughout.

[0082] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or server including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to the process, method, product, or device.

[0083] In order to facilitate understanding of the technical solutions described in the embodiments of the present application and the technical effects generated thereby, the related professional terms involved in the embodiments of the present application are explained:

[0084] Mobile Mapping System, MMS for short, is a special detection instrument used in the fields of earth science, surveying and mapping science and technology, and water conservancy engineering.

[0085] Point cloud: is a mass point set expressing the target space distribution and target surface characteristics in the same space reference system. After obtaining the spatial coordinates of each sampling point of the object surface, the point set is obtained, which is called "point cloud" (PointCloud).

[0086] IMU: Inertial Measurement Unit, which is used to measure the three-axis attitude angle (or angular velocity) and acceleration of the object.

[0087] TIN grid: Triangulated Irregular Network, TIN for short, is a triangular network model composed of discrete and uneven points in space. The digital elevation model based on irregular triangular partition is described by a series of non-intersecting and non-overlapping triangular surfaces connected to each other. The elevation value of any point on the triangular surface can be obtained by weighted average interpolation of the elevation values of the nearby triangular vertices.

[0088] Photometric consistency: photometric loss refers to the luminosity (i.e. gray value) of the same point or patch between two frames almost does not change, and geometric consistency refers to the scale (i.e. size) of the same static point between adjacent frames almost does not change.

[0089] Please refer to Figure 1 , which is an implementation environment diagram of an image enhancement method provided by an embodiment of the present application, as shown in Figure 1 , the implementation environment can at least include a vehicle 110 and a server 120.

[0090] Specifically, the vehicle 110 is equipped with a mobile mapping system to collect road measurement information. The mobile mapping system can include a laser radar scanning system and an image acquisition system, wherein the laser radar scanning system is used to generate point cloud data, and the image acquisition system can be a panoramic camera used to generate panoramic images. The vehicle 110 uploads the generated point cloud data and panoramic images to the server 120 for processing.

[0091] The server 120 can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms. The server 120 can include a network communication unit, a processor, a memory, and the like. The vehicle 110 and the server 120 can be connected through a communication network, which is not limited in the present application. Specifically, the server 120 performs data fusion on the point cloud data and the panoramic image set in the obtained road measurement information, to obtain a fusion result, wherein the fusion result contains the correspondence between the data points in the point cloud data and the pixel points in the panoramic image set; then the server 120 uses the correspondence and the first attribute data of the data points to enhance the attribute data associated with the pixel points in the panoramic image set, that is, to perform image enhancement processing on the panoramic image set, to obtain a target panoramic image set. The target panoramic image set obtained can be enhanced in image depth and pixel color, improving the image quality of the panoramic image obtained in road measurement and enhancing the expressiveness of road scene elements, thereby providing effective and reliable panoramic image data for virtual scene construction, high-precision map and other applications.

[0092] The embodiments of the present application can also be implemented in combination with cloud technology. Cloud technology refers to a kind of hosting technology that unifies a series of resources such as hardware, software and network in a wide area network or local area network to realize data calculation, storage, processing and sharing. It can also be understood as a general term for network technology, information technology, integration technology, management platform technology and application technology based on cloud computing business model application. Cloud technology needs to be supported by cloud computing. Cloud computing is a computing mode that distributes computing tasks on a resource pool composed of a large number of computers, so that various application systems can obtain computing power, storage space and information services according to needs. The network that provides resources is called "cloud". Specifically, the server 120 is located in the cloud, and the server 120 can be a physical machine or a virtual machine.

[0093] Please refer to Figure 2FIG. 1 is a flowchart of an image enhancement method provided by an embodiment of the present application. The present application provides the method operation steps as described in the embodiments or the flowchart, but more or fewer operation steps can be included based on conventional or non-inventive labor. The order of steps listed in the embodiments is only one of the many step execution orders, and does not represent the only execution order. In actual system or server product execution, the method order shown in the embodiments or the drawings can be executed in sequence or in parallel (for example, in a parallel processor or multi-thread processing environment). Please refer to Figure 2 The image enhancement method provided by an embodiment of the present application can include the following steps:

[0094] S210: Obtain point cloud data and a panoramic image set in road measurement information.

[0095] In the embodiments of the present application, a vehicle-mounted mobile measurement system integrating a global satellite positioning system, an inertial navigation system, a laser radar scanning system, an image acquisition system, a high-precision time synchronization system, and a vehicle odometer and other sensors is used for road measurement. The obtained road measurement information can be used as basic data for applications such as virtual scene construction, three-dimensional scene reconstruction, and high-precision maps.

[0096] In the embodiments of the present application, the point cloud data contains a plurality of data points obtained by sampling and attribute data of each data point. For example, according to the laser measurement principle, the attribute data of each data point can include three-dimensional coordinates (X, Y, Z) and laser reflection intensity (Intensity), which is related to the surface material and roughness of the sampling object, the laser incidence angle direction, the laser wavelength, and the instrument emission energy. According to the photogrammetry principle, the attribute data of each data point can include three-dimensional coordinates (X, Y, Z) and color information (R, G, B).

[0097] In the embodiments of the present application, the panoramic image set contains at least one panoramic image. For example, the panoramic image can be obtained through multi-channel signal acquisition, camera calibration, image stitching, etc., and can also be directly captured by a panoramic camera.

[0098] In an embodiment of the present application, according to the time range information of the point cloud data and the panoramic image set of the road measurement information, the point cloud data and the panoramic image set in the same time range are selected to ensure the accuracy of data fusion.

[0099] Considering that the point cloud data is a collection of massive data points, in order to improve the fast and orderly reading of the point cloud data in the fusion process, an embodiment of the present application further provides a construction method of a point cloud data index file. Specifically, the file header information of the point cloud data is obtained, mainly including the total number of point clouds and the range. According to the range of the point cloud, the point cloud is divided according to the grid,Figure 3 A plane schematic diagram is shown according to a 10-meter x 10-meter grid, and an index tree of point cloud data is constructed, the 0th layer is the root node, the entire point cloud range, the middle layer is the parent node corresponding to the divided grid, which does not store point cloud, and the last layer is the leaf node, and the leaf node stores point cloud data. Considering the large amount of point cloud data and the limitation of computer memory, the point cloud data can be traversed first, and the point cloud data in the leaf node is temporarily not stored, but a counter is set, and only the number of point clouds stored in each leaf node is allocated. The second time the point cloud data is traversed, according to the memory size and the number of point clouds stored in the leaf node, the counter is reduced by 1 after storing a data point in the leaf node, until the counter is 0, and the index information corresponding to the leaf node and the point cloud data file organized according to the leaf node are generated. In this way, the point cloud data is traversed in a loop until all leaf node counters are 0, and the index information of each leaf node and the index file of the point cloud data organized according to each leaf node are obtained. In order to improve the efficiency of the index, preferably, concurrent processing can also be performed using Open Multi-Processing (shared memory parallel programming) technology.

[0100] S220: fusing the point cloud data and the panoramic image set to obtain a fusion result, the fusion result containing the correspondence between the data points in the point cloud data and the pixel points in the panoramic image set.

[0101] In the embodiments of the present application, the fusion process of the point cloud data and the panoramic image set is mainly through experiment field calibration and a series of data processing to register the point cloud data and the panoramic images in the panoramic image set, so that through registration, the correspondence between the data points in the point cloud data and the pixel points in the registered panoramic image can be determined. Specifically, the registration conversion relationship between the three-dimensional point cloud data and the panoramic image is the key to fusion, which needs to be strictly converted according to the spatial geometry and physical meaning of the calibration parameters and the trajectory parameters.

[0102] S230: determining first attribute data of the data points.

[0103] In the embodiments of the present application, the first attribute data of the data points can include but is not limited to coordinate data, distance data, intensity data, etc. of the data points. Exemplarily, the coordinate data of the data points can be the coordinates P W (X W , Y W , Z W), the coordinate data can be determined directly from the point cloud data; the distance data of the data point can be the distance between the data point and the center of the coordinate system in which the panoramic image set is located, more specifically, the distance between the data point and the center of the coordinate system in which the panoramic image registered with the data point is located, and the center can be the camera center, and the distance data can be determined according to the fusion result; the intensity data of the data point indicates the reflection intensity of the laser at the sampling point corresponding to the data point, and the intensity data can be directly determined in the point cloud data obtained according to the laser measurement principle.

[0104] S240: performing image enhancement processing on the panoramic image set based on the correspondence and the first attribute data of the data point, to obtain a target panoramic image set.

[0105] In the embodiments of the present application, the associated attribute data of the pixel points in the panoramic image set is enhanced based on the correspondence between the data points in the point cloud data and the pixel points in the panoramic image set and the first attribute data of the data points, that is, the panoramic images in the panoramic image set are subjected to image enhancement processing, to obtain the target panoramic images after enhancement. For example, a depth map can be generated according to the distance data of the data points, so that the panoramic images matched with the data points are fused according to the depth map, to obtain an RGB-D image; and the data points can also be fused into the panoramic images according to the intensity data, to update the RGB (optical three primary colors, R represents Red, G represents Green, and B represents Blue) values in the panoramic images, to highlight the element information contained in the panoramic images.

[0106] As can be seen from the above embodiments, the scheme provided by the present application fuses the point cloud data in the road measurement information and the panoramic image set to obtain a fusion result, and the fusion result contains the correspondence between the data points in the point cloud data and the pixel points in the panoramic image set; then the attribute data associated with the pixel points in the panoramic image set is enhanced using the correspondence and the first attribute data of the data points, that is, the panoramic image set is subjected to image enhancement processing, to obtain a target panoramic image set. The target panoramic image set obtained can be enhanced in terms of image depth and pixel color, improves the image quality of the panoramic images obtained in road measurement, and improves the expressiveness of the road scene elements, so as to provide effective and reliable panoramic image data for applications such as virtual scene construction and high-precision map.

[0107] Please refer to Figure 4 which is a flowchart for fusing point cloud data and a panoramic image set provided by the embodiments of the present application. The panoramic image set includes at least one panoramic image, as shown in Figure 4 The step S220 can include the following steps:

[0108] S310: traversing the data points in the point cloud data, determining an associated panoramic image matching the data point from at least one panoramic image.

[0109] The key of the fusion process of the point cloud data and the panoramic image set is to register the point cloud data and the panoramic images in the panoramic image set, that is, for each data point in the point cloud data, an associated panoramic image is matched.

[0110] S320: mapping the data point into a coordinate system in which the associated panoramic image is located based on the coordinate system conversion relationship, to obtain position data of the data point in the coordinate system.

[0111] The coordinates of the data points in the point cloud data are absolute coordinates in a world coordinate system, representing the actual positions of the data points. According to the collinearity principle, the data point, the photographic center and the imaging point are collinear, so the coordinate data of the corresponding pixel point can be determined according to the three-dimensional coordinate data of the data point. The coordinate system in which the above-mentioned associated panoramic image is located is a pixel coordinate system in which the image is located.

[0112] S330: determining a pixel point corresponding to the data point in the associated panoramic image according to the position data of the data point in the coordinate system.

[0113] Figure 5 A schematic diagram of multiple coordinate systems is shown. Specifically, mapping the data point P into the pixel coordinate system in which the associated panoramic image is located mainly includes:

[0114] Step 1: as shown in formula (1), converting the coordinates P W (X W , Y W , Z W ) of the data point P in the world coordinate system into the coordinates P M (X M , Y M , Z M ) in the geographic coordinate system with the center of the panoramic image as the center:

[0115]

[0116] wherein (Δ X , Δ Y , Δ Z ) represents the offset parameters between the vehicle positioning system and the camera.

[0117] Step 2: as shown in formula (2) and (3), converting P M (X M , Y M , Z M ) into the coordinates P S (X S, Y S , Z S ):

[0118]

[0119]

[0120] in are the parameters of the rotation matrix, which are composed of the three pose angles of the panoramic image: r (roll), p (pitch), and h (heading).

[0121] Step 3: As shown in formulas (4) and (5), S (X S , Y S , Z S ) is converted to polar coordinates P in the panoramic image sphere P (R, φ, θ):

[0122]

[0123]

[0124] The radius of the panoramic image sphere is R, and point P S The line connecting the origin of the spherical coordinate system and the X S OY S The angle between the plane and X is φ. S OZ S The angle between the two planes is θ.

[0125] Step 4: As shown in formulas (6) and (7), the polar coordinates P P (R, φ, θ) is converted to the coordinates (m, n) in the pixel coordinate system of the panoramic image:

[0126]

[0127]

[0128] like Figure 6 As shown, W is the width of the panoramic image, H is the height of the panoramic image, m is the number of columns where the data point P is located in the panoramic image, n is the number of rows where the data point is located in the panoramic image, and the units of m and n are pixels.

[0129] So far, for the data point P, the corresponding pixel point is the pixel point (m, n) in the matching panoramic image.

[0130] S340: Obtain a fusion result based on the panoramic image set, the position data of the data points, and the pixel points corresponding to the data points.

[0131] Through the registration, the coordinate data of the data points in the world coordinate system is converted into coordinate data in a pixel coordinate system of the panoramic image (i.e., the position data described above), and the corresponding pixel points in the panoramic image are determined. The fusion result after the fusion of the point cloud data and the panoramic image set contains all data in the panoramic image set, the converted position data in the registration process, the corresponding relationship between the data points and the pixel points, and of course all data of the point cloud data.

[0132] The above embodiment realizes the accurate fusion of the point cloud data and the panoramic image set, and matches the corresponding panoramic image and the pixel point in the panoramic image for each data point, thereby providing a reliable data basis for subsequent image enhancement of the panoramic image set using the point cloud data.

[0133] Referring to Figure 7 , which is a flowchart for determining the associated panoramic image matched with the data point according to an embodiment of the present application. As Figure 7 indicated, the above step S310 can specifically include the following steps:

[0134] S311: Obtain the pose data of the panoramic camera; the panoramic image set is obtained by the panoramic camera.

[0135] Specifically, the inertial navigation data and the navigation data are jointly adjusted and solved by using Inertial Exploer (a software for processing global positioning data difference and inertial positioning), to generate the inertial navigation pose data; the pose data of the panoramic camera is solved in combination with the high-precision calibration parameters of the panoramic camera and the inertial measurement unit.

[0136] Further, according to the time range corresponding to the point cloud data and the panoramic image set, the pose data of the corresponding panoramic camera is loaded.

[0137] S312: Determine the distance between the data points and each frame of panoramic image based on the pose data.

[0138] Specifically, first, the position data of the panoramic image, i.e., the position data in the world coordinate system, can be determined according to the pose data of the panoramic camera, the focal length of the camera, etc., which can be represented by the position of the image center point of the panoramic image in the world coordinate system. Secondly, each data point is traversed, and the distance between the current traversed data point and the image center of each frame of panoramic image is calculated.

[0139] S313: According to the distance between the data points and each frame of panoramic image, determine the associated panoramic image matched with the data point from at least one frame of panoramic image.

[0140] Specifically, according to the distance between the data point and each panoramic image, the panoramic image closest to the data point is selected from at least one panoramic image as the associated panoramic image matched with the data point.

[0141] Preferably, each panoramic image is continuous in time sequence, and the panoramic image closest to the data point can be found based on dichotomy without calculating the distance between each panoramic image and the data point.

[0142] In the above embodiment, the high-precision calibration parameters are combined with the IMU pose to perform panoramic image pose solving, and accurate matching between the data point and the panoramic image is realized.

[0143] Referring to Figure 8 which is a flow diagram provided by an embodiment of the present application for enhancing panoramic image depth data. The first attribute data includes distance data, which indicates the distance between the data point and the center of the coordinate system in which the panoramic image set is located, as shown in Figure 8 The step S240 can include the following steps:

[0144] S410: determining the depth data of the pixel point based on the correspondence relationship and the distance data of the data point.

[0145] The distance data of the data point is the distance between the data point and the center of the coordinate system in which the panoramic image set is located, and more specifically, the distance between the data point and the center of the coordinate system in which the associated panoramic image matched with the data point is located. The coordinate system is the spherical coordinate system of the panoramic image, and the center of the coordinate system can be the camera center. According to the fusion process described above, the coordinates of the data point in the spherical coordinate system of the panoramic image can be determined, and then the distance between the coordinates and the origin of the coordinate system can be determined.

[0146] Based on the correspondence relationship and the distance data of the data point, the depth data of the pixel point is determined, that is, the distance data of the data point is directly assigned to the pixel point corresponding to the data point as the depth data of the pixel point.

[0147] Due to the difference between the point cloud resolution and the panoramic image resolution, a situation that one pixel point corresponds to multiple data points or one pixel point has no corresponding data point may occur. For the situation that one pixel point corresponds to multiple data points, the distance expected values of the multiple data points can be taken as the depth value of the pixel point; for the situation that one pixel point has no corresponding data point, the distance data of the surrounding data points can be interpolated to assign the depth data to the pixel point.

[0148] Secondly, the point cloud has the phenomenon of occlusion, which can also cause the situation that one pixel point corresponds to multiple data points, and the minimum value of the distance values of the multiple data points can be taken as the depth value of the pixel point.

[0149] In addition, the point cloud also has a projection effect. When the distance of a data point in the center of a window is greater than the distances of surrounding data points in the point cloud, the distance of the data point in the center of the window is replaced by the distance of a surrounding adjacent data point, as a depth value of a pixel point corresponding to the window.

[0150] S430: performing image enhancement processing on the panoramic image set according to the depth data of the pixel points, to obtain the target panoramic image set.

[0151] Specifically, a depth map of a corresponding panoramic image is constructed according to the depth data of the pixel points, the depth map is fused with the corresponding panoramic image, and an RGB-D image is obtained, that is, a target panoramic image containing three primary colors and depth is obtained.

[0152] In the above embodiment, the depth data in the panoramic image is enhanced by the point cloud data. The depth data can directly reflect the geometric shape of a visible object surface, and many problems in three-dimensional target description can be conveniently solved by using the depth data, and three-dimensional scene reconstruction is facilitated by using the target panoramic image.

[0153] Referring to Figure 9 which is a process schematic diagram for enhancing color data of a panoramic image provided by an embodiment of the present application. The first attribute data includes intensity data, and the intensity data indicates the reflection intensity of laser at the data point; as Figure 9 shown, step S240 can further include the following steps:

[0154] S420: constructing a point cloud intensity enhancement model.

[0155] In view of the small range and low resolution of the laser radar echo intensity, the point cloud intensity can be enhanced. According to the laser radar echo intensity, the point cloud distance and other constraint conditions, and considering the case of over-enhancement, a point cloud intensity enhancement model as shown in formula (8) is established:

[0156]

[0157] wherein R is the distance of the data point to the center of the laser radar, MAXR is the maximum range of the point cloud, A is the echo intensity of the laser radar, and I is the enhanced intensity value.

[0158] Meanwhile, the three-dimensional geometric information of the laser point cloud data is the key data for three-dimensional reconstruction, and the reflection intensity data can reflect the differences between different objects and the differences between different colors of the same object to some extent, and embodies the characteristics of point cloud texture. The enhanced intensity information can improve the contrast of rendering when rendering the point cloud.

[0159] S440: inputting the intensity data of the data point into the point cloud intensity enhancement model to perform intensity enhancement processing, to obtain target intensity data of the data point.

[0160] That is, the original intensity data of the data point is A in formula (8), and the target intensity data is I obtained in formula (8).

[0161] S460: updating the color data of the pixel point based on the correspondence and the target intensity data of the data point.

[0162] Specifically, in view of the difference in expressing latitude and range between the point cloud data and the panoramic image, it is necessary to establish a mapping relationship between the intensity data and the color data. First, the intensity data (i.e. the target intensity data) of the enhanced data point is normalized to the [0-255] interval, then the RGB data of the panoramic image is normalized to the [0-255] interval according to the weight ratio of 3:6:1, and a certain proportion of weight is allocated to the intensity data and the RGB data. Finally, the normalized intensity data and the data in the three channels of the panoramic image RGB are respectively calculated according to the allocated weight ratio to obtain the weighted values, and it is judged whether each weighted value exceeds the numerical threshold of the corresponding channel. If it does not exceed, it is used as the updated channel value, and if it exceeds the threshold, the channel value is set to 255. The updated channel values of the three channels are used as the color data of the updated pixel point.

[0163] S480: performing enhancement processing on the panoramic image set according to the color data of the updated pixel point to obtain a target panoramic image set.

[0164] Specifically, the panoramic image can be re-rendered according to the color data of the updated pixel point to obtain a target panoramic image. In the above embodiment, the target panoramic image highlights the element information in the image compared to the original panoramic image, and can improve the extraction accuracy of road elements in high-precision map generation, scene reconstruction and other applications.

[0165] Please refer to Figure 10 , which is a flowchart of point cloud coloring provided by an embodiment of the present application. As Figure 10 shown, the method can further include the following steps:

[0166] S210: obtaining point cloud data and a panoramic image set in road measurement information.

[0167] S220: fusing the point cloud data and the panoramic image set to obtain a fusion result, the fusion result containing the correspondence between the data points in the point cloud data and the pixel points in the panoramic image set.

[0168] Steps S210 and S220 can refer to the foregoing embodiments, which will not be described here.

[0169] S530: determining the second attribute data of the pixel point, the second attribute data including color data.

[0170] The color data of the pixel points includes RGB three-channel numerical values.

[0171] S540: Determine the color data of the data points based on the correspondence and the color data of the pixel points.

[0172] That is, the color data of the pixel points is directly assigned to the corresponding data points as the color data of the data points.

[0173] S550: Generate target point cloud data according to the color data of the data points.

[0174] According to the color data of the data points, the target point cloud data is generated, which indicates a color point cloud or a point cloud of one or more specific colors.

[0175] The point cloud data collected by the laser radar has highly accurate position information, but lacks texture and spectral data. Therefore, intelligent fusion of the point cloud data and the panoramic image can obtain a color point cloud with texture properties. In the above embodiment, the color data of the pixel points in the panoramic image set can be used to optimize the point cloud data, so that a color point cloud with high precision, low noise and high quality can be quickly and efficiently obtained, thereby providing rich and accurate attribute information for classification and segmentation of the point cloud data to improve the effect of point cloud data classification and target extraction, and providing rich texture information for reconstruction of high-precision maps and three-dimensional models.

[0176] Figure 11 A flowchart of optimizing point cloud coloring is shown. As shown in Figure 11 To address the color discontinuity problem caused by different exposure intensities and sensitivities between panoramic images when segmenting and coloring the point cloud, the method can further include the following steps:

[0177] S610: Determine the boundary data points in the point cloud data according to the collection trajectory of the road measurement information.

[0178] Specifically, the collection trajectory points of the panoramic image are determined according to the collection trajectory of the road measurement information; then the adjacent trajectory midpoint is determined by interpolation according to the collection trajectory points of the panoramic image; next, a voxel bounding box is constructed with the trajectory midpoint as the center, the distance from the adjacent trajectory midpoint as the length L, the road width as W, and the height as H, which is a division of the space where the collection trajectory is located. Finally, the data points on the surface of the voxel bounding box are extracted from the point cloud data as the boundary data points.

[0179] S620: Map the boundary data points to the first associated panoramic image in the panoramic image set to determine the first pixel point in the first associated panoramic image corresponding to the boundary data points.

[0180] S630: mapping the boundary data point into a second associated panoramic image in the panoramic image set, and determining a second pixel point in the second associated panoramic image corresponding to the boundary data point.

[0181] The first associated panoramic image and the second associated panoramic image are adjacent frames in the panoramic image set.

[0182] Specifically, the determination of the first associated panoramic image and the second associated panoramic image can refer to step S310 in the foregoing embodiments, and the first associated panoramic image and the second associated panoramic image can be the two frames of panoramic images closest to the data point and adjacent to each other. Meanwhile, the determination of the first pixel point and the second pixel point can refer to steps S320 and S330 in the foregoing embodiments, which will not be described herein again.

[0183] S640: in the case where the similarity between the first pixel point and the second pixel point does not satisfy the preset condition, determining the color data of the boundary data point according to the color data of the first pixel point and the color data of the second pixel point.

[0184] S650: in the case where the similarity between the first pixel point and the second pixel point satisfies the preset condition, determining the color data of the boundary data point according to the panoramic images before the time sequence in the first associated panoramic image and the second associated panoramic image.

[0185] For example, the photometric consistency can be introduced to measure the similarity between the pixel points, and the preset condition limits the threshold of the similarity. If the similarity is less than the threshold, as shown in formula (9), the final color data V of the boundary data point can be determined according to the color data V1 of the first pixel point, the color data V2 of the second pixel point, the distance D1 of the boundary data point from the center of the camera corresponding to the first associated panoramic image, and the distance D2 of the boundary data point from the center of the camera corresponding to the second associated panoramic image.

[0186]

[0187] In the above embodiments, the color data of the boundary data point is adjusted to solve the problem of color discontinuity in the point cloud segmentation coloring due to the differences in exposure intensity and sensitivity between panoramic images, so that the coloring of the color point cloud is uniform and consistent.

[0188] In another possible implementation, a point cloud change template region can be first determined as a reference, and then the root mean square error of the RGB three channel values is calculated according to the attribute information of the point cloud change template region and the average value is taken as the feature information of the point cloud change template region. The point cloud grid index file provided in the foregoing embodiments is used to traverse the point cloud data, Figure 12As shown, a neighborhood search radius R (such as 30 cm) is set around the current data point P being traversed, and the root mean square error of the RGB values of all data points in the neighborhood is calculated and averaged. If the similarity or difference between the feature information of the point cloud change template region and the average value of the RGB values of all data points in the neighborhood meets the preset condition, the average value of the RGB values of all data points in the neighborhood is assigned to the current data point, and the process is repeated until the traversal of the point cloud data is completed.

[0189] Figure 13 A flowchart of the process of optimizing point cloud coloring in the case of data loss is shown. The color data of the pixel points may be missing due to the holes in the panoramic image acquisition or the loss of part of the image in the splicing. In order to optimize the coloring effect of the point cloud, the color data of the pixel points corresponding to the first abnormal data point in the point cloud data is determined according to the corresponding relationship and the color data of the pixel points. Figure 13 As shown, step S540 can further include the following steps:

[0190] S710: determining a first abnormal data point in the point cloud data according to the corresponding relationship and the color data of the pixel points; the color data of the pixel point corresponding to the first abnormal data point is in a missing state.

[0191] For example, due to the size of the image and the installation angle, holes will inevitably occur under the vehicle during panoramic camera shooting, and part of the vehicle information will also be captured. The color data of part of the pixel points is in a missing state or lacks credibility.

[0192] For example, due to the installation of the laser radar at the rear of the vehicle and the large inclination angle, the information of the traffic sign will be lost when the adjacent panoramic images are fused. The color data of part of the pixel points is in a missing state.

[0193] S720: determining a first candidate panoramic image according to the associated panoramic image in the panoramic image set that matches the first abnormal data point.

[0194] Preferably, due to the hole problem caused by the size of the image and the installation angle of the panoramic camera, the next frame of the associated panoramic image is taken as the first candidate panoramic image.

[0195] Preferably, due to the loss of part of the image caused by the installation position and angle of the laser radar, the previous frame of the associated panoramic image is taken as the first candidate panoramic image.

[0196] S730: determining a first candidate pixel point in the first candidate panoramic image that corresponds to the first abnormal data point.

[0197] Specifically, the first abnormal data point is remapped to the pixel coordinate system in which the first candidate panoramic image is located, and the first candidate pixel point corresponding to the first abnormal data point is determined according to the position data of the first abnormal data point in the pixel coordinate system.

[0198] S740: In a case where the color data of the first candidate pixel point is not in the missing state, determining the color data of the first abnormal data point according to the color data of the first candidate pixel point.

[0199] S750: In a case where the color data of the first candidate pixel point is in the missing state, re-determining the first candidate panoramic image.

[0200] Preferably, the first candidate panoramic image determined for the first time and the associated panoramic image are adjacent frame images in the panoramic image set, which can be a previous frame panoramic image or a next frame panoramic image. The first candidate panoramic image determined for the second time can be a second frame panoramic image before the associated panoramic image or a second frame panoramic image after the associated panoramic image.

[0201] In the above embodiment, in view of the problem of missing color data of pixel points caused by holes possibly existing during panoramic image acquisition or partial image loss during splicing, the color data of the first abnormal data point is supplemented by the color data of the first candidate pixel point corresponding to the first abnormal data point in the front and back frame panoramic images of the associated panoramic image determined by initial matching, thereby improving the consistency of point cloud coloring.

[0202] Figure 14 A flowchart for optimizing point cloud coloring in a data interference situation is shown. During data acquisition, moving people and vehicles will interfere with pixel color, thereby affecting the coloring effect of the point cloud. In order to optimize the coloring effect of the point cloud, as shown in Figure 14 Based on the correspondence and the color data of the pixel point, the color data of the data point can also include the following steps:

[0203] S810: Image segmentation and recognition are performed on the panoramic image set to determine the category data of the pixel point.

[0204] Specifically, based on deep learning-based image semantic segmentation and image category recognition, the category data corresponding to each pixel point in each panoramic image is determined, such as category data representing people, vehicles, trees, buildings, road surfaces, etc.

[0205] S820: According to the correspondence and the category data of the pixel point, a second abnormal data point in the point cloud data is determined; the category data of the pixel point corresponding to the second abnormal data point satisfies a preset limit condition.

[0206] Specifically, the preset limit condition indicates one or more sensitive categories that will interfere with point cloud coloring, such as driving and vehicles in motion. If the category data of the pixel point corresponding to the data point represents the sensitive category described above, the data point can be regarded as a second abnormal data point.

[0207] S830: determining a second candidate panoramic image according to the associated panoramic image in the panoramic image set that matches the second abnormal data point.

[0208] In a case where the first determined second candidate panoramic image and the associated panoramic image are adjacent frame images in the panoramic image set, the second candidate panoramic image can be a previous frame panoramic image of the associated panoramic image or a next frame panoramic image of the associated panoramic image, and the application does not limit this.

[0209] S840: determining a second candidate pixel point corresponding to the second abnormal data point in the second candidate panoramic image.

[0210] Specifically, the second abnormal data point is remapped to a pixel coordinate system in which the second candidate panoramic image is located, and the second candidate pixel point corresponding to the second abnormal data point is determined according to the position data of the second abnormal data point in the pixel coordinate system.

[0211] S850: in a case where the category data of the second candidate pixel point does not satisfy a preset restriction condition, determining the color data of the second abnormal data point according to the color data of the second candidate pixel point.

[0212] That is, if the category corresponding to the second candidate pixel point is not a sensitive category that can cause interference, the color data of the second candidate pixel point can be directly assigned to the second abnormal data point.

[0213] S860: in a case where the category data of the second candidate pixel point satisfies the preset restriction condition, re-determining the second candidate panoramic image.

[0214] Preferably, the second candidate panoramic image determined for the second time can be a second frame panoramic image before the associated panoramic image or a second frame panoramic image after the associated panoramic image.

[0215] Preferably, the number of times of re-determining the second candidate panoramic image can be set, for example, at most 3 times.

[0216] In the above embodiment, in order to solve the problem that a person or a vehicle moving in a data collection process can cause interference to pixel color and further affect the coloring effect of point cloud, the category of each pixel point is first identified to extract the second abnormal data point, and then the color data of the second abnormal data point is corrected according to the color data of the second candidate pixel point corresponding to the second abnormal data point in the previous and next frame panoramic images of the initially matched associated panoramic image, thereby improving the consistency of point cloud coloring.

[0217] Figure 15Another flow diagram for optimizing point cloud coloring in a data interference situation is shown. Based on the identification of pixel point category data, the method can further include the following steps:

[0218] S910: Construct a ground triangular grid according to the collection track of the road measurement information.

[0219] S920: Determine the distance of the second abnormal data point to the ground triangular grid.

[0220] As shown in Figure 16 The construction of the ground triangular grid can first determine the track projection point of the track point on the road surface according to the position, pose and calibration parameters of each track point; secondly, the projection point straight line equation is constructed through the track projection point coordinates and pose direction, and the coordinates of the boundary points of each W distance on both sides are calculated; finally, the TIN grid is constructed according to all track projection points and boundary points.

[0221] The vertical distance from the second abnormal data point to the nearest triangle in the ground triangular grid is taken as the distance of the second abnormal data point to the ground triangular grid.

[0222] S930: In the case that the distance represents that the second abnormal data point is a ground data point, determine the second candidate panoramic image, and the second candidate panoramic image is the previous frame panoramic image of the associated panoramic image.

[0223] Specifically, a preset distance threshold is obtained, and if the distance of the second abnormal data point to the ground triangular grid is less than the distance threshold, the second abnormal data point is a ground data point, otherwise it is a non-ground data point.

[0224] When the second abnormal data point is a ground data point, the previous frame panoramic image of the associated panoramic image is taken as the second candidate panoramic image.

[0225] S940: In the case that the distance represents that the second abnormal data point is a non-ground data point, determine the second candidate panoramic image, and the second candidate panoramic image is the next frame panoramic image of the associated panoramic image.

[0226] Further, the steps S840-S860 in the above embodiments can be performed until the color data of the second abnormal data point is determined.

[0227] Furthermore, in the case where the second abnormal data point is a non-ground data point, the difference between the distance from the second abnormal data point to the camera center corresponding to the associated panoramic image and the depth value of the corresponding pixel point is determined; if the difference is less than a preset threshold, it is determined that the interference level is low, and the color data of the corresponding pixel point can be directly assigned to the second abnormal data point; otherwise, the next frame of the associated panoramic image is used as the second candidate panoramic image, and remapping and the above-mentioned judgment process are performed until the color data of the second abnormal data point is determined.

[0228] Preferably, considering that panoramic images at a longer distance have a greater impact on the point cloud coloring accuracy, the maximum number of iterations can be set to three.

[0229] In the above embodiment, the degree of interference of the color data of the corresponding pixel point is measured according to whether the second abnormal data point is a ground data point, and then when the interference is low, the color data of the initial corresponding pixel point is still used as the color data of the second abnormal data point, thereby ensuring the data accuracy of the point cloud coloring.

[0230] Figure 17 FIG. 1 shows a specific fusion and optimization process diagram provided by an embodiment of the present application. Figure 17 As shown, it includes the registration of point cloud data and panoramic image set, mapping, pixel category recognition, ground point cloud recognition, point cloud intensity enhancement, point cloud coloring and panoramic image enhancement process. Each process can refer to the above embodiment and will not be repeated here. It should be noted that, Figure 17 It only provides a specific and operational fusion and optimization process, and does not limit the order of executing the method provided in the embodiments of this application. Other feasible operation step sequences are also within the scope disclosed in the embodiments of this application.

[0231] The embodiment of the present application also provides an image enhancement device 1800, such as Figure 18 As shown, the apparatus 1800 may include:

[0232] An acquisition module 1810 is configured to acquire point cloud data and panoramic image sets from road measurement information;

[0233] a fusion module 1820 for fusing the point cloud data with the panoramic image set to obtain a fusion result, wherein the fusion result includes a correspondence between data points in the point cloud data and pixel points in the panoramic image;

[0234] A first attribute information determining module 1830 is configured to determine first attribute data of the data point;

[0235] The first optimization module 1840 is configured to perform image enhancement processing on the panoramic image set based on the correspondence and the first attribute data of the data points, to obtain a target panoramic image set.

[0236] In an embodiment of the present application, as shown in Figure 19 The apparatus 1800 can further include:

[0237] The second attribute information determination module 1850 is configured to determine second attribute data of the pixel points, the second attribute data including color data.

[0238] The color determination module 1860 is configured to determine color data of the data points based on the correspondence and the color data of the pixel points.

[0239] The second optimization module 1870 is configured to generate the target point cloud data according to the color data of the data points.

[0240] In an embodiment of the present application, the fusion module 1820 can include:

[0241] The first matching unit is configured to traverse the data points in the point cloud data, and determine an associated panoramic image matching the data points from the at least one panoramic image.

[0242] The mapping unit is configured to map the data points into a coordinate system in which the associated panoramic image is located based on a coordinate system conversion relationship, to obtain position data of the data points in the coordinate system.

[0243] The second matching unit is configured to determine pixel points corresponding to the data points in the associated panoramic image according to the position data of the data points in the coordinate system.

[0244] The fusion unit is configured to obtain a fusion result according to the panoramic image set, the position data of the data points, and the pixel points corresponding to the data points.

[0245] In an embodiment of the present application, the first matching unit can include:

[0246] The pose data subunit is configured to obtain pose data of a panoramic camera; the panoramic image set is obtained by the panoramic camera.

[0247] The distance determination subunit is configured to determine distances between the data points and each frame of image based on the pose panoramic data.

[0248] The matching subunit is configured to determine an associated panoramic image matching the data points from the at least one panoramic image according to the distances between the data points and each frame of panoramic image.

[0249] In one embodiment of the present application, the first attribute data includes distance data; the distance data indicates the distance between the data point and the center of the coordinate system in which the panoramic image set is located; the first optimization module 1840 may include:

[0250] a depth determination subunit, configured to determine depth data of the pixel point based on the corresponding relationship and the distance data of the data point;

[0251] The first enhancement subunit is configured to perform image enhancement processing on the panoramic image set according to the depth data of the pixel points to obtain the target panoramic image set.

[0252] In one embodiment of the present application, the first attribute data includes intensity data; the intensity data indicates the reflection intensity of the laser at the data point; the first optimization module 1840 may further include:

[0253] A model building subunit, used to build a point cloud strength enhancement model;

[0254] a point cloud intensity enhancement subunit, configured to input the intensity data of the data point into the point cloud intensity enhancement model, perform intensity enhancement processing, and obtain target intensity data of the data point;

[0255] a color updating subunit, configured to update the color data of the pixel point based on the corresponding relationship and the target intensity data of the data point;

[0256] The second enhancement subunit is configured to perform enhancement processing on the panoramic image set according to the updated color data of the pixel points to obtain the target panoramic image set.

[0257] In one embodiment of the present application, the color determination module 1860 may include:

[0258] a boundary data point determination unit, configured to determine boundary data points in the point cloud data according to a collection trajectory of the road measurement information;

[0259] a first associating unit, configured to map the boundary data point to a first associated panoramic image in the panoramic image set, and determine a first pixel point corresponding to the boundary data point in the first associated panoramic image;

[0260] a second associating unit, configured to map the boundary data point to a second associated panoramic image in the panoramic image set, and determine a second pixel point corresponding to the boundary data point in the second associated panoramic image; the first associated panoramic image and the second associated panoramic image being adjacent frame images in the panoramic image set;

[0261] The boundary color determination unit is configured to determine color data of the boundary data point according to color data of the first pixel point and color data of the second pixel point in a case where similarity between the first pixel point and the second pixel point does not satisfy a preset condition.

[0262] In an embodiment of the present application, the color determination module 1860 can further include:

[0263] The first abnormal data point determination unit is configured to determine a first abnormal data point in the point cloud data according to the correspondence and color data of the pixel point; and color data of a pixel point corresponding to the first abnormal data point is in a missing state.

[0264] The first candidate unit is configured to determine a first candidate panoramic image according to an associated panoramic image in the panoramic image set that matches the first abnormal data point; and the first candidate panoramic image and the associated panoramic image are adjacent frame images in the panoramic image set.

[0265] The first candidate pixel point determination unit is configured to determine a first candidate pixel point in the first candidate panoramic image that corresponds to the first abnormal data point.

[0266] The first color determination unit is configured to determine color data of the first abnormal data point according to color data of the first candidate pixel point in a case where the color data of the first candidate pixel point is not in a missing state.

[0267] In an embodiment of the present application, the color determination module 1860 can further include:

[0268] The pixel recognition unit is configured to perform image segmentation and recognition on the panoramic image set to determine category data of the pixel point.

[0269] The second abnormal data point determination unit is configured to determine a second abnormal data point in the point cloud data according to the correspondence and category data of the pixel point; and category data of a pixel point corresponding to the second abnormal data point satisfies a preset limit condition.

[0270] The second candidate unit is configured to determine a second candidate panoramic image according to an associated panoramic image in the panoramic image set that matches the second abnormal data point; and the second candidate panoramic image and the associated panoramic image are adjacent frame images in the panoramic image set.

[0271] The second candidate pixel point determination unit is configured to determine a second candidate pixel point in the second candidate panoramic image that corresponds to the second abnormal data point.

[0272] The second color determination unit is configured to determine the color data of the second abnormal data point according to the color data of the second candidate pixel point in a case where the category data of the second candidate pixel point does not satisfy the preset restriction condition.

[0273] In an embodiment of the present application, the color determination module 1860 can further include:

[0274] The ground grid construction unit is configured to construct a ground triangular grid according to a collection track of the road measurement information.

[0275] The ground distance determination unit is configured to determine a distance from the second abnormal data point to the ground triangular grid.

[0276] The third candidate unit is configured to determine the second candidate panoramic image, the second candidate panoramic image being a previous frame panoramic image of the associated panoramic image.

[0277] The fourth candidate unit is configured to determine the second candidate panoramic image in a case where the distance indicates that the second abnormal data point is not a ground data point, the second candidate panoramic image being a next frame panoramic image of the associated panoramic image.

[0278] It should be noted that the apparatus provided in the above embodiments, when realizing its functions, only takes the above-mentioned division of each functional module as an example, and in actual application, the above-mentioned functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the above-described functions. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be described here.

[0279] The embodiment of the present application provides a computer device, which comprises a processor and a memory, the memory stores at least one instruction or at least one program, the at least one instruction or the at least one program is loaded and executed by the processor to realize the image enhancement method provided in the above method embodiment.

[0280] Figure 20 A hardware structure schematic diagram of a device for realizing the image enhancement method provided in the embodiment of the present application is shown, and the device can participate in constituting or containing the apparatus or system provided in the embodiment of the present application. As shown in the figure, Figure 20As shown, the device 10 may include one or more (illustrated as 1002a, 1002b, ..., 1002n in the figure) processors 1002 (the processor 1002 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 1004 for storing data, and a transmission device 1006 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a power supply and / or a camera. It will be understood by those skilled in the art that Figure 20 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 20 More or fewer components than shown, or with Figure 20 Different configurations shown.

[0281] It should be noted that the one or more processors 1002 and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry". The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuitry may be a single independent processing module, or may be incorporated in whole or in part into any of the other components of the device 10 (or mobile device). As described in the embodiments of the present application, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0282] The memory 1004 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the methods described in the embodiments of the present application. The processor 1002 executes various functional applications and data processing by running the software programs and modules stored in the memory 1004, thereby implementing the above-mentioned image enhancement method. The memory 1004 may include a high-speed random access memory and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 1004 may further include a memory remotely located relative to the processor 1002, and these remote memories may be connected to the device 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0283] The transmission device 1006 is configured to receive or send data via a network. The network can include a wireless network provided by a communication provider of the device 10. In one example, the transmission device 1006 includes a network interface controller (NIC) that can connect to other network devices through a base station to communicate with the Internet. In one example, the transmission device 1006 can be a radio frequency (RF) module that is configured to communicate with the Internet wirelessly.

[0284] The display can be a liquid crystal display (LCD) that is touch screen, for example, which can enable a user to interact with a user interface of the device 10 (or mobile device).

[0285] The embodiment of the present application further provides a computer readable storage medium, which can be arranged in a server to save at least one instruction or at least one program related to an image enhancement method in the method embodiment, and the at least one instruction or the at least one program is loaded and executed by the processor to realize the image enhancement method provided in the method embodiment.

[0286] Optionally, in the embodiment, the storage medium can be located in at least one of a plurality of network servers in a computer network. Optionally, in the embodiment, the storage medium can include but is not limited to a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk and various storage program codes.

[0287] The embodiment of the present application further provides a computer program product or a computer program, which includes computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the image enhancement method provided in the various optional embodiments.

[0288] It should be noted that the above-mentioned order of the embodiments of the present application is only for description, and does not represent the advantages and disadvantages of the embodiments. And the above-mentioned specific embodiments of the present application are described. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from the order in the embodiments and still achieve the desired result. In addition, the processes depicted in the figures do not necessarily require the particular order shown or sequential order to achieve the desired results. In certain implementations, multi-task processing and parallel processing are possible or can be advantageous.

[0289] Each of the embodiments in the present application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments. Especially, for the device, equipment and storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

[0290] A person of ordinary skill in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by hardware, or by program instructing relevant hardware to complete, and the program can be stored in a computer readable storage medium. The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk.

[0291] The above-mentioned is only the preferred embodiment of the present application, and does not limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An image enhancement method characterized by, The method comprises: acquiring point cloud data and a panoramic image set in road measurement information; fusing the point cloud data and the panoramic image set to obtain a fusion result, the fusion result containing a correspondence between data points in the point cloud data and pixel points in the panoramic image set; determining first attribute data of the data points; performing image enhancement processing on the panoramic image set based on the correspondence and the first attribute data of the data points to obtain a target panoramic image set; wherein the first attribute data comprises intensity data; the intensity data indicates the reflection intensity of laser at the data points; the image enhancement processing on the panoramic image set based on the correspondence and the first attribute data of the data points to obtain a target panoramic image set comprises: constructing a point cloud intensity enhancement model; inputting the intensity data of the data points into the point cloud intensity enhancement model to perform intensity enhancement processing to obtain target intensity data of the data points; updating color data of the pixel points based on the correspondence and the target intensity data of the data points; performing enhancement processing on the panoramic image set according to the updated color data of the pixel points to obtain the target panoramic image set.

2. The method of claim 1, wherein, The method further comprises: determining second attribute data of the pixel points, the second attribute data comprising color data; determining color data of the data points based on the correspondence and the color data of the pixel points; generating target point cloud data according to the color data of the data points.

3. The method of claim 1, wherein, The panoramic image set comprises at least one panoramic image, and the fusion of the point cloud data and the panoramic image set to obtain a fusion result comprises: traversing data points in the point cloud data to determine an associated panoramic image matching the data points from the at least one panoramic image; mapping the data points to a coordinate system in which the associated panoramic image is located based on a coordinate system conversion relationship to obtain position data of the data points in the coordinate system; determining pixel points corresponding to the data points in the associated panoramic image according to the position data of the data points in the coordinate system; obtaining a fusion result according to the panoramic image set, the position data of the data points, and the pixel points corresponding to the data points.

4. The method of claim 3, wherein, The traversal of the data points in the point cloud data to determine an associated panoramic image matching the data points from the at least one panoramic image comprises: acquiring pose data of a panoramic camera; the panoramic image set is obtained by the panoramic camera; determining distances between the data points and each panoramic image based on the pose data; determining an associated panoramic image matching the data points from the at least one panoramic image according to the distances between the data points and each panoramic image.

5. The method of claim 1, wherein, The first attribute data comprises distance data; the distance data indicates the distance between the data points and the center of the coordinate system in which the panoramic image set is located; the enhancement processing on the panoramic image set based on the correspondence and the first attribute data of the data points to obtain a target panoramic image set comprises: determine depth data of the pixel point based on the correspondence and distance data of the data point; perform image enhancement processing on the panoramic image set according to the depth data of the pixel point, to obtain the target panoramic image set.

6. The method of claim 2, wherein, The method further comprises: determine boundary data points in the point cloud data according to a collection track of the road measurement information; map the boundary data points to a first associated panoramic image in the panoramic image set to determine first pixel points corresponding to the boundary data points in the first associated panoramic image; map the boundary data points to a second associated panoramic image in the panoramic image set to determine second pixel points corresponding to the boundary data points in the second associated panoramic image; the first associated panoramic image and the second associated panoramic image are adjacent frame images in the panoramic image set; in a case where similarity of the first pixel points and the second pixel points does not satisfy a preset condition, determine color data of the boundary data points according to color data of the first pixel points and color data of the second pixel points.

7. The method of claim 2, wherein, The determining of the color data of the data point based on the correspondence and the color data of the pixel point comprises: determine first abnormal data points in the point cloud data according to the correspondence and color data of the pixel point; color data of pixel points corresponding to the first abnormal data points is in a missing state; determine a first candidate panoramic image according to an associated panoramic image in the panoramic image set matching the first abnormal data point; the first candidate panoramic image and the associated panoramic image are adjacent frame images in the panoramic image set; determine first candidate pixel points corresponding to the first abnormal data points in the first candidate panoramic image; in a case where color data of the first candidate pixel points is not in a missing state, determine color data of the first abnormal data points according to the color data of the first candidate pixel points.

8. The method of claim 2, wherein, The determining of the color data of the data point based on the correspondence and the color data of the pixel point further comprises: perform image segmentation and identification on the panoramic image set to determine category data of the pixel points; determine second abnormal data points in the point cloud data according to the correspondence and category data of the pixel point; category data of pixel points corresponding to the second abnormal data points satisfies a preset limit condition; determine a second candidate panoramic image according to an associated panoramic image in the panoramic image set matching the second abnormal data point; the second candidate panoramic image and the associated panoramic image are adjacent frame images in the panoramic image set; determine second candidate pixel points corresponding to the second abnormal data points in the second candidate panoramic image; in a case where category data of the second candidate pixel points does not satisfy the preset limit condition, determine color data of the second abnormal data points according to color data of the second candidate pixel points.

9. The method of claim 8, wherein, The method further comprises: construct a ground triangular grid according to a collection track of the road measurement information; determine a distance of the second abnormal data point to the ground triangular grid; In a case where the distance indicates that the second abnormal data point is a ground data point, determining the second candidate panoramic image, the second candidate panoramic image being a previous frame panoramic image of the associated panoramic image; In a case where the distance indicates that the second abnormal data point is not a ground data point, determining the second candidate panoramic image, the second candidate panoramic image being a next frame panoramic image of the associated panoramic image.

10. An image enhancement device, characterized by The device comprises: an acquisition module configured to acquire point cloud data and a set of panoramic images in road measurement information; a fusion module configured to fuse the point cloud data and the set of panoramic images to obtain a fusion result, the fusion result containing a correspondence between a data point in the point cloud data and a pixel point in the set of panoramic images; a first attribute information determination module configured to determine first attribute data of the data point; a first optimization module configured to perform image enhancement processing on the set of panoramic images based on the correspondence and the first attribute data of the data point to obtain a target set of panoramic images; wherein the first attribute data includes intensity data, the intensity data indicating a reflection intensity of laser at the data point, and the first optimization module comprises: a model construction subunit configured to construct a point cloud intensity enhancement model; a point cloud intensity enhancement subunit configured to input the intensity data of the data point into the point cloud intensity enhancement model to perform intensity enhancement processing to obtain target intensity data of the data point; a color updating subunit configured to update color data of the pixel point based on the correspondence and the target intensity data of the data point; a second enhancement subunit configured to perform enhancement processing on the set of panoramic images according to the updated color data of the pixel point to obtain the target set of panoramic images.

Images