A multi-dimensional quality score evaluation method and device for multi-modal data

Abstract

The present invention provides a multi-dimensional quality score evaluation method and device for multi-modal data. The steps of the method include: receiving a three-dimensional point cloud image and a two-dimensional RGB image; calculating a noise dimension evaluation parameter for the three-dimensional point cloud image and the two-dimensional RGB image based on the amount of noise data and the amount of non-noise data; calculating an intensity dimension evaluation parameter for the three-dimensional point cloud image and the two-dimensional RGB image based on the average evaluation intensity and the average reference intensity; calculating a gradient magnitude similarity parameter and a total similarity parameter for the two-dimensional RGB image, and calculating a geometric dimension evaluation parameter for the two-dimensional RGB image based on the gradient magnitude similarity parameter and the total similarity parameter; calculating a three-dimensional point cloud image of the three-dimensional point cloud image, and calculating a geometric dimension evaluation parameter for the three-dimensional point cloud image based on the gradient magnitude similarity parameter and the total similarity parameter; calculating the quality scores of the three-dimensional point cloud image and the two-dimensional RGB image respectively based on the geometric dimension evaluation parameter, the intensity dimension evaluation parameter, and the noise dimension evaluation parameter.

Description

Technical Field

[0001] The present invention relates to the technical field of autonomous driving, and in particular, to a multi-dimensional quality score evaluation method and device for multi-modal data. Background Art

[0002] In recent years, autonomous vehicles have been developing rapidly. However, due to the nature of complex and dynamic driving environments, it is not easy to achieve full-range assistance for the driving process of vehicles. Recent breakthroughs in deep learning and computer vision have enabled the rapid development of autonomous driving, which liberates the driver's hands, is expected to reduce traffic congestion, improve road safety, and reduce carbon emissions. However, the potential of autonomous driving has not been fully released, mainly because the perception performance in real-world driving scenarios is not satisfactory. Therefore, autonomous vehicles are equipped with different sensors to ensure robust and accurate environmental perception. Sensor fusion has become an emerging research topic, which uses multiple types of sensors with complementary characteristics to improve the perception ability and reduce costs. Many works combine different complementary sensors and show more obvious performance advantages than single-modal methods. However, neither the three-dimensional machine point cloud generated by LiDAR nor the two-dimensional RGB image captured by a camera can achieve reliable and stable quality in all-weather and full-scene scenarios: LiDAR is restricted by low resolution, low refresh rate, and bad weather conditions; the basic perception of the camera is also difficult under complex or adverse lighting conditions. Considering the real-time requirements of autonomous driving, instead of traditional subjective quality assessment, the objective assessment of machine perception data is a new challenge.

[0003] In terms of three-dimensional point clouds, the vast majority of works are based on human perception of point clouds, such as the object point clouds generated by three-dimensional scanning, to study the subjective feeling scores of humans. A small number of studies on the point clouds generated by lidar focus on the impact of the compressed three-dimensional point clouds on the accuracy of visual tasks. In terms of two-dimensional images, quality assessment research is divided into two fields: subjective quality assessment and objective quality assessment. Subjective quality assessment refers to the scoring of given images by human testers; objective quality assessment refers to calculating the visual quality of images through a certain algorithm. Objective quality assessment can be divided into three categories: full-reference quality assessment, semi-reference quality assessment, and no-reference quality assessment. Among them, in the field of no-reference quality assessment, due to the data differences between three-dimensional point clouds and two-dimensional images, whether traditional machine learning algorithms or deep learning-based models only take a single score as the output, and do not consider evaluating the image quality from all dimensions of the quality model. The prior art lacks a method for evaluating three-dimensional point clouds and two-dimensional images in the same dimension. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a multi-dimensional quality score evaluation method for multi-modal data to eliminate or improve one or more defects existing in the prior art.

[0005] One aspect of the present invention provides a multi - dimensional quality score evaluation method for multi - modal data. The steps of the method include:

[0006] Receiving data images of different modalities collected by multiple sensors of a driving system, where the data images of different modalities include three - dimensional point cloud images and two - dimensional RGB images;

[0007] Comparing the pixel value of each pixel point in the two - dimensional RGB image with a preset pixel threshold to determine the noise pixel points in the two - dimensional RGB image. Taking the number of noise pixel points in the two - dimensional RGB image as the noise data volume, taking the number of non - noise pixel points as the non - noise data volume, and calculating a noise dimension evaluation parameter based on the noise data volume and the non - noise data volume;

[0008] Dividing the three - dimensional point cloud image into multiple search spaces evenly, comparing the number of point clouds in each search space with a preset point cloud number threshold to determine whether each search space is a noise space. Taking the number of point clouds in all noise spaces as the noise data volume, taking the number of point clouds in all non - noise spaces as the non - noise data volume, and calculating a noise dimension evaluation parameter based on the noise data volume and the non - noise data volume;

[0009] Obtaining a two - dimensional RGB reference image of the two - dimensional RGB image, respectively calculating the average values of the pixel values of the pixel points in the two - dimensional RGB image and the two - dimensional RGB reference image. Taking the average value of the pixel values of the two - dimensional RGB image as the evaluation intensity mean value, taking the average value of the pixel values of the two - dimensional RGB reference image as the reference intensity mean value, and calculating an intensity dimension evaluation parameter based on the evaluation intensity mean value and the reference intensity mean value;

[0010] Obtaining a three - dimensional point cloud reference image of the three - dimensional point cloud image, respectively calculating the average values of the reflectivity values of the point clouds in the three - dimensional point cloud image and the three - dimensional point cloud reference image. Taking the average value of the reflectivity values of the three - dimensional point cloud image as the evaluation intensity mean value, taking the average value of the reflectivity values of the three - dimensional point cloud reference image as the reference intensity mean value, and calculating an intensity dimension evaluation parameter based on the evaluation intensity mean value and the reference intensity mean value;

[0011] Taking the partial derivative of each pixel point in the two - dimensional RGB image with adjacent pixel points based on the pixel value to obtain the gradient parameter of each pixel point. Calculating the gradient amplitude similarity parameter and the total similarity parameter of the two - dimensional RGB image based on the gradient parameter of each pixel point, and calculating the geometric dimension evaluation parameter of the two - dimensional RGB image based on the gradient amplitude similarity parameter and the total similarity parameter;

[0012] Derive the partial function of each point cloud in the three-dimensional point cloud image with respect to the adjacent point cloud based on the reflectance value to obtain the gradient parameter of each point cloud. Calculate the gradient magnitude similarity parameter and the total similarity parameter of the three-dimensional point cloud image based on the gradient parameter of each point cloud, and calculate the geometric dimension evaluation parameter of the three-dimensional point cloud image based on the gradient magnitude similarity parameter and the total similarity parameter;

[0013] Calculate the quality scores of the three-dimensional point cloud image and the two-dimensional RGB image respectively based on the geometric dimension evaluation parameter, the intensity dimension evaluation parameter, and the noise dimension evaluation parameter.

[0014] Adopting the above scheme, this scheme quantitatively evaluates the quality of sensor data such as two-dimensional RGB images and three-dimensional laser point clouds, scores the quality of two-dimensional RGB images and three-dimensional point cloud images from the same three characteristic dimensions of noise, intensity, and geometry, and uses the same three dimensions for quality scoring. On the one hand, scoring from multiple dimensions ensures the accuracy and objectivity of the scoring; on the other hand, scoring the two-dimensional RGB image and the three-dimensional point cloud image using the same three characteristic dimensions improves the unity.

[0015] In some embodiments of the present invention, in the step of calculating the gradient magnitude similarity parameter and the total similarity parameter of the two-dimensional RGB image based on the gradient parameter of each pixel point, calculate the mean similarity parameter and the variance similarity parameter of the two-dimensional RGB image based on the gradient parameter of each pixel point, and calculate the total similarity parameter based on the mean similarity parameter and the variance similarity parameter;

[0016] In the step of calculating the gradient magnitude similarity parameter and the total similarity parameter of the three-dimensional point cloud image based on the gradient parameter of each point cloud, calculate the mean similarity parameter and the variance similarity parameter of the three-dimensional point cloud image based on the gradient parameter of each point cloud, and calculate the total similarity parameter based on the mean similarity parameter and the variance similarity parameter.

[0017] In some embodiments of the present invention, in the steps of calculating the mean similarity parameter and the variance similarity parameter of the two-dimensional RGB image based on the gradient parameter of each pixel point and calculating the mean similarity parameter and the variance similarity parameter of the three-dimensional point cloud image based on the gradient parameter of each point cloud, calculate the mean similarity parameter and the variance similarity parameter according to the following formula:

[0018]

[0019]

[0020] where, m d2 represents the average value of the gradient parameters of the pixel points in the two-dimensional RGB image or the average value of the gradient parameters of the point clouds in the three-dimensional point cloud image, m r2Represents the average of the gradient parameters of the pixel points in the two-dimensional RGB reference image or the average of the gradient parameters of the point clouds in the three-dimensional point cloud reference image, s r Represents the variance of the gradient parameters of the pixel points in the two-dimensional RGB reference image or the variance of the gradient parameters of the point clouds in the three-dimensional point cloud reference image, s d Represents the variance of the gradient parameters of the pixel points in the two-dimensional RGB image or the variance of the gradient parameters of the point clouds in the three-dimensional point cloud image. Both C3 and C4 are constants. B represents the mean similarity parameter, and k represents the variance similarity parameter.

[0021] In some embodiments of the present invention, in the steps of calculating the gradient magnitude similarity parameter and the total similarity parameter of the two-dimensional RGB image based on the gradient parameters of each pixel point and calculating the gradient magnitude similarity parameter and the total similarity parameter of the three-dimensional point cloud image based on the gradient parameters of each point cloud, the gradient magnitude similarity parameter and the total similarity parameter are calculated according to the following formulas:

[0022]

[0023] where, g r2 Represents the sum of the gradient parameters of all pixel points in the two-dimensional RGB reference image or the sum of the gradient parameters of all point clouds in the three-dimensional point cloud reference image, g d2 Represents the sum of the gradient parameters of all pixel points in the two-dimensional RGB image or the sum of the gradient parameters of all point clouds in the three-dimensional point cloud image. C5 is a constant, and q2 represents the gradient magnitude similarity parameter;

[0024] t = B * k;

[0025] where, t represents the total similarity parameter, B represents the mean similarity parameter, and k represents the variance similarity parameter.

[0026] In some embodiments of the present invention, in the step of equally dividing the three-dimensional point cloud image into multiple search spaces, the search center point and the spatial geometric parameters are received, and the position and volume of the search space are determined based on the search center point and the spatial geometric parameters.

[0027] In some embodiments of the present invention, in the step of calculating the noise dimension evaluation parameter based on the amount of noise data and the amount of non-noise data, the noise dimension evaluation parameter is calculated according to the following formula:

[0028]

[0029] where, g r1 and g d1 respectively represent the amount of noise data and the amount of non-noise data. C1 is a constant, and q1 represents the noise dimension evaluation parameter.

[0030] In some embodiments of the present invention, an intensity dimension evaluation parameter is calculated based on the average evaluation intensity and the average reference intensity. The intensity dimension evaluation parameter is calculated according to the following formula:

[0031]

[0032] Where m r1 and m d1 respectively represent the average reference intensity and the average evaluation intensity, C2 is a calculation constant, and C represents the intensity dimension evaluation parameter.

[0033] In some embodiments of the present invention, the steps of obtaining the gradient parameter of each pixel point by taking the partial derivative of the partial function of each pixel point in the two-dimensional RGB image with respect to the adjacent pixel points based on the pixel values include:

[0034] Establish the two-dimensional RGB image in a two-dimensional coordinate system, and obtain the pixel values of the adjacent pixel points of each pixel point in the horizontal coordinate growth direction and the pixel values of the adjacent pixel points of each pixel point in the vertical coordinate growth direction;

[0035] Based on the pixel value of the pixel point and the pixel value of the adjacent pixel point of the pixel point in the horizontal coordinate growth direction, calculate the horizontal axis partial derivative value of the pixel point according to the following formula:

[0036] dx(i1,j1) = I(i1 + 1,j1) - I(i1,j1);

[0037] Where dx(i1,j1) represents the horizontal axis partial derivative value of the pixel point with the horizontal coordinate of i1 and the vertical coordinate of j1; I(i1 + 1,j1) represents the pixel value of the pixel point with the horizontal coordinate of i1 + 1 and the vertical coordinate of j1; I(i1,j1) represents the pixel value of the pixel point with the horizontal coordinate of i1 and the vertical coordinate of j1;

[0038] Based on the pixel value of the pixel point and the pixel value of the adjacent pixel point of the pixel point in the vertical coordinate growth direction, calculate the vertical axis partial derivative value of the pixel point according to the following formula:

[0039] dy(i1,j1) = I(i1,j1 + 1) - I(i1,j1);

[0040] Where dy(i1,j1) represents the vertical axis partial derivative value of the pixel point with the horizontal coordinate of i1 and the vertical coordinate of j1; I(i1,j1 + 1) represents the pixel value of the pixel point with the horizontal coordinate of i1 and the vertical coordinate of j1 + 1;

[0041] Based on the horizontal axis partial derivative value and the vertical axis partial derivative value of the pixel point, calculate the gradient parameter of the pixel point according to the following formula:

[0042] G(i1,j1) = dx(i1,j1) + dy(i1,j1);

[0043] Among them, G(i1, j1) represents the gradient parameter of the pixel point with the abscissa i1 and the ordinate j1.

[0044] In some embodiments of the present invention, the step of obtaining the gradient parameter of each point cloud by performing partial function derivation on each point cloud in the three-dimensional point cloud image and the adjacent point cloud based on the reflectivity value includes:

[0045] Based on the reflectivity value of the point cloud and the reflectivity value of the adjacent point cloud in the abscissa growth direction of the point cloud, calculate the horizontal axis partial derivative value of the point cloud according to the following formula:

[0046] dx(i2, j2, h) = I(i2 + 1, j2, h) - I(i2, j2, h);

[0047] Among them, dx(i2, j2, h) represents the horizontal axis partial derivative value of the point cloud with the abscissa i2, the ordinate j2, and the vertical coordinate h; I(i2 + 1, j2, h) represents the reflectivity value of the point cloud with the abscissa i2 + 1, the ordinate j2, and the vertical coordinate h; I(i2, j2, h) represents the reflectivity value of the point cloud with the abscissa i2, the ordinate j2, and the vertical coordinate h;

[0048] Based on the reflectivity value of the point cloud and the reflectivity value of the adjacent point cloud in the ordinate growth direction of the point cloud, calculate the vertical axis partial derivative value of the point cloud according to the following formula:

[0049] dy(i2, j2, h) = I(i2, j2 + 1, h) - I(i2, j2, h);

[0050] Among them, dy(i2, j2, h) represents the vertical axis partial derivative value of the point cloud with the abscissa i2, the ordinate j2, and the vertical coordinate h; I(i2, j2 + 1, h) represents the reflectivity value of the point cloud with the abscissa i2, the ordinate j2 + 1, and the vertical coordinate h;

[0051] Based on the reflectivity value of the point cloud and the reflectivity value of the adjacent point cloud in the vertical coordinate growth direction of the point cloud, calculate the vertical axis partial derivative value of the point cloud according to the following formula:

[0052] dz(i2, j2, h) = I(i2, j2, h + 1) - I(i2, j2, h);

[0053] Among them, dz(i2, j2, h) represents the vertical axis partial derivative value of the point cloud with the abscissa i2, the ordinate j2, and the vertical coordinate h; I(i2, j2, h + 1) represents the reflectivity value of the point cloud with the abscissa i2, the ordinate j2, and the vertical coordinate h + 1;

[0054] Based on the horizontal axis partial derivative value, vertical axis partial derivative value, and vertical axis partial derivative value of the point cloud, calculate the gradient parameter of the point cloud according to the following formula:

[0055] G(i2, j2, h) = dx(i2, j2, h) + dy(i2, j2, h) + dz(i2, j2, h);

[0056] Among them, G(i2, j2, h) represents the gradient parameter of the point cloud with the abscissa i2, ordinate j2, and vertical coordinate h.

[0057] In some embodiments of the present invention, in the step of calculating the geometric dimension evaluation parameter of the two-dimensional RGB image based on the gradient magnitude similarity parameter and the total similarity parameter, calculate the geometric dimension evaluation parameter according to the following formula:

[0058] s = w1t + w2q2;

[0059] Among them, t represents the total similarity parameter, q2 represents the gradient magnitude similarity parameter, w1 and w2 both represent weight parameters, and s represents the geometric dimension evaluation parameter.

[0060] In some embodiments of the present invention, in the step of calculating the quality scores of the three-dimensional point cloud image and the two-dimensional RGB image based on the geometric dimension evaluation parameter, intensity dimension evaluation parameter, and noise dimension evaluation parameter respectively, calculate the quality scores of the three-dimensional point cloud image and the two-dimensional RGB image according to the following formula:

[0061] s’ = w3q1 + w4c + w5s;

[0062] Among them, q1 represents the noise dimension evaluation parameter, c represents the intensity dimension evaluation parameter, s represents the geometric dimension evaluation parameter, and w3, w4, and w5 all represent weight parameters; when q1, c, and s are respectively the noise dimension evaluation parameter, intensity dimension evaluation parameter, and geometric dimension evaluation parameter of the two-dimensional RGB image, s’ represents the quality score of the two-dimensional RGB image; when q1, c, and s are respectively the noise dimension evaluation parameter, intensity dimension evaluation parameter, and geometric dimension evaluation parameter of the three-dimensional point cloud image, s’ represents the quality score of the three-dimensional point cloud image.

[0063] Another aspect of the present invention also provides a multi-dimensional quality score evaluation device for multi-modal data. The device includes a computer device, the computer device includes a processor and a memory, the memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the device implements the steps implemented by the method described above.

[0064] Additional advantages, objects, and features of the present invention will be partly set forth in the description which follows, and will partly become apparent to those of ordinary skill in the art upon examination of the following, or may be learned by practice of the present invention. The objects and other advantages of the present invention may be realized and attained by the means particularly pointed out in the specification and drawings.

[0065] Those skilled in the art will understand that the objects and advantages that can be achieved by the present invention are not limited to those specifically described above, and the above and other objects that the present invention can achieve will be more clearly understood according to the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] The drawings described herein are used to provide a further understanding of the present invention, and constitute a part of this application, and do not limit the present invention.

[0067] Figure 1 It is a schematic diagram of an embodiment of a multi-dimensional quality score evaluation method for multi-modal data of the present invention;

[0068] Figure 2 It is a schematic structural diagram of an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0069] To make the objects, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the embodiments and the drawings. Herein, the schematic embodiments of the present invention and their descriptions are used to explain the present invention, but do not limit the present invention.

[0070] Herein, it should also be noted that in order to avoid obscuring the present invention due to unnecessary details, only the structures and / or processing steps closely related to the solution of the present invention are shown in the drawings, and other details less related to the present invention are omitted.

[0071] It should be emphasized that the term "comprising / including" when used herein refers to the presence of features, elements, steps, or components, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0072] Herein, it should also be noted that if not otherwise specified, the term "connection" in this article can not only refer to direct connection, but also represent indirect connection with an intermediate.

[0073] In the following, embodiments of the present invention will be described with reference to the drawings. In the drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0074] To solve the above problems, as Figure 1 shown, the present invention proposes a multi-dimensional quality score evaluation method for multi-modal data; the steps of the method include:

[0075] Step S100: Receive data images of different modalities collected by various sensors of the driving system. The data images of different modalities include three-dimensional point cloud images and two-dimensional RGB images;

[0076] In some embodiments of the present invention, the three-dimensional point cloud image can be a three-dimensional point cloud image collected by a millimeter-wave radar or a lidar, and the two-dimensional RGB image can be a two-dimensional RGB image collected by a camera.

[0077] Step S210: Compare the pixel value of each pixel point in the two-dimensional RGB image with a preset pixel threshold to determine the noise pixel points in the two-dimensional RGB image. Use the number of noise pixel points in the two-dimensional RGB image as the noise data volume, and use the number of non-noise pixel points as the non-noise data volume. Calculate the noise dimension evaluation parameter based on the noise data volume and the non-noise data volume;

[0078] In some embodiments of the present invention, the pixel threshold is a preset pixel value range. When the pixel value of a pixel point does not belong to the range of the pixel threshold, it is determined that the pixel point is a noise pixel point. Determine whether each pixel point in the two-dimensional RGB image is a noise pixel point, record the number of noise pixel points and non-noise pixel points respectively. Use the total number of noise pixel points in the two-dimensional RGB image as the noise data volume of the two-dimensional RGB image, and use the total number of non-noise pixel points in the two-dimensional RGB image as the non-noise data volume of the two-dimensional RGB image.

[0079] In the two-dimensional RGB image data, each pixel point includes three color channels, with a size range of 0 to 255. The closer to 0, the darker the channel, and the closer to 255, the brighter the channel. In this solution, the channel average value of the three color channels of each pixel point is used as the pixel value of this point.

[0080] Step S310: Divide the three-dimensional point cloud image into multiple search spaces on average. Compare the number of point clouds in each search space with a preset point cloud number threshold to determine whether each search space is a noise space. Use the number of point clouds in all noise spaces as the noise data volume, and use the number of point clouds in all non-noise spaces as the non-noise data volume. Calculate the noise dimension evaluation parameter based on the noise data volume and the non-noise data volume;

[0081] In some embodiments of the present invention, in the step of dividing the three-dimensional point cloud image into multiple search spaces on average, receive the search center point and spatial geometric parameters, and determine the position and volume of the search space based on the search center point and spatial geometric parameters.

[0082] In some embodiments of the present invention, the spatial geometric parameter can be the search space radius or the search space side length, etc. The search space can be a sphere or a cube,

[0083] If the search space is a sphere, with the search center point as the center of the sphere and the search space radius as the radius of the sphere, a search space of the sphere is constructed.

[0084] If the search space is a cube, with the search center point as the center of the cube and the search space side length as the side length of the cube, a search space of the cube is constructed.

[0085] In some embodiments of the present invention, the point cloud quantity threshold is an interval of the number of point clouds. When the number of point clouds in the search space does not belong to the interval range of the point cloud quantity threshold, it is determined that the search space is a noise space, and the parameter of the number of point clouds in the search space is the parameter of the noise data quantity; when the number of point clouds in the search space belongs to the interval range of the point cloud quantity threshold, it is determined that the search space is a non-noise space, and the parameter of the number of point clouds in the search space is the parameter of the non-noise data quantity. The sum of the number of point clouds in all non-noise spaces is used to obtain the parameter of the non-noise data quantity of the three-dimensional point cloud image, and the sum of the number of point clouds in all noise spaces is used to obtain the parameter of the noise data quantity of the three-dimensional point cloud image.

[0086] Step S220: Obtain the two-dimensional RGB reference image of the two-dimensional RGB image, and calculate the average value of the pixel values of the pixel points of the two-dimensional RGB image and the two-dimensional RGB reference image respectively. Use the average value of the pixel values of the two-dimensional RGB image as the evaluation intensity mean value, and use the average value of the pixel values of the two-dimensional RGB reference image as the reference intensity mean value. Calculate the intensity dimension evaluation parameter based on the evaluation intensity mean value and the reference intensity mean value.

[0087] The two-dimensional RGB reference image can be an image matched to the two-dimensional RGB image in the image library by using image recognition technology, or an image manually selected in the image library.

[0088] Step S320: Obtain the three-dimensional point cloud reference image of the three-dimensional point cloud image, and calculate the average value of the reflectivity values of the point clouds of the three-dimensional point cloud image and the three-dimensional point cloud reference image respectively. Use the average value of the reflectivity values of the three-dimensional point cloud image as the evaluation intensity mean value, and use the average value of the reflectivity values of the three-dimensional point cloud reference image as the reference intensity mean value. Calculate the intensity dimension evaluation parameter based on the evaluation intensity mean value and the reference intensity mean value.

[0089] The three-dimensional point cloud reference image can be an image matched to the three-dimensional point cloud image in the image library by using image recognition technology, or an image manually selected in the image library.

[0090] Step S230: Derive the partial function of each pixel point in the two-dimensional RGB image with respect to its adjacent pixel points based on the pixel values to obtain the gradient parameter of each pixel point. Calculate the gradient magnitude similarity parameter and the total similarity parameter of the two-dimensional RGB image based on the gradient parameter of each pixel point, and calculate the geometric dimension evaluation parameter of the two-dimensional RGB image based on the gradient magnitude similarity parameter and the total similarity parameter.

[0091] Step S330: Derive the partial function of each point cloud in the three-dimensional point cloud image with respect to its adjacent point cloud based on the reflectivity value to obtain the gradient parameter of each point cloud. Calculate the gradient magnitude similarity parameter and the total similarity parameter of the three-dimensional point cloud image based on the gradient parameter of each point cloud, and calculate the geometric dimension evaluation parameter of the three-dimensional point cloud image based on the gradient magnitude similarity parameter and the total similarity parameter.

[0092] In terms of geometric dimension, this solution selects the gradient parameter as the index to characterize the geometric features. The gradient parameter reflects the clarity of the object's edge boundary. The higher the gradient parameter, the more obvious the object's boundary contour and the better the geometric features.

[0093] Step S400: Calculate the quality scores of the three-dimensional point cloud image and the two-dimensional RGB image respectively based on the geometric dimension evaluation parameter, the intensity dimension evaluation parameter, and the noise dimension evaluation parameter.

[0094] Adopting the above solution, this solution quantifies and evaluates the quality of sensor data such as two-dimensional RGB images and three-dimensional laser point clouds. The quality of the two-dimensional RGB image and the three-dimensional point cloud image is scored from the same three characteristic dimensions of noise, intensity, and geometry. Using the same three dimensions for quality scoring, on the one hand, scoring from multiple dimensions ensures the accuracy and objectivity of the scoring; on the other hand, using the same three characteristic dimensions for scoring the two-dimensional RGB image and the three-dimensional point cloud image improves the unity.

[0095] This solution evaluates the quality of the three-dimensional point cloud and the two-dimensional image through three dimensions. During the experiment, the trained evaluation model is put into other unmanned driving scenario tasks for verification. It can be observed that the score quality model obtained in a specific scenario still performs well in other scenarios, which proves that our data quality score model has robustness and generalization.

[0096] In current point cloud quality assessment research, evaluating existing point clouds requires a reference point cloud as the basis for calculating various indicators. However, the characteristics of real-time variability in autonomous driving problems mean that it is almost impossible to obtain a reference point cloud at each moment. By mining the geometric, noise, intensity, resolution, and other characteristics of the three-dimensional point cloud itself, the present invention can evaluate the quality of three-dimensional point clouds without relying on a reference point cloud, achieving quality assessment under reference-free conditions, which will greatly expand the application scenarios of point cloud quality assessment. Two-dimensional images and three-dimensional point clouds belong to different modal data, with obvious characteristic differences in their data structures and geometric distributions. Previous research usually only studied quality assessment methods within their respective modalities. The present invention identifies common key features between two-dimensional images and three-dimensional point clouds, uses these as dimensions for characterizing data quality, and establishes a cross-modal data quality assessment model to address the potential impact of sensor data quality fluctuations in the real world.

[0097] In some embodiments of the present invention, in the step of calculating the gradient magnitude similarity parameter and the total similarity parameter of the two-dimensional RGB image based on the gradient parameter of each pixel point, the mean similarity parameter and the variance similarity parameter of the two-dimensional RGB image are calculated based on the gradient parameter of each pixel point, and the total similarity parameter is calculated based on the mean similarity parameter and the variance similarity parameter;

[0098] In the step of calculating the gradient magnitude similarity parameter and the total similarity parameter of the three-dimensional point cloud image based on the gradient parameter of each point cloud, the mean similarity parameter and the variance similarity parameter of the three-dimensional point cloud image are calculated based on the gradient parameter of each point cloud, and the total similarity parameter is calculated based on the mean similarity parameter and the variance similarity parameter.

[0099] In some embodiments of the present invention, in the steps of calculating the mean similarity parameter and the variance similarity parameter of the two-dimensional RGB image based on the gradient parameter of each pixel point and calculating the mean similarity parameter and the variance similarity parameter of the three-dimensional point cloud image based on the gradient parameter of each point cloud, the mean similarity parameter and the variance similarity parameter are calculated according to the following formula:

[0100]

[0101]

[0102] where m d2 represents the average of the gradient parameters of the pixel points in the two-dimensional RGB image or the average of the gradient parameters of the point clouds in the three-dimensional point cloud image, and m r2 represents the average of the gradient parameters of the pixel points in the two-dimensional RGB reference image or the average of the gradient parameters of the point clouds in the three-dimensional point cloud reference image, and s rRepresents the variance of the gradient parameters of pixel points in a two-dimensional RGB reference image or the variance of the gradient parameters of point clouds in a three-dimensional point cloud reference image, s d Represents the variance of the gradient parameters of pixel points in a two-dimensional RGB image or the variance of the gradient parameters of point clouds in a three-dimensional point cloud image. Both C3 and C4 are constants, B represents the mean similarity parameter, and k represents the variance similarity parameter.

[0103] In some embodiments of the present invention, when calculating the mean similarity parameter or the variance similarity parameter of a two-dimensional RGB image, B represents the mean similarity parameter of the two-dimensional RGB image, k represents the variance similarity parameter of the two-dimensional RGB image, m d2 Represents the average value of the gradient parameters of pixel points in a two-dimensional RGB image, m r2 Represents the average value of the gradient parameters of pixel points in a two-dimensional RGB reference image, s r Represents the variance of the gradient parameters of pixel points in a two-dimensional RGB reference image, s d Represents the variance of the gradient parameters of pixel points in a two-dimensional RGB image.

[0104] In some embodiments of the present invention, when calculating the mean similarity parameter or the variance similarity parameter of a three-dimensional point cloud image, B represents the mean similarity parameter of the three-dimensional point cloud image, k represents the variance similarity parameter of the three-dimensional point cloud image, m d2 Represents the average value of the gradient parameters of point clouds in a three-dimensional point cloud image, m r2 Represents the average value of the gradient parameters of point clouds in a three-dimensional point cloud reference image, s r Represents the variance of the gradient parameters of point clouds in a three-dimensional point cloud reference image, s d Represents the variance of the gradient parameters of point clouds in a three-dimensional point cloud image.

[0105] The average value or variance of the gradient parameters of the pixel points is the average value or variance of the gradient parameters of all pixel points in the two-dimensional RGB image or the two-dimensional RGB reference image; the average value or variance of the gradient parameters of the point clouds is the average value or variance of the gradient parameters of all point clouds in the three-dimensional point cloud image or the three-dimensional point cloud reference image.

[0106] Adopting the above scheme, the two-dimensional RGB image and the three-dimensional point cloud image use the same formula to calculate the mean similarity parameter and the variance similarity parameter, unifying the calculation method and ensuring the objectivity of the calculation.

[0107] In some embodiments of the present invention, in the steps of calculating the gradient magnitude similarity parameter and the total similarity parameter of the two-dimensional RGB image based on the gradient parameters of each pixel point and calculating the gradient magnitude similarity parameter and the total similarity parameter of the three-dimensional point cloud image based on the gradient parameters of each point cloud, the gradient magnitude similarity parameter and the total similarity parameter are calculated according to the following formula:

[0108]

[0109] wherein, g r2 represents the sum of the gradient parameters of all pixel points in the two-dimensional RGB reference image or the sum of the gradient parameters of all point clouds in the three-dimensional point cloud reference image, and g d2 represents the sum of the gradient parameters of all pixel points in the two-dimensional RGB image or the sum of the gradient parameters of all point clouds in the three-dimensional point cloud image, C5 is a constant, and q2 represents the gradient magnitude similarity parameter;

[0110] t = B * k;

[0111] wherein, t represents the total similarity parameter, B represents the mean similarity parameter, and k represents the variance similarity parameter.

[0112] In some embodiments of the present invention, when calculating the gradient magnitude similarity parameter of the two-dimensional RGB image, g r2 represents the sum of the gradient parameters of all pixel points in the two-dimensional RGB reference image, and g d2 represents the sum of the gradient parameters of all pixel points in the two-dimensional RGB image; when calculating the gradient magnitude similarity parameter of the three-dimensional point cloud image, g r2 represents the sum of the gradient parameters of all point clouds in the three-dimensional point cloud reference image, and g d2 represents the sum of the gradient parameters of all point clouds in the three-dimensional point cloud image.

[0113] In some embodiments of the present invention, when calculating the total similarity parameter of the two-dimensional RGB image, B represents the mean similarity parameter of the two-dimensional RGB image, and k represents the variance similarity parameter of the two-dimensional RGB image; when calculating the total similarity parameter of the three-dimensional point cloud image, B represents the mean similarity parameter of the three-dimensional point cloud image, and k represents the variance similarity parameter of the three-dimensional point cloud image.

[0114] In some embodiments of the present invention, in the step of calculating the noise dimension evaluation parameter based on the noise data amount and the non-noise data amount, the noise dimension evaluation parameter is calculated according to the following formula:

[0115]

[0116] wherein, g r1 and g d1 respectively represent the noise data amount and the non-noise data amount, C1 is a constant, and q1 represents the noise dimension evaluation parameter.

[0117] In some embodiments of the present invention, when calculating the noise dimension evaluation parameter of the two-dimensional RGB image, g r1 and g d1respectively represent the noise data volume and the non-noise data volume of the two-dimensional RGB image; when calculating the noise dimension evaluation parameter of the three-dimensional point cloud image, g r1 and g d1 respectively represent the noise data volume and the non-noise data volume of the three-dimensional point cloud image.

[0118] In some embodiments of the present invention, the intensity dimension evaluation parameter is calculated based on the average evaluation intensity and the average reference intensity, and the intensity dimension evaluation parameter is calculated according to the following formula:

[0119]

[0120] where, m r1 and m d1 respectively represent the average reference intensity and the average evaluation intensity, C2 is a calculation constant, and c represents the intensity dimension evaluation parameter.

[0121] In some embodiments of the present invention, when calculating the intensity dimension evaluation parameter of the two-dimensional RGB image, m r1 and m d1 respectively represent the average reference intensity and the average evaluation intensity of the two-dimensional RGB image; when calculating the intensity dimension evaluation parameter of the three-dimensional point cloud image, m r1 and m d1 respectively represent the average reference intensity and the average evaluation intensity of the three-dimensional point cloud image.

[0122] In some embodiments of the present invention, the step of obtaining the gradient parameter of each pixel point by taking the partial derivative of the function based on the pixel value between each pixel point in the two-dimensional RGB image and the adjacent pixel points includes:

[0123] Establish the two-dimensional RGB image in a two-dimensional coordinate system, and obtain the pixel values of the adjacent pixel points of each pixel point in the horizontal coordinate growth direction and the pixel values of the adjacent pixel points of each pixel point in the vertical coordinate growth direction;

[0124] Based on the pixel value of the pixel point and the pixel value of the adjacent pixel point of this pixel point in the horizontal coordinate growth direction, calculate the horizontal axis partial derivative value of this pixel point according to the following formula:

[0125] dx(i1, j1) = I(i1 + 1, j1) - I(i1, j1);

[0126] where, dx(i1, j1) represents the horizontal axis partial derivative value of the pixel point with the horizontal coordinate of i1 and the vertical coordinate of j1; I(i1 + 1, j1) represents the pixel value of the pixel point with the horizontal coordinate of i1 + 1 and the vertical coordinate of j1; I(i1, j1) represents the pixel value of the pixel point with the horizontal coordinate of i1 and the vertical coordinate of j1;

[0127] Based on the pixel value of a pixel and the pixel values of the adjacent pixels of this pixel in the vertical coordinate growth direction, calculate the vertical axis partial derivative value of this pixel according to the following formula:

[0128] dy(i1, j1) = I(i1, j1 + 1) - I(i1, j1);

[0129] Where, dy(i1, j1) represents the vertical axis partial derivative value of the pixel with the abscissa i1 and the ordinate j1; I(i1, j1 + 1) represents the pixel value of the pixel with the abscissa i1 and the ordinate j1 + 1;

[0130] Based on the horizontal axis partial derivative value and the vertical axis partial derivative value of a pixel, calculate the gradient parameter of the pixel according to the following formula:

[0131] G(i1, j1) = dx(i1, j1) + dy(i1, j1);

[0132] Where, G(i1, j1) represents the gradient parameter of the pixel with the abscissa i1 and the ordinate j1.

[0133] In some embodiments of the present invention, the step of performing partial function derivation on each point cloud in the three-dimensional point cloud image and the adjacent point clouds based on the reflectivity value to obtain the gradient parameter of each point cloud includes:

[0134] Based on the reflectivity value of a point cloud and the reflectivity values of the adjacent point clouds of this point cloud in the horizontal coordinate growth direction, calculate the horizontal axis partial derivative value of this point cloud according to the following formula:

[0135] dx(i2, j2, h) = I(i2 + 1, j2, h) - I(i2, j2, h);

[0136] Where, dx(i2, j2, h) represents the horizontal axis partial derivative value of the point cloud with the abscissa i2, the ordinate j2, and the vertical coordinate h; I(i2 + 1, j2, h) represents the reflectivity value of the point cloud with the abscissa i2 + 1, the ordinate j2, and the vertical coordinate h; I(i2, j2, h) represents the reflectivity value of the point cloud with the abscissa i2, the ordinate j2, and the vertical coordinate h;

[0137] Based on the reflectivity value of a point cloud and the reflectivity values of the adjacent point clouds of this point cloud in the vertical coordinate growth direction, calculate the vertical axis partial derivative value of this point cloud according to the following formula:

[0138] dy(i2, j2, h) = I(i2, j2 + 1, h) - I(i2, j2, h);

[0139] Among them, dy(i2, j2, h) represents the partial derivative value of the vertical axis of the point cloud with the abscissa i2, ordinate j2, and vertical coordinate h; I(i2, j2 + 1, h) represents the reflectivity value of the point cloud with the abscissa i2, ordinate j2 + 1, and vertical coordinate h.

[0140] Based on the reflectivity value of the point cloud and the reflectivity values of the adjacent point clouds in the growth direction of the vertical coordinate of this point cloud, according to the following formula, calculate the partial derivative value of the vertical axis of this point cloud:

[0141] dz(i2, j2, h) = I(i2, j2, h + 1) - I(i2, j2, h);

[0142] Among them, dz(i2, j2, h) represents the partial derivative value of the vertical axis of the point cloud with the abscissa i2, ordinate j2, and vertical coordinate h; I(i2, j2, h + 1) represents the reflectivity value of the point cloud with the abscissa i2, ordinate j2, and vertical coordinate h + 1.

[0143] Based on the partial derivative value of the horizontal axis, the partial derivative value of the vertical axis, and the partial derivative value of the vertical axis of the point cloud, according to the following formula, calculate the gradient parameter of this point cloud:

[0144] G(i2, j2, h) = dx(i2, j2, h) + dy(i2, j2, h) + dz(i2, j2, h);

[0145] Among them, G(i2, j2, h) represents the gradient parameter of the point cloud with the abscissa i2, ordinate j2, and vertical coordinate h.

[0146] In some embodiments of the present invention, in the step of calculating the geometric dimension evaluation parameter of the two-dimensional RGB image based on the gradient magnitude similarity parameter and the total similarity parameter, calculate the geometric dimension evaluation parameter according to the following formula:

[0147] s = w1t + w2q2;

[0148] Among them, t represents the total similarity parameter, q2 represents the gradient magnitude similarity parameter, both w1 and w2 represent weight parameters, and s represents the geometric dimension evaluation parameter.

[0149] In some embodiments of the present invention, when calculating the geometric dimension evaluation parameter of the three-dimensional point cloud image, q2 represents the gradient magnitude similarity parameter of the three-dimensional point cloud image, and t represents the total similarity parameter of the three-dimensional point cloud image; when calculating the geometric dimension evaluation parameter of the two-dimensional RGB image, q2 represents the gradient magnitude similarity parameter of the two-dimensional RGB reference image, and t represents the total similarity parameter of the two-dimensional RGB reference image.

[0150] In some embodiments of the present invention, in the step of calculating the quality scores of the three-dimensional point cloud image and the two-dimensional RGB image based on the geometric dimension evaluation parameter, the intensity dimension evaluation parameter, and the noise dimension evaluation parameter respectively, the quality scores of the three-dimensional point cloud image and the two-dimensional RGB image are calculated based on the following formula:

[0151] s’ = w3ql + w4c + w5s;

[0152] Wherein, q1 represents the noise dimension evaluation parameter, c represents the intensity dimension evaluation parameter, s represents the geometric dimension evaluation parameter, and w3, w4, and w5 all represent weight parameters; when q1, c, and s are respectively the noise dimension evaluation parameter, the intensity dimension evaluation parameter, and the geometric dimension evaluation parameter of the two-dimensional RGB image, s’ represents the quality score of the two-dimensional RGB image; when q1, c, and s are respectively the noise dimension evaluation parameter, the intensity dimension evaluation parameter, and the geometric dimension evaluation parameter of the three-dimensional point cloud image, s’ represents the quality score of the three-dimensional point cloud image.

[0153] Although images and point clouds belong to different modalities, they have common key features, and these key features will have an important impact on the target recognition accuracy in autonomous driving. For example Figure 2 As shown, this solution proposes a data quality model containing three dimensions, including: noise feature, intensity feature, and gradient feature. Starting from these three dimensions, comprehensive analysis of heterogeneous data is realized, and a unified quantitative evaluation score is provided for the data quality of different modalities.

[0154] The present invention can determine the quality scores of different modality data based on the same dimension to cope with a series of situations such as bad weather, unfamiliar environments, and even sensor degradation or failure, providing help for the stable operation of autonomous driving vehicles. Considering the real-time, complexity, and dynamics of autonomous driving, as well as coping with possible bad weather or sensor quality fluctuations, etc., in order to provide robust and reliable performance for the autonomous driving system, it is necessary to make real-time data quality evaluations on the data collected by lidar and two-dimensional RGB cameras at each moment, and finally output the quality scores under a unified standard, so as to truly realize the quantitative evaluation of sensor data.

[0155] The beneficial effects of this solution include:

[0156] 1. Unified dimension characterization of heterogeneous data. There are various differences between images and point clouds: First, the data storage methods are different. Three-dimensional point cloud data is structured data, while RGB images belong to unstructured data; second, the characterized features are different. Three-dimensional point clouds characterize the characteristics of objects in three-dimensional space, while RGB images describe the distribution in two-dimensional space at a specific moment. Based on the common key features inside these two types of data, this solution uses them as the model dimensions for characterizing data quality.

[0157] 2. The quality of sensor data represents the ability of the sensor to depict and restore the real world at a given moment. The higher the data quality, the more accurate the sensor's description of the environment, and vice versa. There is no doubt that the quality of sensor data has an important impact on the target recognition task of data fusion. However, it is still unknown how much the data quality affects data fusion. This solution converts this qualitative relationship into a quantitative relationship to better serve the autonomous driving scenario.

[0158] 3. In the autonomous driving environment, the perception system usually includes more than just cameras and lidar. Currently, in the industrial and academic fields, a series of sensors such as millimeter-wave radars and infrared radars have also begun to receive more and more attention. This solution can select the one with the highest recognition accuracy among various data fusion schemes under multi-sensor conditions based on the quality score.

[0159] 4. This solution establishes a multi-dimensional data quality evaluation model and uses this model to evaluate the quality of multi-modal data;

[0160] 5. This invention reveals the evaluability of multi-modal sensor data. With the introduction of multiple evaluation dimensions, the noise characteristics, intensity characteristics, and geometric characteristics of multi-modal data will be fully evaluated, and a robust data quality evaluation method is proposed accordingly.

[0161] 6. This invention has extremely high generalization and can be applied to common sensors in the current field of driverless driving. It can be used in many driverless tasks based on sensor data fusion, such as target recognition and semantic segmentation.

[0162] An embodiment of this invention also provides a multi-dimensional quality score evaluation device for multi-modal data. The device includes a computer device, the computer device includes a processor and a memory, the memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the device implements the steps implemented by the method described above.

[0163] An embodiment of this invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps implemented by the foregoing multi-dimensional quality score evaluation method for multi-modal data. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, floppy disk, hard disk, removable storage disk, CD-ROM, or any other form of storage medium well-known in the technical field.

[0164] Those of ordinary skill in the art should understand that the various exemplary components, systems, and methods described in connection with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Specifically, whether to implement in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application-specific integrated circuit (ASIC), appropriate firmware, a plug-in, a functional card, and so on. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium or transmitted via a data signal carried in a carrier wave on a transmission medium or a communication link.

[0165] It should be clear that the present invention is not limited to the specific configurations and processes described above and illustrated in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and illustrated as examples. However, the method process of the present invention is not limited to the specific steps described and illustrated. Those skilled in the art can make various changes, modifications, and additions, or change the order between steps after understanding the spirit of the present invention.

[0166] In the present invention, the features described and / or exemplified for one embodiment can be used in the same or a similar manner in one or more other embodiments, and / or combined with the features of other embodiments or replace the features of other embodiments.

[0167] The above are only the preferred embodiments of the present invention and are not used to limit the present invention. For those skilled in the art, various changes and variations can be made to the embodiments of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A multi-dimensional quality score evaluation method for multi-modal data, characterized in that, The steps of the method include: Receiving data images of different modalities collected by various sensors of the driving system, where the data images of different modalities include three-dimensional point cloud images and two-dimensional RGB images; Comparing the pixel value of each pixel point in the two-dimensional RGB image with a preset pixel threshold to determine the noise pixel points in the two-dimensional RGB image, taking the number of noise pixel points in the two-dimensional RGB image as the noise data volume, taking the number of non-noise pixel points as the non-noise data volume, and calculating a noise dimension evaluation parameter based on the noise data volume and the non-noise data volume; Dividing the three-dimensional point cloud image into multiple search spaces evenly, comparing the number of point clouds in each search space with a preset point cloud number threshold to determine whether each search space is a noise space, taking the number of point clouds in all noise spaces as the noise data volume, taking the number of point clouds in all non-noise spaces as the non-noise data volume, and calculating a noise dimension evaluation parameter based on the noise data volume and the non-noise data volume; Obtaining a two-dimensional RGB reference image of the two-dimensional RGB image, calculating the average value of the pixel values of the pixel points of the two-dimensional RGB image and the two-dimensional RGB reference image respectively, taking the average value of the pixel values of the two-dimensional RGB image as the evaluation intensity mean value, taking the average value of the pixel values of the two-dimensional RGB reference image as the reference intensity mean value, and calculating an intensity dimension evaluation parameter based on the evaluation intensity mean value and the reference intensity mean value; Obtaining a three-dimensional point cloud reference image of the three-dimensional point cloud image, calculating the average value of the reflectivity values of the point clouds of the three-dimensional point cloud image and the three-dimensional point cloud reference image respectively, taking the average value of the reflectivity values of the three-dimensional point cloud image as the evaluation intensity mean value, taking the average value of the reflectivity values of the three-dimensional point cloud reference image as the reference intensity mean value, and calculating an intensity dimension evaluation parameter based on the evaluation intensity mean value and the reference intensity mean value; Taking the partial derivative of each pixel point in the two-dimensional RGB image with an adjacent pixel point based on the pixel value to obtain the gradient parameter of each pixel point, calculating the gradient amplitude similarity parameter and the total similarity parameter of the two-dimensional RGB image based on the gradient parameter of each pixel point, and calculating the geometric dimension evaluation parameter of the two-dimensional RGB image based on the gradient amplitude similarity parameter and the total similarity parameter; Taking the partial derivative of each point cloud in the three-dimensional point cloud image with an adjacent point cloud based on the reflectivity value to obtain the gradient parameter of each point cloud, calculating the gradient amplitude similarity parameter and the total similarity parameter of the three-dimensional point cloud image based on the gradient parameter of each point cloud, and calculating the geometric dimension evaluation parameter of the three-dimensional point cloud image based on the gradient amplitude similarity parameter and the total similarity parameter; Calculating the quality scores of the three-dimensional point cloud image and the two-dimensional RGB image respectively based on the geometric dimension evaluation parameter, the intensity dimension evaluation parameter, and the noise dimension evaluation parameter.

2. The multi-dimensional quality score evaluation method for multi-modal data according to claim 1, wherein In the step of calculating the gradient amplitude similarity parameter and the total similarity parameter of the two-dimensional RGB image based on the gradient parameter of each pixel point, calculating the mean similarity parameter and the variance similarity parameter of the two-dimensional RGB image based on the gradient parameter of each pixel point, and calculating the total similarity parameter based on the mean similarity parameter and the variance similarity parameter; In the step of calculating the gradient magnitude similarity parameter and the total similarity parameter of the 3D point cloud image based on the gradient parameters of each point cloud, the mean similarity parameter and the variance similarity parameter of the 3D point cloud image are calculated based on the gradient parameters of each point cloud, and the total similarity parameter is calculated based on the mean similarity parameter and the variance similarity parameter.

3. The multi-dimensional quality score evaluation method for multi-modal data according to claim 2, wherein In the steps of calculating the mean similarity parameter and the variance similarity parameter of the 2D RGB image based on the gradient parameters of each pixel point and calculating the mean similarity parameter and the variance similarity parameter of the 3D point cloud image based on the gradient parameters of each point cloud, the mean similarity parameter and the variance similarity parameter are calculated according to the following formula: where m d2 represents the average value of the gradient parameters of the pixel points in the two-dimensional RGB image or the average value of the gradient parameters of the point cloud in the three-dimensional point cloud image, and m r2 represents the average value of the gradient parameters of the pixel points in the two-dimensional RGB reference image or the average value of the gradient parameters of the point cloud in the three-dimensional point cloud reference image, s r represents the variance of the gradient parameters of the pixel points in the two-dimensional RGB reference image or the variance of the gradient parameters of the point cloud in the three-dimensional point cloud reference image, s d represents the variance of the gradient parameters of the pixel points in the two-dimensional RGB image or the variance of the gradient parameters of the point cloud in the three-dimensional point cloud image. Both C3 and C4 are constants, B represents the mean similarity parameter, and k represents the variance similarity parameter.

4. The multi-dimensional quality score evaluation method for multi-modal data according to claim 2 or 3, characterized in that In the steps of calculating the gradient magnitude similarity parameter and the total similarity parameter of the 2D RGB image based on the gradient parameters of each pixel point and calculating the gradient magnitude similarity parameter and the total similarity parameter of the 3D point cloud image based on the gradient parameters of each point cloud, the gradient magnitude similarity parameter and the total similarity parameter are calculated according to the following formula: where, g r2 represents the sum of the gradient parameters of all pixel points in the two-dimensional RGB reference image or the sum of the gradient parameters of all point clouds in the three-dimensional point cloud reference image, and g d2 represents the sum of the gradient parameters of all pixel points in the two-dimensional RGB image or the sum of the gradient parameters of all point clouds in the three-dimensional point cloud image, C5 is a constant, and q2 represents the gradient magnitude similarity parameter; t = B * k; Wherein, t represents the total similarity parameter, B represents the mean similarity parameter, and k represents the variance similarity parameter.

5. The multi-dimensional quality score evaluation method for multi-modal data according to claim 1, wherein In the step of calculating the noise dimension evaluation parameter based on the amount of noise data and the amount of non-noise data, the noise dimension evaluation parameter is calculated according to the following formula: Among them, g r1 and g d1 respectively represent the amount of noise data and the amount of non-noise data, C1 is a constant, and q1 represents the noise dimension evaluation parameter.

6. The multi-dimensional quality score evaluation method for multi-modal data according to claim 1, characterized in that Calculate the intensity dimension evaluation parameter based on the mean evaluation intensity and the mean reference intensity. The intensity dimension evaluation parameter is calculated according to the following formula: where m r1 and m d1 represent the reference intensity mean and the evaluation intensity mean respectively, C2 is a calculation constant, and c represents the intensity dimension evaluation parameter.

7. The multi-dimensional quality score evaluation method for multi-modal data according to claim 1, characterized in that The steps of obtaining the gradient parameter of each pixel point by performing partial function derivation on each pixel point in the 2D RGB image and the adjacent pixel points based on the pixel values include: Establish the 2D RGB image in a 2D coordinate system, and obtain the pixel values of the adjacent pixel points of each pixel point in the horizontal coordinate increasing direction and the pixel values of the adjacent pixel points of each pixel point in the vertical coordinate increasing direction; Based on the pixel value of the pixel point and the pixel value of the adjacent pixel point of this pixel point in the horizontal coordinate increasing direction, calculate the horizontal axis partial derivative value of this pixel point according to the following formula: dx(i1,j1) = I(i1 + 1,j1) - I(i1,j1); Wherein, dx(i1,j1) represents the horizontal axis partial derivative value of the pixel point with the horizontal coordinate i1 and the vertical coordinate j1; I(i1 + 1,j1) represents the pixel value of the pixel point with the horizontal coordinate i1 + 1 and the vertical coordinate j1; I(i1,j1) represents the pixel value of the pixel point with the horizontal coordinate i1 and the vertical coordinate j1; Based on the pixel value of the pixel point and the pixel value of the adjacent pixel point of this pixel point in the vertical coordinate increasing direction, calculate the vertical axis partial derivative value of this pixel point according to the following formula: dy(i1,j1) = I(i1,j1 + 1) - I(i1,j1); Wherein, dy(i1,j1) represents the vertical axis partial derivative value of the pixel point with the horizontal coordinate i1 and the vertical coordinate j1; I(i1,j1 + 1) represents the pixel value of the pixel point with the horizontal coordinate i1 and the vertical coordinate j1 + 1; Based on the horizontal axis partial derivative value and the vertical axis partial derivative value of the pixel point, calculate the gradient parameter of the pixel point according to the following formula: G(i1,j1) = dx(i1,j1) + dy(i1,j1); Among them, G(i1, j1) represents the gradient parameter of the pixel point with the abscissa i1 and the ordinate j1.

8. The multi-dimensional quality score evaluation method for multi-modal data according to claim 1, wherein The step of taking the partial derivative of each point cloud in the three-dimensional point cloud image with respect to the adjacent point cloud based on the reflectivity value to obtain the gradient parameter of each point cloud includes: Based on the reflectivity value of the point cloud and the reflectivity value of the adjacent point cloud in the increasing direction of the abscissa of this point cloud, calculate the horizontal axis partial derivative value of this point cloud according to the following formula: dx(i2, j2, h) = I(i2 + 1, j2, h) - I(i2, j2, h); Among them, dx(i2, j2, h) represents the horizontal axis partial derivative value of the point cloud with the abscissa i2, the ordinate j2, and the vertical coordinate h; I(i2 + 1, j2, h) represents the reflectivity value of the point cloud with the abscissa i2 + 1, the ordinate j2, and the vertical coordinate h; I(i2, j2, h) represents the reflectivity value of the point cloud with the abscissa i2, the ordinate j2, and the vertical coordinate h; Based on the reflectivity value of the point cloud and the reflectivity value of the adjacent point cloud in the increasing direction of the ordinate of this point cloud, calculate the vertical axis partial derivative value of this point cloud according to the following formula: dy(i2, j2, h) = I(i2, j2 + 1, h) - I(i2, j2, h); Among them, dy(i2, j2, h) represents the vertical axis partial derivative value of the point cloud with the abscissa i2, the ordinate j2, and the vertical coordinate h; I(i2, j2 + 1, h) represents the reflectivity value of the point cloud with the abscissa i2, the ordinate j2 + 1, and the vertical coordinate h; Based on the reflectivity value of the point cloud and the reflectivity value of the adjacent point cloud in the increasing direction of the vertical coordinate of this point cloud, calculate the vertical axis partial derivative value of this point cloud according to the following formula: dz(i2, j2, h) = I(i2, j2, h + 1) - I(i2, j2, h); Among them, dz(i2, j2, h) represents the vertical axis partial derivative value of the point cloud with the abscissa i2, the ordinate j2, and the vertical coordinate h; I(i2, j2, h + 1) represents the reflectivity value of the point cloud with the abscissa i2, the ordinate j2, and the vertical coordinate h + 1; Based on the horizontal axis partial derivative value, vertical axis partial derivative value, and vertical axis partial derivative value of the point cloud, calculate the gradient parameter of this point cloud according to the following formula: G(i2, j2, h) = dx(i2, j2, h) + dy(i2, j2, h) + dz(i2, j2, h); Among them, G(i2, j2, h) represents the gradient parameter of the point cloud with the abscissa i2, the ordinate j2, and the vertical coordinate h.

9. The multi-dimensional quality score evaluation method for multi-modal data according to claim 1, characterized in that In the step of calculating the geometric dimension evaluation parameter of the two-dimensional RGB image based on the gradient amplitude similarity parameter and the total similarity parameter, calculate the geometric dimension evaluation parameter according to the following formula: s = w1t + w2q2; Among them, t represents the total similarity parameter, q2 represents the gradient amplitude similarity parameter, both w1 and w2 represent weight parameters, and s represents the geometric dimension evaluation parameter.

10. The multi-dimensional quality score evaluation method for multi-modal data according to claim 1, characterized in that, In the step of calculating the quality scores of the three-dimensional point cloud image and the two-dimensional RGB image respectively based on the geometric dimension evaluation parameter, intensity dimension evaluation parameter, and noise dimension evaluation parameter, calculate the quality scores of the three-dimensional point cloud image and the two-dimensional RGB image according to the following formula: s’ = w3q1 + w4c + w5s; Wherein, q1 represents a noise dimension evaluation parameter, c represents an intensity dimension evaluation parameter, s represents a geometric dimension evaluation parameter, and w3, w4, and w5 all represent weight parameters; when q1, c, and s are respectively the noise dimension evaluation parameter, intensity dimension evaluation parameter, and geometric dimension evaluation parameter of a two-dimensional RGB image, s’ represents the quality score of the two-dimensional RGB image; when q1, c, and s are respectively the noise dimension evaluation parameter, intensity dimension evaluation parameter, and geometric dimension evaluation parameter of a three-dimensional point cloud image, s’ represents the quality score of the three-dimensional point cloud image.

11. A multi-dimensional quality score evaluation device for multi-modal data, characterized in that The device includes a computer device, the computer device includes a processor and a memory, the memory stores computer instructions, and the processor is configured to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the device implements the steps implemented by the method according to any one of claims 1-10.

Images