A method, device and storage medium for cross-modal data optimization in autonomous driving technology

Abstract

The present invention provides a method, device and storage medium for cross-modal data optimization in autonomous driving technology, the method comprising the following steps: receiving a data image; calculating an integrity parameter based on missing parts of the data image; denoising the data image to obtain real image parameters, and calculating an accuracy parameter based on the original image parameters and the real image parameters; calculating a consistency parameter based on the frames before and after the original image; using the resolution of the data image as a resolution parameter; using the time consumed by the sensor to output the data image as a time cost parameter; calculating a quality score based on the integrity parameter, accuracy parameter, consistency parameter, resolution parameter and time cost parameter; inputting an initial quality score sequence into a long short-term memory network model to obtain a predicted quality score sequence; calculating a final quality score based on the quality score of the data image of each modality in multiple predicted quality score sequences, and determining the applicable modality based on comparison of the final quality scores.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a method, device, and storage medium for optimizing cross-modal data in autonomous driving technology. Background Art

[0002] In recent years, with the widespread application of deep learning technology, autonomous driving technology has made explosive progress. Compared to traditional mathematical models, deep learning technology has significantly improved data processing capabilities, thereby enriching the data collection systems for autonomous driving technology. An increasing number of sensor types are being used in autonomous driving systems to collect data from various modalities. Currently, a complete autonomous driving system typically includes laser radar (Lidar), cameras, millimeter-wave radar (RADAR), global positioning systems (GPS), and inertial measurement units (IMUs).

[0003] Traditional autonomous driving data processing typically considers the quality of a single modality, using quality models or deep learning methods to assess data quality and subsequently optimize and cleanse the data. However, with the advancement of autonomous driving technology, an increasing number of sensor types are being used for environmental perception, and multimodal data processing has become a hot topic in the autonomous driving field.

[0004] Due to the different characteristics of these sensors, the data quality they can provide in different environments is different. For example, the performance of lidar will be greatly affected in fog or heavy rain, while cameras will have difficulty functioning in the dark. Existing technologies make it difficult to adjust sensors according to actual scenarios. Summary of the Invention

[0005] In view of this, an embodiment of the present invention provides a cross-modal data optimization method in autonomous driving technology to eliminate or improve one or more defects existing in the prior art.

[0006] One aspect of the present invention provides a method for cross-modal data optimization in autonomous driving technology, the method comprising the following steps:

[0007] receiving data images of different modalities collected by multiple sensors of the driving system, wherein the data images of different modalities include video images, lidar images, and millimeter-wave radar images;

[0008] The integrity parameter is calculated based on the ratio of the missing part of the data image to the total image size; the original image parameters of the data image are denoised to obtain the real image parameters, and the accuracy parameter is calculated based on the difference between the original image parameters and the real image parameters corresponding to each original image parameter; the mean square error is calculated based on all the original image parameters of the data image and all the original image parameters of the data image of the previous frame or the next frame of the data image, and the mean square error is used as the consistency parameter; the resolution of the data image is used as the resolution parameter; the time consumed by the sensor to output the data image is used as the time cost parameter;

[0009] The quality score of each modality data image is obtained by weighted calculation based on the integrity parameter, accuracy parameter, consistency parameter, resolution parameter and time cost parameter of each modality data image;

[0010] The quality scores of the data images of each modality are combined to obtain an initial quality score sequence, which is then input into a preset long short-term memory network model to obtain a predicted quality score sequence;

[0011] Multiple predicted quality score sequences are calculated, and a final quality score of the data image corresponding to each modality is calculated based on the quality score of the data image of each modality in the multiple predicted quality score sequences. The applicable modality is determined based on the final quality score comparison results of the data images of multiple modalities.

[0012] By adopting the above scheme, this scheme builds a cross-modal data quality assessment model, uses the five dimensions of completeness, consistency, accuracy, resolution and time cost to evaluate the quality of data in different modalities, and obtains a quality score that can be compared across modalities; at the same time, it can use the LSTM model to make reasonable predictions about the trend of future quality scores based on known quality scores, thereby effectively reducing the system's latency in a real-time environment. This scheme can be widely applied to data types of multiple modalities and can adapt well to the low-latency environment of autonomous driving. It can be used in most autonomous driving systems supported by multi-modal data, and can determine the modality to be used based on the actual data obtained, and adjust the adaptive sensors according to the actual scenario.

[0013] In some embodiments of the present invention, in the step of calculating the integrity parameter based on the ratio of the missing portion of the data image to the total image size, if the data image is a video image, the ratio of the missing portion of the pixel blocks of the video image to the total number of pixel blocks in the image is calculated; if the data image is a lidar image or a millimeter-wave radar image, the ratio of the missing portion of the point cloud in the video image to the total number of point clouds in the total image is calculated.

[0014] In some embodiments of the present invention, in the step of calculating the accuracy parameter based on the difference between the original image parameters and the real image parameters corresponding to each original image parameter, the difference between the real image parameters and the original image parameters at the corresponding position is calculated, and the average value of all the differences is calculated as the accuracy parameter.

[0015] In some embodiments of the present invention, if the data image is a video image, the real image parameters and original image parameters of the corresponding position are the RGB parameters or grayscale value parameters of the pixel block at the corresponding position; if the data image is a lidar image or a millimeter-wave radar image, the real image parameters and original image parameters of the corresponding position are the RGB parameters or reflection intensity parameters of the point cloud at the corresponding position in the lidar image or millimeter-wave radar image.

[0016] In some embodiments of the present invention, in the step of obtaining a quality score of each modality data image by weighted calculation based on the integrity parameter, accuracy parameter, consistency parameter, resolution parameter, and time cost parameter of the data image of each modality, the quality score of each modality data image is calculated according to the following formula:

[0017] S=w1·v1+w2·v2+w3·v3+w4·v4+w5·v5;

[0018] v1, v2, v3, v4 and v5 represent the completeness parameter, accuracy parameter, consistency parameter, resolution parameter and time cost parameter respectively; w1, w2, w3, w4 and w5 represent the weight parameters corresponding to the completeness parameter, accuracy parameter, consistency parameter, resolution parameter and time cost parameter respectively, and S represents the quality score.

[0019] In some embodiments of the present invention, in the step of denoising the original image parameters of the data image to obtain the real image parameters, the original image parameters of the data image are denoised by using a 3x3 standardized box filter to perform mean filtering.

[0020] In some embodiments of the present invention, in the step of calculating the final quality score of the data image corresponding to each modality based on the quality score of the data image of each modality in multiple predicted quality score sequences, multiple parameters of the quality score of the data image of the same modality in the multiple predicted quality score sequences are extracted, and the average value of the multiple parameters is calculated as the final quality score of the data image of that modality.

[0021] In some embodiments of the present invention, in the step of determining the applicable modality based on the final quality score comparison results of data images of multiple modalities, the final quality scores of the data images of each modality are compared, and the modality corresponding to the data image of the modality with the largest final quality score is selected as the applicable modality.

[0022] The present invention also provides a cross-modal data optimization device in autonomous driving technology, which includes a computer device, wherein the computer device includes a processor and a memory, wherein the memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the device implements the steps implemented by the method described above.

[0023] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps implemented by the cross-modal data optimization method in the aforementioned autonomous driving technology.

[0024] Additional advantages, objects, and features of the present invention will be described in part in the following description and will become apparent to those skilled in the art after studying the following or may be learned by practice of the present invention. The objects and other advantages of the present invention may be particularly pointed out and attained in the description and drawings.

[0025] Those skilled in the art will understand that the purposes and advantages that can be achieved by the present invention are not limited to the above specific descriptions, and the above and other purposes that can be achieved by the present invention will be more clearly understood based on the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention.

[0027] Figure 1 Schematic diagram of an embodiment of a method for optimizing cross-modal data in the autonomous driving technology of the present invention;

[0028] Figure 2 Schematic diagram of the processing flow of the present invention. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0030] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, the accompanying drawings only show structures and / or processing steps closely related to the solutions according to the present invention, while other details that are not closely related to the present invention are omitted.

[0031] It should be emphasized that the term "include / comprises" when used herein refers to the existence of features, elements, steps or components, but does not exclude the existence or addition of one or more other features, elements, steps or components.

[0032] It should also be noted that, unless otherwise specified, the term "connection" herein may refer not only to a direct connection but also to an indirect connection involving an intermediate.

[0033] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0034] To solve the above problems, Figure 1 As shown, the present invention proposes a method for optimizing cross-modal data in autonomous driving technology; the steps of the method include:

[0035] Step S100: receiving data images of different modalities collected by multiple sensors of a driving system, wherein the data images of different modalities include video images, lidar images, and millimeter-wave radar images;

[0036] In some embodiments of the present invention, the sensor includes a lidar, a camera, and a millimeter-wave radar (RADAR). The image captured by the lidar is a lidar image, the image captured by the camera is a video image, and the image captured by the millimeter-wave radar is a millimeter-wave radar image. The sensor is multi-modal, and the data images captured by the multiple sensors are data images of different modalities.

[0037] Step S200: Calculate an integrity parameter based on the ratio of the missing portion of the data image to the total image size; perform denoising on the original image parameters of the data image to obtain true image parameters, and calculate an accuracy parameter based on the difference between the original image parameters and the true image parameters corresponding to each original image parameter; calculate a mean square error based on all original image parameters of the data image and all original image parameters of the data image of the previous or next frame of the data image, and use the mean square error as a consistency parameter; use the resolution of the data image as a resolution parameter; and use the time consumed by the sensor to output the data image as a time cost parameter.

[0038] In some embodiments of the present invention, in the step of calculating the integrity parameter based on the ratio of the missing part of the data image to the total image size, if the data image is a video image, the pixel blocks in the video image whose RGB parameters or grayscale value parameters are greater than the missing threshold are determined as missing pixel blocks, and the ratio of the number of missing pixel blocks to the total number of pixel blocks is the integrity parameter; if the data image is a lidar image or a millimeter-wave radar image, the set number of point clouds in each lidar image or millimeter-wave radar image is first obtained according to the hardware parameters of the lidar or millimeter-wave radar, and then the number of point clouds of the collected lidar image or millimeter-wave radar image is determined, and the difference between the set number of point clouds and the number of point clouds of the collected lidar image or millimeter-wave radar image is calculated as the number of missing point clouds, which is the missing part of the data image, and the ratio of the number of missing point clouds to the set number of point clouds is calculated as the integrity parameter.

[0039] The number of point clouds of the lidar image or millimeter-wave radar image is the number of point cloud points in the image.

[0040] Missing points are represented differently in video images and radar point clouds. For video images, regardless of missing data, the image data fills the entire area, and each pixel in the image has grayscale and RGB values. For radar point clouds, however, if data is missing for some reason, it will be reflected as a missing point in the radar point cloud. Therefore, for video images, we consider points with RGB values ​​less than 5 to be missing, using a threshold of 5. For radar point clouds, missing points are simply considered missing.

[0041] Step S300, obtaining a quality score of each modality data image by weighted calculation based on the integrity parameter, accuracy parameter, consistency parameter, resolution parameter, and time cost parameter of each modality data image;

[0042] Adopting the above scheme, this scheme evaluates the quality of data of different modalities collected by various sensors of the autonomous driving system through five dimensions: completeness, accuracy, consistency, resolution and time cost. The quality evaluation of this model is used to obtain the corresponding quality scores of data of different modalities, and these quality scores of different modalities are used as the standard for cross-modal quality evaluation of data of different modalities, thereby improving the accuracy of modal evaluation.

[0043] First, we combined quality evaluation metrics from relational databases and streaming data to establish a cross-modal quality assessment model for deriving quality scores for modal data images. Given that this model needs to assess the quality of data from multiple modalities, we primarily considered basic, common quality metrics for each modality. Therefore, our quality assessment model primarily considers the quality of multimodal data from five dimensions: completeness, accuracy, consistency, resolution, and time cost.

[0044] Integrity refers to whether there is any missing information in the video image or radar data, specifically missing pixel values ​​or radar point clouds that shouldn't be present. This missing information is reflected as black blocks or points in the video image data, and as missing points in the radar point cloud data. If this missing information is missing for some reason, the integrity of the frame has been compromised.

[0045] Accuracy was first developed in statistics. In statistics, accuracy, also known as precision, refers to the degree of closeness between the observed value of a test indicator or trait and its true value in an experiment or survey.

[0046] In the autonomous driving system involved in this article, accuracy refers to the ratio between the true value of the video image or radar and the noise (the difference between the measured value and the true value). However, due to the conditions of the real-time working environment of the autonomous driving system, it will be difficult to obtain the so-called "true value" of the video image or radar during the operation of the autonomous driving system. Therefore, the present invention uses a method of denoising the measured value to simulate the "true value" of the video image or radar.

[0047] In file storage systems, consistency refers to the logical consistency between data and metadata, as well as overall logical consistency across the data. Maintaining file system consistency reduces the likelihood of file corruption and saves disk space.

[0048] In the autonomous driving systems discussed in this article, consistency is considered the logical consistency of previous and subsequent data, that is, the stability of the acquired information. If a video image or radar image experiences significant jitter during a certain period, we consider the consistency of that period to be low. This article measures the consistency of video images or point clouds by calculating the mean squared error (MSE) of the pixel values ​​or point cloud value distributions between two frames of video images or radar images.

[0049] Step S400: combining the quality scores of the data images of each modality to obtain an initial quality score sequence, and inputting the initial quality score sequence into a preset long short-term memory (LSTM) network model to obtain a predicted quality score sequence;

[0050] In some embodiments of the present invention, the LSTM model can well match the system requirements and complete the prediction of quality scores. Under the condition of using existing quality scores as input, the LSTM model can predict the data quality scores in the future period of time. By predicting the trend of data quality scores, timely and dynamic adjustments can be made to the data optimization, effectively improving the overall operating efficiency.

[0051] The LSTM model is a special recurrent neural network (RNN) model that improves on the RNN model. Its main contribution is to solve the problems of vanishing and exploding gradients during long sequence training. This improvement to RNN enables it to truly and effectively utilize long-distance temporal information, enabling LSTM to perform better in longer sequences.

[0052] Compared to RNN, which only has one transfer state h t , the LSTM model has two transmission states, one c t (cell state), and an h t (hidden state)(h in RNN t For c in LSTM t ).

[0053] For the c passed down t Changes slowly, usually outputting c t It is c passed from the previous state t-1 Add some values. And h t There are often big differences at different nodes.

[0054] In step S500, a plurality of predicted quality score sequences are calculated, and a final quality score of the data image corresponding to each modality is calculated based on the quality score of the data image of each modality in the plurality of predicted quality score sequences. The applicable modality is determined based on the comparison results of the final quality scores of the data images of the plurality of modalities.

[0055] In some embodiments of the present invention, the multiple predicted quality score sequences are all output by a long short-term memory network model. In the acquisition step, the output predicted quality score sequence can be input again to obtain the predicted quality score sequence output by the long short-term memory network model again, and the predicted quality score sequences obtained multiple times are combined to obtain multiple predicted quality score sequences.

[0056] By adopting the above scheme, this scheme builds a cross-modal data quality assessment model, uses the five dimensions of completeness, consistency, accuracy, resolution and time cost to evaluate the quality of data in different modalities, and obtains a quality score that can be compared across modalities; at the same time, it can use the LSTM model to make reasonable predictions about the trend of future quality scores based on known quality scores, thereby effectively reducing the system's latency in a real-time environment. This scheme can be widely applied to data types of multiple modalities and can adapt well to the low-latency environment of autonomous driving. It can be used in most autonomous driving systems supported by multi-modal data, and can determine the modality to be used based on the actual data obtained, and adjust the adaptive sensors according to the actual scenario.

[0057] In some embodiments of the present invention, in the step of calculating the integrity parameter based on the ratio of the missing portion of the data image to the total image size, if the data image is a video image, the ratio of the missing portion of the pixel blocks of the video image to the total number of pixel blocks in the image is calculated; if the data image is a lidar image or a millimeter-wave radar image, the ratio of the missing portion of the point cloud in the video image to the total number of point clouds in the total image is calculated.

[0058] By adopting the above solution, the integrity parameters of data images of different modalities can be obtained according to the corresponding algorithms matched with different modalities.

[0059] In some embodiments of the present invention, in the step of calculating the accuracy parameter based on the difference between the original image parameters and the real image parameters corresponding to each original image parameter, the difference between the real image parameters and the original image parameters at the corresponding position is calculated, and the average value of all the differences is calculated as the accuracy parameter.

[0060] In some embodiments of the present invention, if the data image is a video image, then in the step of calculating the difference between the real image parameters and the original image parameters at the corresponding position and calculating the average value of all the differences as the accuracy parameter, the difference between the real image parameters and the original image parameters of each pixel block is calculated, and the average value of the corresponding differences of all pixel blocks is calculated as the accuracy parameter; if the data image is a lidar image or a millimeter wave radar image, then in the step of calculating the difference between the real image parameters and the original image parameters at the corresponding position and calculating the average value of all the differences as the accuracy parameter, the difference between the real image parameters and the original image parameters of each point cloud is calculated, and the average value of the corresponding differences of all point clouds is calculated as the accuracy parameter.

[0061] By using the above scheme, all pixel blocks or point clouds are compared, and the differences between each part of the two are accurately compared, thereby improving the accuracy of the accuracy parameters.

[0062] In some embodiments of the present invention, if the data image is a video image, the real image parameters and original image parameters of the corresponding position are the RGB parameters or grayscale value parameters of the pixel block at the corresponding position; if the data image is a lidar image or a millimeter-wave radar image, the real image parameters and original image parameters of the corresponding position are the RGB parameters or reflection intensity parameters of the point cloud at the corresponding position in the lidar image or millimeter-wave radar image.

[0063] In some embodiments of the present invention, in the step of obtaining a quality score of each modality data image by weighted calculation based on the integrity parameter, accuracy parameter, consistency parameter, resolution parameter, and time cost parameter of the data image of each modality, the quality score of each modality data image is calculated according to the following formula:

[0064] S=w1·v1+w2·v2+w3·v3+w4·v4+w5·v5;

[0065] v1, v2, v3, v4 and v5 represent the completeness parameter, accuracy parameter, consistency parameter, resolution parameter and time cost parameter respectively; w1, w2, w3, w4 and w5 represent the weight parameters corresponding to the completeness parameter, accuracy parameter, consistency parameter, resolution parameter and time cost parameter respectively, and S represents the quality score.

[0066] Using the above scheme, based on the integrity parameter, accuracy parameter, consistency parameter, resolution parameter and time cost parameter, the quality score of the data image of each modality can be calculated separately, fully reflecting the objectivity of the quality score evaluation.

[0067] In the specific implementation process, resolution and time cost are inherent properties of the hardware. We will score these two dimensions based on the corresponding values ​​of the hardware. For example, the highest video image pixel value used by most autonomous driving systems is approximately 900×1600. If the image data has a resolution of 900×1600, the resolution parameter can be 0.8, 0.9, or 1. If the resolution is lower than 900×1600, the corresponding resolution parameter is reduced to 0.6, 0.7, or 0.8. Similarly, time cost mainly depends on the autonomous driving system's ability to process video images and radar point clouds. We also regard this as an inherent metric of the hardware, using the time consumed by the sensor to output the data image as the time cost parameter, and scoring this dimension based on the data processing capabilities of the hardware.

[0068] In some embodiments of the present invention, in the step of denoising the original image parameters of the data image to obtain the real image parameters, the original image parameters of the data image are denoised by performing mean filtering using a 3x3 standardized box filter.

[0069] In some embodiments of the present invention, a 3x3 normalized box filter is as follows:

[0070]

[0071] In some embodiments of the present invention, in the step of calculating the final quality score of the data image corresponding to each modality based on the quality score of the data image of each modality in multiple predicted quality score sequences, multiple parameters of the quality score of the data image of the same modality in the multiple predicted quality score sequences are extracted, and the average value of the multiple parameters is calculated as the final quality score of the data image of that modality.

[0072] In the specific implementation process, if the selected modalities include lidar, camera, and millimeter-wave radar (RADAR), the predicted quality score sequence is a three-dimensional vector corresponding to the three modalities respectively. If the quality score of the lidar is in the first dimension, the quality score of the camera is in the second dimension, and the quality score of the millimeter-wave radar is in the third dimension; if the final quality score of the lidar is calculated, the first dimension parameters of multiple predicted quality score sequences are extracted respectively, and the average value of the multiple first dimension parameters is calculated as the final quality score of the data image of the lidar modality.

[0073] In some embodiments of the present invention, in the step of determining the applicable modality based on the final quality score comparison results of data images of multiple modalities, the final quality scores of the data images of each modality are compared, and the modality corresponding to the data image of the modality with the largest final quality score is selected as the applicable modality.

[0074] The above scheme is adopted to screen out the mode to be finally used.

[0075] like Figure 2 As shown, Figure 2 The quality assessment model outputs a quality score, and the data optimization algorithm outputs an applicable modality. After this solution obtains multiple predicted quality score sequences that change over time, the dynamic optimization algorithm will put the values ​​of the latest 50 predicted quality score sequences into a sequence of size 50. We use the average of the quality scores in the queue as the current final quality score of a modality data. Based on the comparison of the final quality scores between different modalities, the system will select the data modality that best suits the current environment. Moreover, as the latest calculated comprehensive quality score continuously updates the data in the queue, the final quality scores of the two modalities will also change accordingly, and the system will also continuously make dynamic adjustments based on the latest data.

[0076] Figure 2 The quality ranking of different modal data is the calculation step of the data optimization algorithm.

[0077] After obtaining a certain number of quality score sequences, LSTM can use the existing data quality score sequences as input to predict the possible changing trends of the quality scores in the future. After obtaining sufficient LSTM prediction values, the system will replace the data in the queue with 25 frames of actual values ​​+ 25 frames of predicted values. These prediction values ​​can enable the data optimization system to respond promptly to changes in the surrounding environment, reduce system latency, and improve real-time performance.

[0078] There is also a data fusion method in the prior art to evaluate data from multiple modalities. The method involves fusing data from two different modalities in time or space to address the problem of reduced data quality from different sensors under certain circumstances.

[0079] Data fusion is used to process data from different modalities, combining their strengths to improve data reliability and stability. However, this data fusion method is computationally intensive, requiring a large amount of data for pre-training and exhibiting high latency, which increases the risk of autonomous driving systems. Compared to this solution, these data fusion methods suffer from high latency and computational complexity.

[0080] This approach integrates data from different modalities into a cross-modal quality assessment model for quality evaluation. The model then scores the quality of each modality based on the different dimensions of the model. Based on the scoring results, the system determines, with low latency or in real time, which modality possesses the highest quality data in the current environment, making it the most suitable data source for the environment. Compared to data fusion methods, the data optimization method proposed in this paper is more concise and efficient, making it more suitable for autonomous driving systems with high real-time requirements.

[0081] The beneficial effects of this program include:

[0082] 1. Dimension selection problem of cross-modal quality assessment model

[0083] The existing single-modality data quality assessment models only consider the quality assessment indicators of a single modality. If we want to build a cross-modal data quality assessment model, the quality assessment indicators in this model, that is, the dimensions of the model, must be applicable to the evaluation of data quality of multiple modalities and be able to accurately measure the quality scores of data from different modalities. To address this problem, the features selected as the dimensions of the quality assessment model in this invention have the following characteristics:

[0084] (1) Features are shared among all modal data;

[0085] (2) Features are relatively common for streaming data;

[0086] (3) Features are relatively easy to calculate efficiently;

[0087] 2. Dynamic Selection of Multimodal Data

[0088] During the driving process of the autonomous driving system, scene switching may occur at any time. When the surrounding environment changes significantly, our system is required to dynamically select the best modal data based on the changes in the environment and the real-time quality scores of each modal data. Since the autonomous driving system is usually in a relatively fast environment, the algorithm we designed is required to predict the environment in which the autonomous driving system may be located in a period of time in the future based on the current trend of environmental changes, and to make reasonable adjustments to the data modality that will be prioritized next based on the results of this prediction. To address this problem, the present invention uses the existing quality score as input and can make predictions about the quality score change trend over a period of time.

[0089] 3. This solution assesses the quality of data in different modalities collected by various sensors in the autonomous driving system using five dimensions: completeness, accuracy, consistency, resolution, and time cost. Using this model's quality assessment, it derives corresponding quality scores for data in different modalities. These quality scores serve as the standard for cross-modal quality assessment of data in different modalities.

[0090] 4. This invention takes advantage of the LSTM model's ability to learn and predict long-term series data. By inputting existing data quality scores, it predicts the data quality scores for a period of time in the future. It also makes timely and dynamic adjustments to the data optimization based on the predicted data quality score trends, which can effectively improve the overall operating efficiency of the system.

[0091] 5. The present invention has extremely high generalization, simple calculation, can be universally applied to data types of multiple modalities, and can adapt well to the low-latency environment of autonomous driving. It can be used in most autonomous driving systems supported by multimodal data.

[0092] An embodiment of the present invention also provides a cross-modal data optimization device in autonomous driving technology, which includes a computer device, wherein the computer device includes a processor and a memory, wherein the memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the device implements the steps implemented by the method described above.

[0093] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps implemented by the aforementioned cross-modal data optimization method in autonomous driving technology. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the art.

[0094] It should be understood by those skilled in the art that the various exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is specifically performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0095] It should be understood that the present invention is not limited to the specific configurations and processes described above and illustrated in the figures. For the sake of brevity, a detailed description of known methods is omitted. In the above embodiments, several specific steps are described and illustrated as examples. However, the method of the present invention is not limited to the specific steps described and illustrated. Those skilled in the art may make various changes, modifications, and additions, or change the order of the steps after understanding the spirit of the present invention.

[0096] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or replace features of other embodiments.

[0097] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations to the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A cross-modal data optimization method in autonomous driving technology, characterized in that: The steps of the method include: receiving data images of different modalities collected by multiple sensors of the driving system, wherein the data images of different modalities include video images, lidar images, and millimeter-wave radar images; The integrity parameter is calculated based on the ratio of the missing part of the data image to the total image size; the original image parameters of the data image are denoised to obtain the real image parameters, and the accuracy parameter is calculated based on the difference between the original image parameters and the real image parameters corresponding to each original image parameter; the mean square error is calculated based on all the original image parameters of the data image and all the original image parameters of the data image of the previous frame or the next frame of the data image, and the mean square error is used as the consistency parameter; the resolution of the data image is used as the resolution parameter; the time consumed by the sensor to output the data image is used as the time cost parameter; The quality score of each modality data image is obtained by weighted calculation based on the integrity parameter, accuracy parameter, consistency parameter, resolution parameter and time cost parameter of each modality data image. The quality score of each modality data image is calculated according to the following formula: 、 、 、 and They represent completeness parameter, accuracy parameter, consistency parameter, resolution parameter and time cost parameter respectively; 、 、 、 and They represent the weight parameters corresponding to the completeness parameter, accuracy parameter, consistency parameter, resolution parameter and time cost parameter, respectively, and S represents the quality score; The quality scores of the data images of each modality are combined to obtain an initial quality score sequence, which is then input into a preset long short-term memory network model to obtain a predicted quality score sequence; Multiple predicted quality score sequences are calculated, and a final quality score of the data image corresponding to each modality is calculated based on the quality score of the data image of each modality in the multiple predicted quality score sequences. The applicable modality is determined based on the final quality score comparison results of the data images of multiple modalities.

2. The cross-modal data optimization method in autonomous driving technology according to claim 1, characterized in that: In the step of calculating the integrity parameter based on the ratio of the missing part of the data image to the total image size, if the data image is a video image, the ratio of the missing part of the pixel blocks of the video image to the total number of pixel blocks in the image is calculated; if the data image is a lidar image or a millimeter-wave radar image, the ratio of the missing part of the point cloud in the video image to the total number of point clouds in the total image is calculated.

3. The cross-modal data optimization method in autonomous driving technology according to claim 1, characterized in that: In the step of calculating the accuracy parameter according to the difference between the original image parameter and the true image parameter corresponding to each original image parameter, The difference between the real image parameters and the original image parameters at the corresponding position is calculated, and the average value of all the differences is calculated as the accuracy parameter.

4. The cross-modal data optimization method in autonomous driving technology according to claim 1, characterized in that: If the data image is a video image, the real image parameters and original image parameters at the corresponding position are the RGB parameters or grayscale value parameters of the pixel block at the corresponding position; if the data image is a lidar image or a millimeter-wave radar image, the real image parameters and original image parameters at the corresponding position are the RGB parameters or reflection intensity parameters of the point cloud at the corresponding position in the lidar image or millimeter-wave radar image.

5. The cross-modal data optimization method in autonomous driving technology according to claim 1, characterized in that: In the step of performing denoising on the original image parameters of the data image to obtain the real image parameters, the original image parameters of the data image are denoised by performing mean filtering using a 3x3 standardized box filter.

6. The cross-modal data optimization method in autonomous driving technology according to claim 1, characterized in that: In the step of calculating the final quality score of the data image corresponding to each modality according to the quality score of the data image of each modality in the plurality of predicted quality score sequences, Extract multiple parameters of the quality score of the data image of the same modality in multiple predicted quality score sequences, and calculate the average value of the multiple parameters as the final quality score of the data image of the modality.

7. The cross-modal data optimization method in autonomous driving technology according to claim 1, characterized in that: In the step of determining the applicable modality based on the final quality score comparison results of the data images of multiple modalities, the final quality scores of the data images of the various modalities are compared, and the modality corresponding to the data image of the modality with the largest final quality score is determined as the applicable modality.

8. A cross-modal data optimization device in autonomous driving technology, characterized in that: The apparatus includes a computer device, the computer device includes a processor and a memory, the memory stores computer instructions, the processor is used to execute the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the apparatus implements the steps implemented by the method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that A computer program is stored thereon, and when the computer program is executed by a processor, the steps implemented by the method according to any one of claims 1 to 7 are implemented.

Images