Data alignment method and intelligent vehicle

Abstract

The present disclosure proposes a data alignment method and an intelligent vehicle. The method includes: obtaining a road environment video through an image capturing device on the intelligent vehicle; determining a current driving mode of the intelligent vehicle to determine a prompt and a test frame accordingly; inputting the road environment video and the prompt into a language model to segment the road environment video into multiple clips; calculating the similarity between each frame in the clips and the test frame to determine a first time point of the clips; obtaining sensor data through a sensor on the intelligent vehicle and generating a one-dimensional signal based on the sensor data; obtaining an extremum of the one-dimensional signal within a window at the first time point, wherein the extremum occurs at a second time point; and aligning the road environment video with the sensor data based on the first time point and the second time point.

Description

Technical Field

[0001] The present disclosure relates to a method for aligning data from various sensors on a smart car. Prior Art

[0002] Autonomous driving technology has garnered significant investment and development in recent years. According to the Society of Automotive Engineers (SAE) standards, autonomous driving technology can be categorized from Level 0 to Level 5. Advanced Driver Assistance Systems (ADAS) play a key role in lower levels of autonomous driving, assisting the driver and improving driving safety. However, a complete automated driving system (ADS) relies on sensors, artificial intelligence algorithms, and other technologies for decision-making, enabling fully autonomous driving without driver intervention at higher levels (Levels 4-5).

[0003] As the level of autonomous driving increases, the amount of data vehicles need to process and store increases significantly. For example, Level 2 vehicles require approximately 4 to 10 petabytes of data, while Level 5 vehicles require at least 3 exabytes of storage space. This massive amount of data poses significant challenges to the processing, transmission, and fusion of sensor data. Currently, some automakers primarily use pure vision solutions, relying on cameras for environmental perception, while others employ multi-sensor technologies such as radar, lidar, and ultrasonic sensors to enhance sensing capabilities in diverse environments. However, the data formats of these sensors vary. For example, cameras provide image data, radar generates 3D point cloud data, and ultrasonic sensors provide distance information. Therefore, aligning and fusing data from these multiple sensors has become a key challenge in the development of autonomous driving technology. Summary of the Invention

[0004] In order to solve the above problems, the present disclosure provides a data processing method and a smart car.

[0005] This disclosure discloses a data alignment method applicable to smart cars. The method includes: obtaining a road environment video through an image capture device on the smart car; determining a current driving mode from among multiple driving modes of the smart car, and determining a first prompt and a test frame based on the current driving mode; inputting the road environment video and the first prompt into a language model to segment the road environment video into multiple segments, each segment comprising multiple frames; for one of the segments, calculating the similarity between each corresponding frame and the test frame to determine a first time point of the segment; obtaining sensor data through a sensor on the smart car and generating a one-dimensional signal based on the sensor data; obtaining an extreme value of the one-dimensional signal within a window at the first time point, wherein the extreme value occurs at a second time point; and aligning the road environment video and the sensor data based on the first time point and the second time point.

[0006] From another perspective, embodiments of the present disclosure provide a smart car comprising a processor, an image capture device, and a sensor. The image capture device is used to capture a video of the road environment, and the sensor is used to acquire sensor data. The processor is electrically connected to the image capture device and the sensor to perform the aforementioned data alignment method.

[0007] To make the above features and advantages of the present invention more clearly understood, embodiments are given below and described in detail with reference to the accompanying drawings. Simple diagram description

[0008] FIG1 is a schematic diagram illustrating a smart car according to an embodiment. FIG. 2 is a system block diagram of a smart car according to an embodiment. FIG. 3 is a flow chart illustrating a data alignment method according to an embodiment. FIG. 4 shows an example of a test screen according to an embodiment. FIG. 5 is a flow chart illustrating an alternative solution to step 303 according to another embodiment. FIG. 6 is a schematic diagram illustrating calculation of a feature vector according to an embodiment. FIG. 7 is a schematic diagram illustrating alignment of sensing data and a road environment video according to an embodiment. FIG. 8 is a schematic diagram illustrating a state after time displacement according to an embodiment. Implementation Method

[0009] Some embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Reference numbers will be used throughout the following description to identify identical or similar elements when the same reference numbers appear in different drawings. These embodiments constitute only a portion of the present invention and do not purportedly disclose all possible implementations of the present invention. Rather, these embodiments are merely illustrative of the systems and methods within the scope of the present invention.

[0010] The terms "first," "second," etc. used herein do not specifically refer to an order or sequence, but are merely used to distinguish elements or operations described with the same technical terms.

[0011] Figure 1 is a schematic diagram of a smart car according to one embodiment. Referring to Figure 1 , smart car 100 is a car equipped with an advanced driver assistance system, an autonomous driving system, or other intelligent systems. Smart car 100 is equipped with various sensors for detecting various environmental information.

[0012] FIG2 is a system block diagram of a smart car according to one embodiment. Referring to FIG2 , smart car 100 includes a processor 210, an image capture device 220, and multiple sensors 231-233. Processor 210 is electrically connected to image capture device 220 and sensors 231-233. Processor 210 may include a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor, a microcontroller, an image processing chip, a deep-learning processing unit (DPU), a neural network processing unit (NPU), a tensor processing unit (TPU), an application-specific integrated circuit (ASIC), a programmable logic device (PLD), etc. Image capture device 220 may include a charge-coupled device (CCD) sensor, a complementary metal-oxide semiconductor (CMOS) sensor, or other suitable photosensitive element. Sensors 231-233 include, for example, lidars, radars, and ultrasonic sensors. This disclosure does not limit the number or placement of the image capture devices 220 and sensors 231-233. For example, image capture devices 220 can be placed on the front windshield, rear, or left and right sides of the vehicle; lidars can be placed on the roof, front, or rear of the vehicle; and radars and ultrasonic sensors can be placed on the front bumper, rear bumper, or left and right sides of the vehicle.

[0013] The image capture device 220 captures video (also known as road environment video). The data generated by other sensors (referred to as sensor data) may include point clouds or various numerical values ​​(such as distance, speed, angle, etc.). However, these videos and sensor data may not be synchronized. The processor 210 executes a data alignment method to align the time axes of these videos and sensor data.

[0014] FIG3 is a flow chart illustrating a data alignment method according to one embodiment. In step 301, a video of the road environment is acquired via the image capture device 220 of the smart car 100. This video of the road environment can be of the front, right, left, or rear, and the position and capture direction of the image capture device 220 are not limited.

[0015] In step 302, a current driving mode among multiple driving modes of the smart car 100 is determined, and a first prompt and a test screen are determined based on the current driving mode. The driving mode is determined based on the driving status of the smart car 100 or whether a specific function is enabled. In some embodiments, the smart car 100 includes an adaptive cruise control (ACC) system, a rear cross traffic alert system (RCTA), a lane departure warning system (LDWS), a forward collision warning (FCW) system, an automatic emergency braking (AEB) system, a lane keeping assist (LKA) system, a blind spot detection (BSD) system, etc. The activation (or warning) of each system can correspond to different driving modes. The smart car 100 has at least four gear positions: P, R, N, and D. In some embodiments, different gear positions can correspond to different driving modes. In some embodiments, the driving mode can also be determined based on vehicle speed. The driving modes set in this embodiment include at least an adaptive cruise control (ACC) mode, a reverse mode, and a turning mode.

[0016] Because different modes focus on different road environments, different prompts and test screens can be set. For example, Figure 4 illustrates a test screen 400 in ACC mode. In reverse mode, the test screen might be an image captured from behind smart car 100. Prompts are explained in conjunction with step 303 below.

[0017] In step 303, the road environment video and the first prompt are input into a language model to segment the road environment video into multiple segments, each segment comprising multiple frames. This language model can be a GPT (Generative Pretrained Transformer) series, a BERT (Bidirectional Encoder Representations from Transformers) series, LLaMA (Large Language Model Meta AI), DeepSeek, Gemini, etc., but the present invention is not limited thereto. In some embodiments, the language model is a multimodal large language model (MLLM). In some embodiments, the prompt may include the driving mode and the orientation of the image capture device. For example, the prompt may be, "Currently in ACC mode, the camera is facing forward. Please segment the video based on whether there is a car ahead." If the current driving mode is reverse, the prompt may be, "Currently in reverse mode, the camera is facing backward. Please segment the video based on whether there is a car behind." Similarly, corresponding prompts can be generated when the road environment video is facing left, right, or other directions.

[0018] In some embodiments, any driving condition or environmental description of the smart car 100 can be incorporated into the prompt. For example, when the smart car 100 brakes, the prompt might be, "Currently in ACC mode, braking, camera facing forward, please split the video into multiple segments based on whether there are cars ahead." If other sensors detect rain, the prompt might be, "Currently in ACC mode, raining outside, camera facing forward, please split the video into multiple segments based on whether there are cars ahead." These driving condition and environmental descriptions can also include descriptions of neighboring vehicles cutting in line, rear-end collisions, and overtaking. These prompts are user-defined and editable, allowing for more flexible utilization of environmental information and the processing power of language models, and are more adaptable to vehicle and environmental conditions than fixed segmentation algorithms.

[0019] Figure 5 is a flowchart illustrating an alternative to step 303 according to another embodiment. Referring to Figure 5 , steps 501-504 can be used to replace step 303. In step 501, the road environment video and prompts are input into a language model to segment the road environment video into multiple segments. This step 501 is identical to step 303. However, the embodiment of Figure 5 also employs an evaluation model to measure the effectiveness of the segmentation. In step 502, the segmented segments are input into the evaluation model to obtain a score. This evaluation model can be another language model or any pre-trained neural network. In some embodiments, the evaluation model can input the start and end points of each segment, as well as the road environment image, or the road environment image and the separation point between two segments. Alternatively, multiple video segments can be directly input into the evaluation model. This disclosure does not limit the data type used to represent these segments.

[0020] In step 503, a determination is made as to whether the score is greater than a threshold. If so, the segmentation of the road environment video is complete, and the process proceeds to step 304. If the result of step 503 is negative, a second prompt is generated based on the score, such as "The segmentation score is 60 points. Please optimize the segmentation method." If the evaluation model is a language model, the evaluation model can also output a reason for the score of 60, and the evaluation model's output can also be incorporated into the second prompt. For example, the second prompt could be "The segmentation score is 60 points because there are continuous vehicles between the first and second segments. Please optimize the segmentation method." Next, step 501 is re-executed, and the road environment video and the second prompt are input into the language model to re-segment the road environment video.

[0021] In such an embodiment, the segmentation step 501 is iterated multiple times until the score exceeds a threshold. In some embodiments, an upper limit for iterations can also be set; iterations are terminated when the number of iterations exceeds this limit. Such an embodiment is equivalent to treating the evaluation model as an optimizer, and using natural language to describe the segmentation results to optimize the segmentation results. This allows for more accurate optimization than conventional techniques. In autonomous vehicles, there is a large amount of uncertain information, such as noise from multiple sensors and the sudden behavior of objects such as pedestrians and vehicles. However, many optimization techniques are iterative: the optimization begins with an initial solution and then iteratively updates the solution to optimize the objective function. Language models can be used to iteratively generate new solutions. The primary advantage of language models in optimization is their ability to understand natural language, which allows people to describe their optimization tasks without formal specifications. This advantage is particularly suitable for intelligent vehicles that handle complex functions and a large amount of uncertain information.

[0022] In some embodiments, the length of each segment can also be limited. If the segment length is too short, it doesn't contain enough information, while if it is too long, it may contain multiple scene changes. Therefore, the segment length limit can be added to the prompt.

[0023] Referring to Figure 3 , in step 304 , for a particular segment, the similarity between each frame in the segment and the test frame is calculated to determine the first time point of the segment. For example, if the road environment video depicts the scene ahead of smart car 100 , and the test frame is shown in Figure 4 , a similarity greater than a threshold indicates the presence of a car ahead, and the time point with the highest similarity can be set as the first time point.

[0024] In some embodiments, a feature vector for the frame and a feature vector for the test frame can be calculated first, and then the similarity between the two feature vectors can be calculated. Figure 6 is a schematic diagram illustrating the calculation of a feature vector according to one embodiment. Referring to Figure 6 , for a frame 601 in a segment (also referred to as the first frame), frame 601 can be reduced to a predetermined size to generate frame 602. This predetermined size is, for example, 8*8 and has only one channel, such as brightness. Next, the average value of all pixels in frame 602 is calculated, and then each pixel in frame 602 is determined to be greater than this average value to generate a bitmap 603. If the pixel is greater than the average value, a "1" is entered into the corresponding position in bitmap 603; if the pixel is less than or equal to the average value, a "0" is entered into the corresponding position in bitmap 603. Therefore, bitmap 603 contains a total of 64 bits, which are then combined to form a feature vector (also referred to as the first feature vector).

[0025] For the test image, a feature vector (also called the second feature vector) can also be calculated according to the process in Figure 6. Next, the similarity between the first feature vector and the second feature vector is calculated. This similarity can be cosine similarity, Hamming distance, Euclidean distance, etc. If using Hamming distance or Euclidean distance, the reciprocal of the distance can be calculated as the similarity.

[0026] Figure 7 is a schematic diagram illustrating the alignment of sensor data and a road environment video according to one embodiment. Referring to Figure 7 , multiple frames 713 are depicted, divided into segments 701, 702, and 703. Curve 711 represents the probability of an object (e.g., the preceding vehicle) appearing in a segment. In this embodiment, it represents the similarity between the first eigenvector and the second eigenvector. From another perspective, the similarity at different time points can also be called a time difference histogram. This time difference histogram shows that the similarity gradually increases and then decreases. Here, it can be determined whether the similarity exceeds a threshold. If so, the time point of the corresponding frame is set as the first time point T1. If the similarity of multiple frames in a segment exceeds the threshold, the time point T1 corresponding to the highest similarity is selected.

[0027] Referring to Figures 3 and 7 , in step 305 , sensor data is acquired through sensors on the smart car 100 and a one-dimensional signal is generated based on the sensor data. If the dimension of the sensor data itself is one (e.g., representing the distance directly ahead), the sensor data can be directly treated as a one-dimensional signal. If the dimension of the sensor data is greater than one (e.g., a point cloud or distances in multiple directions), the sensor data can be reduced in dimension to obtain a one-dimensional signal. For example, principal component analysis (PCA) can be performed on the sensor data at a certain time point to obtain a principal component vector. This principal component vector is the eigenvector corresponding to the largest eigenvalue. The length of the principal component vector, such as its 2-norm, is then calculated as the value of the one-dimensional signal at that time point. By performing this calculation for each time point, the values ​​of multiple time points can be obtained, thereby forming a one-dimensional signal, such as the one-dimensional signal 712 in Figure 7 .

[0028] In step 306, the extreme value of the one-dimensional signal 712 is obtained within the window 710 at the first time point T1. This extreme value occurs at the second time point T2. Window 710 is a time range representing the sensor's temporal error range, and the first time point T1 is located at the center of window 710. This extreme value can be a maximum or minimum value, depending on the properties of the one-dimensional signal 712. For example, if a larger value of the one-dimensional signal 712 indicates a closer distance between the preceding vehicle and the smart car 100, the time point of the maximum value can be set to the second time point T2.

[0029] In step 307, the road environment video and sensor data are aligned based on the first time point T1 and the second time point T2. For example, the difference between the first time point T1 and the second time point T2 can be calculated, and then the road environment video or sensor data can be time-shifted based on this difference. Figure 8 is a schematic diagram illustrating the time shift according to one embodiment. Here, the one-dimensional signal 712 is shifted forward on the time axis (by the same amount as the aforementioned difference), so that the maximum value of the shifted one-dimensional signal 810 is aligned with the extreme value of curve 711.

[0030] The sensor data captured by each sensor 231-233 can be aligned with the road environment video using the steps in Figure 3. This data alignment method is based on driving mode and scene data, and uses language models to process complex patterns and environments, enabling more efficient and flexible use of sensor data. For example, when driving in poor visibility and wishing to enable ACC, the driving mode can be set to ACC, and the scene (environmental description) can be set to: rainy day ahead, rainy day behind, rainy day behind, rainy day neighboring car. In this scenario, in addition to the image capture module, radar is also required. Radar performs better than the image capture module in poor weather and low light conditions, so the aforementioned data alignment method is necessary for both the image capture module and radar.

[0031] Although the present invention has been disclosed above with reference to the embodiments, they are not intended to limit the present invention. Anyone with ordinary skill in the art may make slight changes and modifications without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the scope of the appended patent applications.

[0032] 100: Smart Car 210: Processor 220: Image capture device 231~233: Sensor 301~307,501~504: Steps 400: Test screen 601,602,713:Screen 603: Bitmap 701~703: Fragments 710: Window 711: Curve 712,810: One-dimensional signal T1: First time point T2: Second time point

Claims

1. A data alignment method for a smart car, the method comprising: acquiring a road environment video via an image capture device on the smart car; determining a current driving mode of a plurality of driving modes of the smart car, and determining a first prompt and a test frame based on the current driving mode; inputting the road environment video and the first prompt into a language model to segment the road environment video into a plurality of segments, each of the segments comprising a plurality of frames; for one of the segments, calculating a similarity between each of the corresponding frames and the test frame, thereby determining a first time point of the segment; acquiring sensor data via a sensor on the smart car, and generating a one-dimensional signal based on the sensor data; obtaining an extreme value of the one-dimensional signal within a window at the first time point, wherein the extreme value occurs at a second time point; and aligning the road environment video and the sensor data based on the first time point and the second time point.

2. The data alignment method according to claim 1, wherein the driving modes include an adaptive cruise control mode and a reverse mode.

3. The data alignment method of claim 1, further comprising: Inputting the segments into an evaluation model to obtain a score; And if the score is less than or equal to a critical value, a second prompt is generated according to the score, and the road environment video and the second prompt are input into the language model to re-segment the road environment video.

4. The data alignment method according to claim 1, wherein the step of calculating the similarity between each of the corresponding frames and the test frame comprises: For a first frame among the frames, reducing the first frame to a preset size to generate a second frame; Calculating an average value of a plurality of pixels in the second frame; determining whether the pixels are greater than the average value to generate a first feature vector; and calculating a similarity between the first feature vector and a feature vector of the test frame.

5. The data alignment method according to claim 4, further comprising: It is determined whether the similarity between the first feature vector and the feature vector of the test frame is greater than a threshold value, and if so, the time point of the first frame is set as the first time point.

6. The data alignment method according to claim 1, wherein the sensor comprises a radar or a lidar.

7. The data alignment method of claim 1, further comprising: If the dimension of the sensing data is greater than 1, the sensing data is reduced in dimension to obtain the one-dimensional signal.

8. The data alignment method according to claim 7, wherein the step of reducing the dimension of the sensing data to obtain the one-dimensional signal comprises: Performing a principal component analysis on the sensing data to obtain a principal component vector; And for a time point, a length of the principal component vector corresponding to the time point is calculated as the value of the one-dimensional signal at the time point.

9. The data alignment method according to claim 1, wherein the step of aligning the road environment video and the sensing data according to the first time point and the second time point comprises: calculating a difference between the first time point and the second time point; and performing a time shift on the road environment video or the sensing data according to the difference.

10. The data alignment method according to claim 1, wherein the first prompt comprises a direction of the image capture device or an environment description.

11. A smart car, comprising: an image capture device for capturing a video of a road environment; a sensor for acquiring sensing data; and a processor electrically connected to the image capture device and the sensor, configured to perform a plurality of steps: capturing the video of the road environment; determining a current driving mode of a plurality of driving modes of the smart car, and determining a first prompt and a test frame based on the current driving mode; inputting the video of the road environment and the first prompt into a language model to segment the video of the road environment into a plurality of segments, each of the segments comprising a plurality of frames; for one of the segments, calculating a similarity between each of the corresponding frames and the test frame to thereby determine a first time point of the segment; capturing the sensing data and generating a one-dimensional signal based on the sensing data; obtaining an extreme value of the one-dimensional signal within a window at the first time point, wherein the extreme value occurs at a second time point; and aligning the video of the road environment and the sensing data based on the first time point and the second time point.

12. The smart car of claim 11, wherein the driving modes include an adaptive cruise control mode and a reverse mode.

13. The smart car of claim 11, wherein the steps further comprise: Inputting the segments into an evaluation model to obtain a score; And if the score is less than or equal to a critical value, a second prompt is generated according to the score, and the road environment video and the second prompt are input into the language model to re-segment the road environment video.

14. The smart car of claim 11, wherein the step of calculating the similarity between each of the corresponding frames and the test frame comprises: For a first frame among the frames, reducing the first frame to a preset size to generate a second frame; Calculating an average value of a plurality of pixels in the second frame; determining whether the pixels are greater than the average value to generate a first feature vector; and calculating a similarity between the first feature vector and a feature vector of the test frame.

15. The smart car of claim 14, wherein the steps further comprise: It is determined whether the similarity between the first feature vector and the feature vector of the test frame is greater than a threshold value, and if so, the time point of the first frame is set as the first time point.

16. The smart car of claim 11, wherein the sensor comprises a radar or a lidar.

17. The smart car of claim 11, wherein the steps further comprise: If the dimension of the sensing data is greater than 1, the sensing data is reduced in dimension to obtain the one-dimensional signal.

18. The smart car of claim 17, wherein the step of reducing the dimension of the sensing data to obtain the one-dimensional signal comprises: Performing a principal component analysis on the sensing data to obtain a principal component vector; And for a time point, a length of the principal component vector corresponding to the time point is calculated as the value of the one-dimensional signal at the time point.

19. The smart car of claim 11, wherein the step of aligning the road environment video and the sensing data according to the first time point and the second time point comprises: calculating a difference between the first time point and the second time point; and performing a time shift on the road environment video or the sensing data according to the difference.

20. The smart car of claim 11, wherein the first prompt comprises a direction of the image capture device or an environment description.

Images