Rearview mirror image generation method, device, vehicle-mounted equipment and readable storage medium

Abstract

The present application relates to a rearview mirror image generation method, device, vehicle-mounted equipment and readable storage medium. The method comprises: obtaining side view camera image data, rear view camera image data and rear radar point cloud data; inputting the image data and point cloud data into a pre-trained rearview mirror image generation model, fusing the image features and the point cloud features of the point cloud data through the rearview mirror image generation model to obtain multimodal feature data; inputting the multimodal feature data into an encoder module in the rearview mirror image generation model, obtaining encoding feature data matching the multimodal feature data, and performing dimensionality reduction processing on the encoding feature data to obtain dimensionality reduction feature data; inputting the dimensionality reduction feature data into a decoder module in the rearview mirror image generation model, decoupling the dimensionality reduction feature data through the decoder module, and generating a rearview mirror image of a target vehicle. The present method can improve the generation efficiency of rearview mirror images.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, vehicle-mounted equipment, computer-readable storage medium and computer program product for generating a rearview mirror image. Background Art

[0002] With the rapid development of automotive electronics and intelligent technology, electronic rearview mirrors have gradually entered the market as new vehicle-mounted equipment and are gradually replacing traditional glass rearview mirrors. With their significant advantages such as digital display, expanded field of view, and enhanced night vision, electronic rearview mirrors have significantly improved driving safety and comfort. However, once an electronic rearview mirror fails or fails, it will directly affect the driver's field of view and increase safety risks during driving.

[0003] Traditional technology often achieves physical backup and switching by adding additional hardware components or special sensors when electronic rearview mirrors malfunction or fail. However, traditional technology not only increases the overall weight, complexity and cost of the vehicle, but may also introduce new sources of failure. In addition, the data obtained through additional hardware components or special sensors will vary greatly depending on factors such as vehicle model and installation angle, and data processing needs to be combined with specific scenarios, which is not conducive to improving the efficiency of rearview mirror image generation. Summary of the invention

[0004] Based on this, it is necessary to provide a rearview mirror image generation method, device, vehicle-mounted equipment, computer-readable storage medium and computer program product that can improve the generation efficiency of the rearview mirror image in order to address the above technical problems.

[0005] In a first aspect, the present application provides a method for generating a rearview mirror image, comprising:

[0006] Acquire first image data captured by a side-view camera of the target vehicle, second image data captured by a rear-view camera, and point cloud data collected by a rear-side radar;

[0007] Inputting the first image data, the second image data, and the point cloud data into a pre-trained rearview mirror image generation model, and fusing image features corresponding to the first image data, image features corresponding to the second image data, and point cloud features of the point cloud data through the rearview mirror image generation model to obtain multimodal feature data;

[0008] Inputting the multimodal feature data into an encoder module in the rearview mirror image generation model, obtaining encoded feature data matching the multimodal feature data, and performing dimensionality reduction processing on the encoded feature data to obtain reduced dimensionality feature data;

[0009] The reduced-dimensional feature data is input into a decoder module in the rearview mirror image generation model, and the reduced-dimensional feature data is decoupled by the decoder module to generate a rearview mirror image of the target vehicle.

[0010] In one embodiment, the fusing of the image features corresponding to the first image data, the image features corresponding to the second image data, and the point cloud features of the point cloud data to obtain the multimodal feature data includes:

[0011] fusing the image features corresponding to the first image data and the image features corresponding to the second image data to obtain fused image features;

[0012] The fused image features and the point cloud features of the point cloud data are fused to obtain the multimodal feature data.

[0013] In one embodiment, fusing the image features corresponding to the first image data and the image features corresponding to the second image data to obtain the fused image features includes:

[0014] Normalizing the image features corresponding to the first image data and the image features corresponding to the second image data to obtain a first attention weight corresponding to the first image data and a second attention weight corresponding to the second image data;

[0015] According to the first attention weight and the second attention weight, the image features corresponding to the first image data and the image features corresponding to the second image data are weightedly fused to obtain the fused image features.

[0016] In one embodiment, fusing the fused image features and the point cloud features of the point cloud data to obtain the multimodal feature data includes:

[0017] Normalizing the fused image feature data and the point cloud feature to obtain a third attention weight corresponding to the fused data mapping feature and a fourth attention weight corresponding to the point cloud feature;

[0018] The fused image features and the point cloud features are weightedly fused according to the third attention weight and the fourth attention weight to obtain the multimodal feature data.

[0019] In one embodiment, the obtaining of the coded feature data matching the multimodal feature data includes:

[0020] Performing convolution processing on the multimodal feature data, and normalizing the convolution processing result to obtain normalized feature data;

[0021] The normalized feature data is mapped to the encoded feature data using a preset activation function.

[0022] In one embodiment, the performing dimensionality reduction processing on the encoded feature data to obtain the reduced dimensionality feature data includes:

[0023] Decomposing a sparse matrix from the encoded feature data using a sparse factor;

[0024] The coded feature data is subjected to dimensionality reduction processing according to the sparse matrix to obtain the reduced dimensionality feature data.

[0025] In one embodiment, the decoder module includes a pre-trained residual network, and the decoupling of the reduced-dimensional feature data to generate a rearview mirror image of the target vehicle includes:

[0026] Mapping the dimension-reduced feature data to a target space to obtain mapping features;

[0027] Performing upsampling processing on the mapping features to obtain sampling data;

[0028] Inputting the encoded feature data and the sampled data into the pre-trained residual network to obtain rearview mirror image data for the target car;

[0029] The rearview mirror image is generated according to the rearview mirror image data.

[0030] In one embodiment, generating the rearview mirror image according to the rearview mirror image data includes:

[0031] generating warning data for the target vehicle according to the first image data, the second image data and the point cloud data;

[0032] Fusing the rearview mirror image data and the warning data to obtain target image data;

[0033] The rearview mirror image is generated according to the target image data.

[0034] In one embodiment, the method further comprises:

[0035] Acquire first sample image data taken by a side-view camera of a sample car, second sample image data taken by a rear-view camera, and sample point cloud data collected by a rear-side radar, and acquire a rear-view mirror reference image corresponding to the sample car;

[0036] Inputting the first sample image data, the second sample image data and the sample point cloud data into a rearview mirror image generation model to be trained to obtain a rearview mirror image of the sample car;

[0037] The rearview mirror image generation model to be trained is trained according to the rearview mirror reference image and the rearview mirror image of the sample car.

[0038] In a second aspect, the present application also provides a rearview mirror image generating device, comprising:

[0039] An acquisition module, used to acquire first image data taken by a side-view camera of a target vehicle, second image data taken by a rear-view camera, and point cloud data collected by a rear-side radar;

[0040] a fusion module, configured to input the first image data, the second image data, and the point cloud data into a pre-trained rearview mirror image generation model, and fuse image features corresponding to the first image data, image features corresponding to the second image data, and point cloud features of the point cloud data through the rearview mirror image generation model to obtain multimodal feature data;

[0041] An encoding module, used for inputting the multimodal feature data into an encoder module in the rearview mirror image generation model, obtaining encoding feature data matching the multimodal feature data, and performing dimensionality reduction processing on the encoding feature data to obtain dimensionality reduction feature data;

[0042] A decoding module is used to input the reduced-dimensional feature data into a decoder module in the rearview mirror image generation model, and decouple the reduced-dimensional feature data through the decoder module to generate a rearview mirror image of the target car.

[0043] In a third aspect, the present application further provides a vehicle-mounted device, wherein the vehicle-mounted device comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the steps of the above method are implemented.

[0044] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0045] In a fifth aspect, the present application further provides a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0046] The above-mentioned rearview mirror image generation method, device, vehicle-mounted equipment, computer-readable storage medium and computer program product obtain the first image data taken by the side-view camera of the target car, the second image data taken by the rear-view camera and the point cloud data collected by the rear-side radar, thereby using the side-view camera, rear-view camera and rear-side radar set on the target car to timely obtain the image data and point cloud data of the environment in which the target car is located, and provide a data basis for the subsequent rearview mirror image; the first image data, the second image data and the point cloud data are input into a pre-trained rearview mirror image generation model, and the image features corresponding to the first image data, the image features corresponding to the second image data and the point cloud features of the point cloud data are fused through the rearview mirror image generation model to obtain multimodal feature data, thereby cross-modally fusion of the image data and the point cloud data into a unified representation, and when the image data and the point cloud data complement each other, feature data containing multimodal information is obtained, thereby improving data processing efficiency; the multimodal feature data is input into the encoder module in the rearview mirror image generation model to obtain multimodal feature data. The method comprises the following steps: first, encoding and reducing the dimensionality of the multimodal feature data, and performing dimensionality reduction processing on the encoded feature data to obtain reduced dimensionality feature data, thereby encoding and reducing the dimensionality of the multimodal feature data, removing redundant information, retaining key features, and improving the accuracy of the reduced dimensionality feature data; inputting the reduced dimensionality feature data into the decoder module in the rearview mirror image generation model, decoupling the reduced dimensionality feature data through the decoder module, and generating the rearview mirror image of the target car, thereby generating a rearview mirror image as similar as possible to the real image based on the reduced dimensionality feature data by using the decoder, and being able to perform cross-modal fusion of the image data and the point cloud data based on the first image data taken by the side view camera of the target car, the second image data taken by the rear view camera, and the point cloud data collected by the rear radar, and performing encoding, dimensionality reduction and decoding on the cross-modal features, and efficiently generating a rearview mirror image with the same viewing angle as the electronic rearview mirror, thereby improving the generation efficiency of the rearview mirror image, and then using the generated rearview mirror image to provide the driver with an image from the rearview mirror perspective when the electronic rearview mirror fails or fails, to ensure driving safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 is a schematic flow chart of a method for generating a rearview mirror image in an embodiment;

[0049] Figure 2A schematic diagram of a data alignment process in an embodiment;

[0050] Figure 3 A schematic diagram of a process of encoding and decoding multimodal feature data in an embodiment;

[0051] Figure 4 A schematic diagram of a process of generating a rearview mirror image in one embodiment;

[0052] Figure 5 is a structural block diagram of a rearview mirror image generating device in one embodiment;

[0053] Figure 6 The figure is a diagram of the internal structure of a vehicle-mounted device in one embodiment. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0055] In an exemplary embodiment, Figure 1 As shown, a rearview mirror image generation method is provided, and the method is applied to a vehicle-mounted device as an example for description, including the following steps S102 to S106. Among them:

[0056] Step S102, obtaining first image data taken by a side view camera of the target vehicle, second image data taken by a rear view camera, and point cloud data collected by a rear radar.

[0057] The target car may refer to a car equipped with a side view camera, a rear view camera and a rear radar. In actual applications, the target car is also equipped with an electronic rearview mirror.

[0058] Among them, the side-view camera may refer to a camera set on the side surface of the target car. In actual applications, the side-view camera may include a side-view fisheye camera. The setting position of the side-view camera may include but is not limited to below the electronic rearview mirror and the door frame.

[0059] The first image data may refer to image data captured by a side-view camera of the target vehicle. In practical applications, the first image data may represent the environment to the side of the target vehicle in the form of an image.

[0060] Among them, the rearview camera may refer to a camera installed on the rear surface of the target car. In actual applications, the rearview camera may include a rearview fisheye camera. The installation position of the rearview camera may include but is not limited to the vicinity of the license plate light, the inside of the bumper, the rear windshield and the trunk lid.

[0061] The second image data may refer to image data captured by a rear-view camera of the target vehicle. In practical applications, the second image data may represent the environment behind the target vehicle in the form of an image.

[0062] Among them, the rear side radar may refer to a radar sensor arranged on the rear surface of the target car. In actual applications, the rear side radar may include a millimeter wave radar, and the setting position of the rear side radar may include but is not limited to the side of the rear bumper.

[0063] Among them, point cloud data can refer to data that characterizes information such as the position and speed of objects within the detection range corresponding to the rear radar. In practical applications, point cloud data can include (three-dimensional) coordinate data, object reflection area (Radar Cross-Section, RCS) and object speed (Doppler speed / Doppler speed / absolute value of Doppler speed).

[0064] As an example, when the electronic rearview mirror of the target car malfunctions or fails, it will directly affect the driver's field of vision and increase the safety risks during driving (especially when performing operations such as changing lanes, overtaking, and reversing, the driver's accurate perception of the surrounding environment is essential). The failure of the electronic rearview mirror is different from the sensor failure in the Advanced Driving Assistance System (ADAS). The sensor failure in the automated driving assistance system only limits some advanced driving assistance functions, and the driver can still maintain vehicle control by relying on his own observation and judgment. At this time, the vehicle control is switched from intelligent driving to human driving, but the failure of the electronic rearview mirror directly deprives the driver of vital visual information about the vehicle's surrounding environment, especially in key driving operations such as changing lanes, overtaking, reversing, and real-time road condition monitoring. The driver's safety situation awareness ability is severely weakened, which may lead to major safety hazards until the vehicle cannot be driven safely. When the electronic rearview mirror fails, the traditional technology only issues an alarm, but still does not control the vehicle (such as slowing down, braking, calling redundant mechanisms, etc.), and the driver needs to take over manually to handle such emergencies. Traditional technology adds additional hardware components or dedicated sensors to achieve physical backup and switching, but this method will increase the overall weight, complexity and cost of the vehicle, and may also introduce new sources of failure, which is contrary to the trend of modern automobiles pursuing lightweight, integrated and economical performance. In existing vehicle systems, sensors in intelligent driving assistance systems (such as side cameras and rearview cameras, etc.) have been widely deployed. The above (intelligent driving) sensors are used to provide environmental perception capabilities and support autonomous driving. Function, although the above-mentioned (intelligent driving) sensor is not designed to completely replace the function of the electronic rearview mirror, the above-mentioned (intelligent driving) sensor does have the ability to capture the side and rear environment of the vehicle. Therefore, the above-mentioned rearview mirror image generation method can be used to process the first image data taken by the side view camera of the target car, the second image data taken by the rear view camera, and the point cloud data collected by the rear radar without increasing the additional hardware cost, so as to generate a rearview mirror image with the same viewing angle as the rearview mirror. Specifically, the vehicle-mounted device can obtain the first image data taken by the side view camera of the target car, the second image data taken by the rear view camera, and the point cloud data collected by the rear radar according to a preset period.

[0065] Step S104: input the first image data, the second image data, and the point cloud data into a pre-trained rearview mirror image generation model, and fuse the image features corresponding to the first image data, the image features corresponding to the second image data, and the point cloud features of the point cloud data through the rearview mirror image generation model to obtain multimodal feature data.

[0066] Among them, the rearview mirror image generation model may refer to a model for generating an image with a rearview mirror perspective. In practical applications, the rearview mirror image generation model may generate a rearview mirror image by processing the first image data, the second image data, and the point cloud data. It can be understood that the above rearview mirror image may refer to the image of the left rearview mirror and the right rearview mirror in the side rearview mirror system of the car. The reasons for using the image of the left rearview mirror and the right rearview mirror as the rearview mirror image may include: ① Difference in field of vision requirements: The central rearview mirror between the main driver and the co-driver is mainly used to observe the situation directly behind the vehicle, while the left and right side rearview mirrors need to provide a wider field of vision, including the side rear and objects relatively far away from the vehicle body, which is particularly important for key vehicle control actions; ② Frequency of use and functional focus: In normal driving, the frequency of use of the central rearview mirror is relatively low, and its main function is to assist in observing the rear traffic situation, rather than for precise vehicle control. In contrast, the side rearview mirror is used more frequently in the above key actions and contributes more significantly to safety; ③ Technical difficulty: When the central rearview mirror fails, it can be completely replaced by the rearview camera of the intelligent driving system. The technical challenges of left and right side mirrors are greater because they need to integrate information from different angles and distances, and need to provide clear and reliable vision in all light and weather conditions. This challenge makes the potential for improving side mirrors through technological innovation greater; ④ Safety first: In safety design, priority is usually given to key components that are most likely to be involved in avoiding collisions and accidents. Left and right side mirrors play a vital role in preventing side collisions and reducing blind spots; ⑤ Structural and spatial limitations: The position and structure of the central rearview mirror are relatively fixed, while the left and right side mirrors face more design and space limitations.

[0067] The multimodal feature data may refer to data obtained by fusing feature data corresponding to the first image data, the second image data, and the point cloud data.

[0068] As an example, the vehicle-mounted device can input the first image data, the second image data and the point cloud data into a pre-trained rearview mirror image generation model. The rearview mirror image generation model can perform stage-by-stage feature fusion of the image features corresponding to the first image data, the image features corresponding to the second image data and the point cloud features of the point cloud data according to the weights corresponding to the image features and the point cloud features, to obtain multimodal feature data.

[0069] Step S106, inputting the multimodal feature data into an encoder module in the rearview mirror image generation model, obtaining encoded feature data matching the multimodal feature data, and performing dimensionality reduction processing on the encoded feature data to obtain reduced dimensionality feature data.

[0070] Among them, the encoder module may refer to a model for feature encoding of multimodal feature data.

[0071] The encoded feature data may refer to data output by the encoder module based on the multimodal feature data. In practical applications, the encoded feature data may include feature codes corresponding to the multimodal feature data.

[0072] The dimension-reduced feature data may refer to data obtained after dimension reduction processing is performed on the encoded feature data. In practical applications, the dimension-reduced feature data may include compressed feature codes corresponding to the encoded feature data.

[0073] As an example, the vehicle-mounted device can input multimodal feature data into an encoder module in a pre-trained rearview mirror image generation model. The encoder module can perform convolution processing and normalization processing on the multimodal feature data to obtain encoded feature data that matches the multimodal feature data. The vehicle-mounted device can also perform dimensionality reduction processing on the encoded feature data through the rearview mirror image generation model to obtain reduced dimensionality feature data corresponding to the encoded feature data that matches the multimodal feature data.

[0074] Step S108, inputting the reduced-dimensionality feature data into a decoder module in the rearview mirror image generation model, decoupling the reduced-dimensionality feature data through the decoder module, and generating a rearview mirror image of the target vehicle.

[0075] Among them, the decoder module may refer to a model used to perform feature decoding on reduced-dimensional feature data.

[0076] Among them, the rearview mirror image may refer to the images of the left rearview mirror and the right rearview mirror in the side rearview mirror system of the car. In actual applications, the rearview mirror image may include the image from the perspective of the left electronic rearview mirror of the car and / or the image from the perspective of the right electronic rearview mirror of the car.

[0077] As an example, the on-board device can input the reduced dimension feature data into the decoder module in the rearview mirror image generation model. Since the reduced dimension feature data is obtained by encoding and reducing the dimension of the multimodal feature data, the decoder module can decouple the reduced dimension feature data, thereby ensuring that the sensor data of each sensor (such as the side view camera, rear view camera and rear radar, etc.) can be independently and accurately processed and analyzed. Then the decoder module outputs the rearview mirror image data, and the on-board device outputs the rearview mirror image based on the rearview mirror image data, and displays the rearview mirror image to the driver of the target car through the display interface of the target car (such as the vehicle's internal display screen).

[0078] The above-mentioned rearview mirror image generation method can cross-modally fuse the image data and the point cloud data based on the first image data taken by the side-view camera of the target car, the second image data taken by the rear-view camera and the point cloud data collected by the rear radar, and encode, reduce the dimension and decode the cross-modal features, so as to efficiently generate a rearview mirror image with the same viewing angle as the electronic rearview mirror, thereby improving the efficiency of rearview mirror image generation. When the electronic rearview mirror malfunctions or fails, the generated rearview mirror image is used to provide the car driver with an image from the rearview mirror perspective to ensure driving safety.

[0079] In an exemplary embodiment, image features corresponding to the first image data, image features corresponding to the second image data, and point cloud features of the point cloud data are fused to obtain multimodal feature data, including: fusing image features corresponding to the first image data and image features corresponding to the second image data to obtain fused image features; fusing the fused image features and point cloud features of the point cloud data to obtain multimodal feature data.

[0080] The fused image feature may refer to data obtained by fusing the image feature corresponding to the first image data and the image feature corresponding to the second image data.

[0081] As an example, the input data of the rearview mirror image generation model includes image data (such as camera data) and point cloud data (such as millimeter wave radar data). Data of different modalities often have different characteristics, ranges and formats. Multimodal data fusion requires that data of different modalities can correspond and coordinate with each other to some extent, so that the rearview mirror image generation model can effectively extract and integrate information from multiple modalities. Directly inputting the original image data and point cloud data into the rearview mirror image generation model may cause the performance of the rearview mirror image generation model to be poor or difficult to train. Therefore, before fusing the image features corresponding to the first image data, the image features corresponding to the second image data, and the point cloud features of the point cloud data, the vehicle-mounted device can perform data preprocessing on the first image data, the second image data, and the point cloud data.

[0082] In some embodiments, the rearview mirror image generation model may include a data preprocessing layer (layer A), which performs preprocessing such as dedistortion, format conversion, normalization, and scaling on the input original side view camera data (first image data), the original rear view camera data (second image data), and the original rear millimeter-wave radar data (point cloud data). The above data preprocessing layer will be used in both the training and application stages of the rearview mirror image generation model. The format of the original first image data (such as side view fisheye camera data) input into the rearview mirror image generation model is usually raw image data (such as JPEG, PNG, TIFF, etc.). The side view fisheye camera data may contain strong distortion, and the data type of the side view fisheye camera data is usually 8-bit unsigned integer. The format of the original second image data (such as rear-view fisheye camera data) input to the rearview mirror image generation model is usually raw image data (such as JPEG, PNG, TIFF, etc.). The rear-view fisheye camera data may contain strong distortion. The data type of the rear-view fisheye camera data is usually an 8-bit unsigned integer (uint8) representing the intensity or color value of the pixel. The size of the rear-view fisheye camera data depends on the camera resolution and image size. The format of the original point cloud data (such as rear-side millimeter-wave radar data) input to the rearview mirror image generation model is point cloud data, which contains the three-dimensional coordinates (x, y, z), RCS (object reflection area), and Doppler (object speed) of each point. The data type is floating point (float32 / float64) to represent precise coordinates and attributes.

[0083] The format of the preprocessed first image data output by the data preprocessing layer based on the original first image data is still image data, but the distortion is corrected, and the data type is converted to a floating point number (float32 / float64) for subsequent mathematical operations. The data size will be scaled according to the input requirements of the model to match the input size expected by the model; The format of the preprocessed second image data output by the data preprocessing layer based on the original second image data is still image data, but the distortion is corrected, and the data type is converted to a floating point number (float32 / float64) for subsequent mathematical operations. The data size will be scaled according to the input requirements of the model to match the input size expected by the model; The format of the preprocessed point cloud data output by the data preprocessing layer based on the original point cloud data is converted into a feature vector (by extracting key features), which is essentially a numerical array used to represent the key information of the data. The data type is also converted to a floating point number (float32 / float64) for model processing, where the point cloud data contains the three-dimensional coordinates of each point (x, y, z), RCS (radar cross section, indicating the ability of an object to reflect electromagnetic waves) and Doppler (Doppler speed, indicating the radial speed of an object), the rearview mirror image generation model can perform ① data cleaning; ② feature extraction; ③ feature combination on the point cloud data, wherein ① data cleaning includes removing noise points or invalid points according to thresholds or statistical methods. Noise points or invalid points may be caused by radar errors, multipath effects or other reasons, such as removing points with abnormally low or high RCS values ​​or removing points with abnormally high speeds. ② Feature extraction can include extracting the following features for each point: 1. Position features: some combination of three-dimensional coordinates can be used, such as range (distance to the radar), azimuth, pitch angle, etc.; 2. RCS features: directly use the RCS value or perform some transformation on it (such as logarithmic transformation); 3. Doppler features: directly use the Doppler speed value or its absolute value to indicate the speed of the object's movement. ③ Feature combination can include combining the extracted features into a feature vector. In practical applications, the above feature vector can be expressed as:

[0084] .

[0085] Among them, range is the distance from the point to the radar. azimuth and elevation are the azimuth and elevation of the point, respectively, obtained by coordinate transformation. RCS_log is the result of taking the logarithm of the RCS value, which is used to improve the distribution of features. |Doppler| is the absolute value of the Doppler velocity. Take a simplified example to illustrate: the radar is located at the origin (0, 0, 0), for point 1 (x, y, z) = (10, 20, 30), RCS = 1.5, Doppler = -5m / s, and the range Range is the straight-line distance from the radar to the target, which can be calculated using the Euclidean distance formula of three-dimensional coordinates. In practical applications, the Euclidean distance formula can be expressed as:

[0086] .

[0087] The azimuth and elevation angles are usually determined based on the radar's coordinate system and installation method. Without specific information about the radar installation and scanning method, these two angles cannot be directly calculated. However, for the sake of example, a simplified scenario can be assumed where the radar is located at the origin (0, 0, 0), and the azimuth angle is the angle of the target on the xy plane (measured counterclockwise from the positive x-axis), and the elevation angle is the angle between the target and the xy plane. In specific implementation, the calculation expression for the azimuth angle can be expressed as:

[0088] .

[0089] The calculation expression of the pitch angle Elevation can be expressed as:

[0090] .

[0091] The RCS value can use the natural logarithm to improve the distribution of features. In the specific implementation, the expression of the RCS value RCSlog can be expressed as:

[0092] .

[0093] Doppler speed Doppler can directly use its value as a feature, because the absolute value of Doppler speed indicates the speed of the target's movement, and the positive and negative values ​​of Doppler speed represent the direction of the target. When the target is close to the radar, the target speed is positive, and when the target is far away from the radar, the target speed is negative. In summary, the feature vector of point 1 (point cloud feature) can be expressed as:

[0094] .

[0095] During specific implementation, in some computing or machine learning frameworks, it may be necessary to convert the azimuth and elevation into radians (the above azimuth and elevation are given in degrees). In addition, in practical applications, depending on the specific radar system and mission requirements, it may be necessary to normalize or standardize the features.

[0096] After the vehicle-mounted device preprocesses the first image data, the second image data and the point cloud data through the rearview mirror image output model, in order to ensure that the data from different sensors (side view camera, rear view camera and rear millimeter wave radar) can be accurately matched in time and space during multimodal feature fusion, so as to perform accurate multimodal fusion, the vehicle-mounted device can align the preprocessed data. The rearview mirror image output model can include a multimodal fusion layer (layer B). The multimodal fusion layer can fuse the preprocessed different modal data into a unified representation so that subsequent layers can simultaneously utilize information from multiple sensors; it can also extract cross-modal correlation features to enhance the perception ability of the model. The multimodal fusion layer can realize data alignment. The multimodal fusion layer will be used in both the training and application stages of the rearview mirror image generation model, such as Figure 2As shown, a flow chart of data alignment is provided. The vehicle-mounted equipment can sequentially perform a. timestamp synchronization; b. sampling rate matching; c. spatial coordinate conversion; d. feature point matching; e. verification and adjustment on the pre-processed data. Since different sensors may have time deviations, GPS time or network time protocol (NTP) can be used to calibrate the timestamps of each sensor to achieve timestamp synchronization; sampling rate matching is to check whether the sampling rates of each sensor are consistent. If not, a unified sampling rate needs to be determined, and the data needs to be resampled or interpolated. In practical applications, since the sampling rates of different sensors of different models are likely to be inconsistent, frame rate adjustment is required. In specific implementation, for video data (such as image data), the frame rates of different cameras may need to be adjusted to the same to ensure the correspondence between video frames. For radar data, its sampling frequency may need to be matched with the video frame rate so that each frame of video has corresponding radar data; since different sensors are installed at different positions on the vehicle, the viewing angles and coordinate systems of sensor data are different. Therefore, spatial coordinate conversion is required to convert the data of each sensor into a unified coordinate system. In specific implementation, spatial coordinate conversion involves camera calibration, radar calibration, and calibration of the relative position relationship between them. Through calibration, the conversion matrix between each sensor can be obtained, so that the data of one sensor can be converted to the coordinate system of another sensor; for image data and radar data, spatial alignment is achieved through feature point extraction and matching methods; after completing data alignment, the accuracy of alignment needs to be verified. In practical applications, dynamic alignment issues during vehicle motion also need to be considered to ensure stable alignment of data under different motion states.

[0097] After the data alignment process, the vehicle-mounted equipment can perform feature extraction on the first image data, the second image data and the point cloud data through the multimodal fusion layer, and then fuse the image features corresponding to the first image data and the image features corresponding to the second image data to obtain fused image features, and fuse the fused image features with the point cloud features of the point cloud data to obtain multimodal feature data.

[0098] In this embodiment, by fusing the image features corresponding to the first image data and the image features corresponding to the second image data, a fused image feature is obtained; and by fusing the fused image feature and the point cloud feature of the point cloud data, multimodal feature data is obtained. The image features of the image data and the point cloud features of the point cloud data can be fused into a unified representation (multimodal feature data), which facilitates the subsequent use of the multimodal feature data to simultaneously analyze the image data and the point cloud data, extract cross-modal correlation features, enhance the perception ability of the rearview mirror image generation model, obtain an accurate rearview mirror image, and improve the accuracy of the rearview mirror image.

[0099] In one of the exemplary embodiments, image features corresponding to first image data and image features corresponding to second image data are fused to obtain fused image features, including: normalizing image features corresponding to the first image data and image features corresponding to the second image data to obtain first attention weights corresponding to the first image data and second attention weights corresponding to the second image data; and weightedly fusing image features corresponding to the first image data and image features corresponding to the second image data according to the first attention weight and the second attention weight to obtain fused image features.

[0100] The first attention weight may refer to information representing the importance of the image feature of the first image data relative to the fused image feature.

[0101] The second attention weight may refer to information representing the importance of the image features of the second image data relative to the fused image features.

[0102] As an example, the vehicle-mounted device may use a multimodal fusion layer in the rearview mirror image generation model to extract image features of the image data and point cloud features of the point cloud data using a feature extraction network. Specifically, the vehicle-mounted device may use a convolutional neural network to extract image features F of the first image data. side (such as image feature map), a convolutional neural network is used to extract the image feature F of the second image data rear (such as image feature map), the point cloud feature V of the point cloud data is extracted using the fully connected layer radar (such as feature vector), and then the vehicle-mounted device uses the multimodal fusion layer in the rearview mirror image generation model to generate the image feature F corresponding to the first image data. side The image feature F corresponding to the second image data rear Perform feature fusion in the first stage (such as the early fusion stage). Specifically, the multimodal fusion layer can combine the attention mechanism to perform weighted fusion of image features. In practical applications, for the early fusion stage, the attention mechanism of the image feature map is mainly used to determine the importance of feature maps of different perspectives (such as the first image data and the second image data) during fusion. At this time, the weight setting is usually based on the following steps: ① Feature mapping; ② Softmax normalization; ③ Feature fusion of feature maps, where:

[0103] Feature Mapping: For each camera view, the feature map (F side and F rear ), firstly, it is mapped to a feature vector of the same dimension through a learnable weight matrix (W), and then a learnable bias term (b) is added. The first image data after feature mapping can be expressed as Z side , the second image data after feature mapping can be expressed as Z rearIn the specific implementation, Z side It can be expressed as:

[0104] .

[0105] Z rear It can be expressed as:

[0106] .

[0107] Softmax normalization: The mapped feature vector is normalized by the softmax function to obtain the attention weight (A) of each feature map. The first attention weight A side It can be expressed as:

[0108] .

[0109] The second attention weight A rear It can be expressed as:

[0110] .

[0111] It should be noted that the softmax function ensures that the sum of all weights is 1, and each weight value represents the importance score of the corresponding feature map.

[0112] Feature fusion of feature maps: Apply attention weights to each feature map and perform weighted summation to achieve fusion. In summary, the fused image feature F early_att It can be expressed as:

[0113] .

[0114] in, It can represent element-wise multiplication. Through the above image feature fusion method, in the early fusion stage, the attention mechanism can automatically learn the importance of camera features from different perspectives and dynamically assign weights according to the actual situation, so that the fused image features after fusion are more focused on information that is beneficial to the task. During the training process, the rearview mirror image generation model will optimize the first attention weight and the second attention weight according to the objective function, so that the model has better performance when processing multimodal data.

[0115] In this embodiment, the first attention weight corresponding to the first image data and the second attention weight corresponding to the second image data are obtained by normalizing the image features corresponding to the first image data and the second attention weight. According to the first attention weight and the second attention weight, the image features corresponding to the first image data and the image features corresponding to the second image data are weightedly fused to obtain fused image features. The dynamic allocation weights of the first image data and the second image data can be accurately obtained based on the attention mechanism, so that the generated fused image features pay more attention to information that is more beneficial to the rearview mirror image generation task, thereby improving the accuracy of the fused image features.

[0116] In one of the exemplary embodiments, fused image features and point cloud features of point cloud data are fused to obtain multimodal feature data, including: normalizing the fused image feature data and the point cloud features to obtain a third attention weight corresponding to the fused data mapping feature and a fourth attention weight corresponding to the point cloud feature; and weightedly fusing the fused image features and the point cloud features according to the third attention weight and the fourth attention weight to obtain multimodal feature data.

[0117] The third attention weight may refer to information characterizing the importance of the fused image feature relative to the multimodal feature data.

[0118] Among them, the fourth attention weight may refer to information representing the importance of the point cloud features of the point cloud data relative to the multimodal feature data.

[0119] As an example, the vehicle-mounted device uses the multimodal fusion layer in the rearview mirror image generation model to fusion image feature F early_att And the point cloud feature V of the point cloud data radar The feature fusion of the second stage (such as the late fusion stage) is performed. Specifically, the multimodal fusion layer can combine the attention mechanism to perform weighted fusion of multimodal features on the fused image features and the point cloud features. In practical applications, the multimodal fusion layer can first project the fused image features and the point cloud features to the same dimension. The fused image features after projection can be expressed as F late_pre , the projected point cloud feature can be expressed as V late_pre , and then calculate the cross-modal attention weights. Specifically, the point cloud features V radar Projecting to the plane corresponding to the fused image feature is used as an example to illustrate the attention weight A cross The calculation expression can be expressed as:

[0120] .

[0121] Among them, K is the vector transformed from another modal feature, and the point cloud feature V radarWhen projected onto the plane corresponding to the fused image feature, K can refer to the point cloud feature V radar After the linear layer, dk can refer to the square root of the feature dimension for normalization. At this time, the third attention weight can be 1, and the fourth attention weight can be expressed as A cross Finally, the multimodal fusion layer applies cross-modal attention weights (such as the third attention weight and the fourth attention weight) to perform multimodal fusion, applies the attention weights to the respective feature data, and performs weighted summation to achieve fusion and obtain the multimodal feature F late_att In practical applications, the multimodal feature F late_att It can be expressed as:

[0122] .

[0123] In the late fusion stage, the cross-modal attention mechanism comprehensively considers image features and radar features, enhancing the model's ability to understand and utilize multimodal data. Throughout the process, the model automatically learns how to assign attention weights through training to achieve the best fusion of different modal data.

[0124] In this embodiment, the first attention weight corresponding to the first image data and the second attention weight corresponding to the second image data are obtained by normalizing the image features corresponding to the first image data and the image features corresponding to the second image data; according to the first attention weight and the second attention weight, the image features corresponding to the first image data and the image features corresponding to the second image data are weightedly fused to obtain fused image features, which can accurately obtain the dynamically allocated weights of the fused image features and the point cloud features based on the attention mechanism, achieve the optimal fusion of different modal data, and improve the accuracy of multimodal feature data.

[0125] In one of the exemplary embodiments, the encoding feature data matching the multimodal feature data is obtained, including: performing convolution processing on the multimodal feature data and normalizing the convolution processing result to obtain normalized feature data; and using a preset activation function to map the normalized feature data to the encoding feature data.

[0126] The convolution processing result may refer to data obtained after performing convolution processing on multimodal feature data.

[0127] Among them, the normalized feature data may refer to data obtained after normalizing the convolution processing results corresponding to the multimodal feature data.

[0128] The preset activation function may refer to a pre-set function for mapping normalized feature data. In practical applications, the preset activation function may include a swish activation function.

[0129] As an example, Figure 3 As shown, a schematic diagram of a process for encoding and decoding multimodal feature data is provided. The vehicle-mounted device can obtain the encoded feature data matching the multimodal feature data through the encoder module in the rearview mirror image generation model. Specifically, the encoder module can include a modular hybrid feature extraction layer. The modular hybrid feature extraction layer includes a new convolution module. The convolution module combines three parts: traditional deep convolution, group normalization, and an adaptive activation function. The modular hybrid feature extraction layer can more flexibly adjust the number and grouping strategy of convolution kernels, optimize the balance between computational efficiency and model capacity, and enhance the nonlinear expression ability of the model. For example: if the input multimodal feature data X (such as F late_att ) can be expressed as:

[0130] .

[0131] Among them, H can represent height, W can represent width, and C can represent the number of channels. The construction steps of the above new convolution module may include: ① depth convolution; ② point convolution; ③ group normalization; ④ adaptable activation function.

[0132] Among them, deep convolution: convolution is applied independently to each input channel, without increasing the number of channels, and filtering is performed only in the spatial dimension. If a convolution kernel of size k*k is used, then:

[0133] .

[0134] in, It can refer to the result of deep convolution. It can refer to the convolution kernel of depthwise convolution, and * can represent the convolution operation.

[0135] Point convolution: Use point convolution to increase nonlinear expression capabilities and interactions between channels:

[0136] .

[0137] Among them, Y can be the result of point convolution processing, It can be the weight of the point convolution, It can refer to the number of output channels, and b can refer to the bias term.

[0138] Group normalization: Group normalization is applied on the basis of deep convolution and point convolution to stabilize the training process and reduce the dependence on batch size. Assuming that the channels are divided into G groups, combined with the above deep convolution and point convolution processes, the formula of group normalization GN can be expressed as:

[0139] .

[0140] Among them, μ c and σ c 2 are the mean and variance of the cth channel in the group, γ c and β c are learnable scaling and offset factors and ϵ is a small constant for numerical stability.

[0141] Adaptable activation function: Apply the adaptive swish activation function to enhance the nonlinear expression ability of the model. The swish activation function can be expressed as:

[0142] .

[0143] Among them, 𝜎 can represent the sigmoid function, and 𝛽 can be a learnable parameter in the Swish function, which increases the flexibility of the activation function. In practical applications, the above formula can also be simplified to:

[0144] .

[0145] Among them, Z represents the output after being processed by the Swish function.

[0146] By integrating the above-mentioned deep convolution, point convolution, group normalization and adaptive activation function, we can obtain the modular hybrid depth separable convolution process corresponding to the modular hybrid feature extraction layer. This modular hybrid depth separable convolution process not only optimizes the computational efficiency and reduces the number of model parameters, but also enhances the expression and generalization capabilities of the model through group normalization and adaptive activation function. It is particularly suitable for resource-sensitive applications. The data processing process of the modular hybrid feature extraction layer can be expressed as:

[0147] .

[0148] In this embodiment, normalized feature data is obtained by performing convolution processing on the multimodal feature data and normalizing the convolution processing result; and a preset activation function is used to map the normalized feature data to encoded feature data, which can optimize computing efficiency, reduce the number of model parameters, and enhance the expressiveness and generalization capabilities of the model, thereby accurately extracting the encoded feature data corresponding to the multimodal feature data and improving the accuracy of the encoded feature data.

[0149] In one of the exemplary embodiments, the coded feature data is subjected to dimensionality reduction processing to obtain the reduced-dimensional feature data, including: decomposing a sparse matrix from the coded feature data using a sparse factor; and subjecting the coded feature data to dimensionality reduction processing according to the sparse matrix to obtain the reduced-dimensional feature data.

[0150] The sparse factor may refer to an indicator used to measure the ratio of non-zero elements to the total number of elements in a matrix. In practical applications, the sparse factor may include data used to perform matrix decomposition on encoded feature data.

[0151] The sparse matrix may refer to data obtained by performing sparse factor decomposition on the encoded feature data.

[0152] As an example, Figure 3 As shown, a schematic diagram of the process of encoding and decoding multimodal feature data is provided. The encoder module may also include a lightweight adaptive feature dimensionality reduction layer. The lightweight adaptive feature dimensionality reduction layer can reduce the dimension based on the high-dimensional feature Z (multimodal feature) obtained by the modular hybrid feature extraction layer, while retaining the main information components. The lightweight adaptive feature dimensionality reduction layer may include sparse factor decomposition and dynamic channel pruning, wherein the sparse factor decomposition can significantly reduce the feature dimension and effectively extract the key information in the data to meet the dual requirements of the vehicle-mounted system for efficiency and accuracy. In the specific implementation, the high-dimensional feature Z output by the modular hybrid feature extraction layer is regarded as the product of a low-rank matrix and a sparse matrix to achieve dimensionality reduction. In practical applications, a lightweight decomposition network layer can be designed, which performs operations similar to matrix decomposition, but adds sparsity constraints. For example: suppose , where A is a low-rank matrix representing the basis representation of the feature, and S is a sparse matrix representing the contribution of each input feature to the basis. This process can be achieved by optimizing the min target. The calculation formula of the min target can be expressed as:

[0153] .

[0154] in, can refer to the Frobenius norm, It can refer to the LI norm, and λ can refer to the sparsity factor, which is used to control the sparsity.

[0155] Dynamic channel pruning can implement channel-level pruning on feature maps. Through dynamic channel pruning, channels that contribute less to the final task are removed to reduce the computational burden. In specific implementation, the difference between dynamic channel pruning and most channel pruning is that the pruning method of dynamic channel pruning is "dynamic". The "dynamic" is reflected in: after sparse factor decomposition, the channel pruning is guided by the sparsity of S. In practical applications, the L1 norm of the S vector of each channel is calculated as a measure of channel importance. In each training or inference cycle, the top k highest-scoring channels are retained (these channels are considered to contribute the most to the final task), and the remaining channels are temporarily dormant (the calculation and transmission of the corresponding feature maps are skipped), and the reduced-dimensional feature data (compressed feature representation) Z is finally obtained. low_dim,Dynamic channel pruning can dynamically adjust according to the changes of input data, making the algorithm robust.

[0156] In this embodiment, a sparse matrix is ​​decomposed from the encoded feature data by utilizing a sparse factor; and the encoded feature data is subjected to dimensionality reduction processing according to the sparse matrix to obtain reduced-dimensional feature data, which can reduce the feature dimension and effectively extract key information of the data, thereby improving the accuracy of the reduced-dimensional feature data.

[0157] In one of the exemplary embodiments, the decoder module includes a pre-trained residual network, which decouples the reduced-dimensionality feature data to generate a rearview mirror image of a target car, including: mapping the reduced-dimensionality feature data to a target space to obtain mapping features; upsampling the mapping features to obtain sampling data; inputting the encoded feature data and the sampling data into a pre-trained residual network to obtain rearview mirror image data for the target car; and generating a rearview mirror image based on the rearview mirror image data.

[0158] Among them, the residual network can refer to a deep neural network structure. In practical applications, the residual network can include several residual blocks.

[0159] The target space may refer to a pre-set vector space. In practical applications, the vector dimension / feature dimension of the target space is higher than the vector dimension / feature dimension of the reduced-dimensional feature data.

[0160] The mapping feature may refer to data obtained after mapping the dimension-reduced feature data to the target space.

[0161] The sampled data may refer to data obtained by upsampling the mapping features corresponding to the reduced-dimensional feature data in the target space.

[0162] The rearview mirror image data may refer to data of an image used to represent the viewing angle of the rearview mirror.

[0163] As an example, Figure 3 As shown, a schematic diagram of a process for encoding and decoding multimodal feature data is provided, wherein the vehicle-mounted device inputs the reduced-dimensional feature data into a decoder module in a rearview mirror image generation model, and the decoder module may include a decoupled generation network and a warning information fusion layer, wherein the decoupled generation network may include a residual network, and the decoupled generation network may utilize the reduced-dimensional feature data (compressed feature representation) Z generated by the encoder module low_dim As a basis, high-resolution, low-latency rearview mirror view images and warning information are generated. Specifically, assuming that the dimension-reduced feature data (compressed feature representation) Z low_dimThe dimension is N. The decoupled generation network can map the reduced-dimensional feature data to a higher-dimensional target space through a fully connected layer to better initialize the generation process. This step can be regarded as a preparation before decoding Z. Then the decoupled generation network has an initial upsampling layer, and uses the upsampling layer to expand the mapping features corresponding to the reduced-dimensional feature data into a basic image structure to obtain the sampling data. In practical applications, the decoupled generation network can start to build a low-resolution version of the image through transposed convolution or sub-pixel convolution, achieve lightweight image size enlargement, and reduce computing resource consumption. Then the decoupled generation network inputs the encoded feature data and the sampling data into a pre-trained residual network. The decoupled generation network can use a series of residual blocks to gradually improve the resolution of the image and incorporate the information of the encoded feature data Z at each scale. Each block contains the influence of the conditional input Z. In specific implementation, it can be achieved through conditional scaling or conditional deviation. Then the decoupled generation network uses a regular convolutional layer to decouple high-resolution rearview mirror image data for the target car. In specific implementation, the convolutional layer needs to be attached with a tanh or sigmoid activation function to ensure that the output pixel value is within a reasonable range (such as [-1,1] or [0, 1]).

[0164] In this embodiment, mapping features are obtained by mapping reduced-dimensional feature data to a target space; upsampling is performed on the mapping features to obtain sampling data; the encoded feature data and the sampling data are input into a pre-trained residual network to obtain rearview mirror image data for a target vehicle; a rearview mirror image is generated based on the rearview mirror image data, and rearview mirror image data similar to a real electronic rearview mirror image can be gradually generated based on the reduced-dimensional feature data using a residual network, thereby improving the accuracy of the rearview mirror image.

[0165] In one of the exemplary embodiments, a rearview mirror image is generated based on rearview mirror image data, including: generating warning data for a target vehicle based on first image data, second image data and point cloud data; fusing the rearview mirror image data and the warning data to obtain target image data; and generating a rearview mirror image based on the target image data.

[0166] Among them, the warning data may refer to data used to inform the driver of safety risks of the car. In practical applications, the warning data may include door opening warning data and blind spot warning data. Among them, the door opening warning data may be obtained by the door opening warning system (DOW, Door Open Warning, DOW). The door opening warning system may detect the information of moving obstacles in the alarm area behind the side of the vehicle. When the vehicle is in the working speed range, the system determines the possibility of collision between the obstacle and the door, and issues a warning to the driver based on the possibility of collision and the intention to open the door; the blind spot warning data may be obtained by the blind spot monitoring and warning system (Blind Spot Detection, BSD). When the vehicle speed is within the range of (10km / h~150km / h), the blind spot monitoring and warning system may use radar and cameras to detect target information behind and in front of the vehicle, determine whether there is a target in the blind spot and whether there is a collision risk with the vehicle, and issue an alarm to the driver.

[0167] The target image data may refer to data used to simultaneously represent the rearview mirror image and the warning data in the form of an image.

[0168] As an example, Figure 3 As shown, a schematic diagram of a process for encoding and decoding multimodal feature data is provided. The vehicle-mounted device inputs the first image data, the second image data and the point cloud data into a pre-trained rearview mirror image generation model. After feature fusion, encoding, dimensionality reduction and other processing, the decoupled generation network in the decoder module in the rearview mirror image generation model can generate high-resolution rearview mirror image data for the target car, and can also generate warning data such as door opening warning data and blind spot warning data based on the dimensionality reduction feature data according to preset risk analysis rules. The warning information fusion layer in the decoder module in the rearview mirror image generation model can superimpose and fuse the warning data such as door opening warning data and blind spot warning data with the rearview mirror image data. The rearview mirror image data (i.e., target image data) fused with the warning data is obtained. In practical applications, the warning data can represent the warning information. The display of the warning information can include drawing a bounding box, a text label, or other visual indicators on the image corresponding to the rearview mirror image data. At this time, the image of the rearview mirror perspective carrying the bounding box, the text label, or other visual indicators can be used as the target image data. Furthermore, the rearview mirror image carrying the warning information can be displayed on the display interface of the target car (such as the vehicle's internal display screen). The display of the rearview mirror image does not require a specific deep learning network layer, but uses functions in an image processing library and a graphical user interface (GUI) library (such as Qt or wxWidgets).

[0169] In this embodiment, warning data for the target vehicle is generated based on the first image data, the second image data and the point cloud data; the rearview mirror image data and the warning data are fused to obtain the target image data; and a rearview mirror image is generated based on the target image data. On the basis of the image, the warning data for risk warning can be fused, so that while displaying the image from the rearview mirror perspective, the warning information can be displayed to the driver intuitively and in a timely manner, thereby improving the information richness of the rearview mirror image.

[0170] In one of the exemplary embodiments, the method further includes: obtaining first sample image data taken by a side-view camera of the sample car, second sample image data taken by a rear-view camera, and sample point cloud data collected by a rear-side radar, and obtaining a rearview mirror reference image corresponding to the sample car; inputting the first sample image data, the second sample image data, and the sample point cloud data into a rearview mirror image generation model to be trained to obtain a rearview mirror image of the sample car; and training the rearview mirror image generation model to be trained based on the rearview mirror reference image and the rearview mirror image of the sample car.

[0171] The sample car may refer to a car corresponding to the sample image data and the sample point cloud data when training the rearview mirror image generation model. In practical applications, the sample car may include a target car or a car of a different model from the target car.

[0172] The first sample image data may refer to image data captured by a side-view camera of a sample car.

[0173] The second sample image data may refer to image data captured by a rearview camera of a sample car.

[0174] The sample point cloud data may refer to data representing information such as the position and speed of an object within the detection range corresponding to the rear radar of the sample car.

[0175] The rearview mirror reference image may refer to an image displayed when the electronic rearview mirror of the sample car works normally.

[0176] The rearview mirror image of the sample car may refer to an image output by the rearview mirror image generation model to be trained based on the first sample image data, the second sample image data and the sample point cloud data.

[0177] As an example, in order to ensure that the rearview mirror image generation model can generate accurate rearview mirror images, the rearview mirror image generation model can be trained. Specifically, the first sample image data taken by the side view camera of the sample car, the second sample image data taken by the rearview camera, and the sample point cloud data collected by the rear radar are used as training data, and the rearview mirror reference image corresponding to the sample car is used as label data. The first sample image data, the second sample image data, and the sample point cloud data are input into the rearview mirror image generation model to be trained. The rearview mirror image generation model to be trained can output the rearview mirror image for the sample car. By comparing the rearview mirror reference image with the rearview mirror image for the sample car, Train the rearview mirror image generation model to be trained. In actual applications, during the training process of the rearview mirror image generation model, based on the degree of difference between the rearview mirror reference image and the rearview mirror image for the sample car, the data preprocessing layer, the multimodal fusion layer, the modular hybrid feature extraction layer in the encoder module, the lightweight adaptive feature dimensionality reduction layer in the encoder module, the decoupled generation network in the decoder module, and the warning information fusion layer in the decoder module can be adjusted to correspond to the model parameters of each of the model until the degree of difference between the rearview mirror reference image and the rearview mirror image for the sample car meets the preset degree of difference requirement, at which time the training of the rearview mirror image generation model is completed.

[0178] In this embodiment, first sample image data taken by a side-view camera of a sample car, second sample image data taken by a rear-view camera, and sample point cloud data collected by a rear-side radar are obtained, and a rearview mirror reference image corresponding to the sample car is obtained; the first sample image data, the second sample image data, and the sample point cloud data are input into a rearview mirror image generation model to be trained to obtain a rearview mirror image of the sample car; the rearview mirror image generation model to be trained is trained according to the rearview mirror reference image and the rearview mirror image of the sample car, and the rearview mirror image generation model is trained using the first sample image data, the second sample image data, the sample point cloud data, and the rearview mirror reference image as training samples, thereby ensuring the accuracy of the rearview mirror image output by the rearview mirror image generation model.

[0179] In some embodiments, Figure 4As shown, a flow chart of generating a rearview mirror image is provided. When the electronic rearview mirror of the target vehicle fails or malfunctions, in order to avoid the loss of the electronic rearview mirror image affecting the safety of the target vehicle, the on-board device can obtain the first image data through the side view camera of the target vehicle, obtain the second image data through the rear view camera of the target vehicle, and collect point cloud data through the rear side radar of the target vehicle. Then, the on-board device inputs the first image data, the second image data and the point cloud data into the data preprocessing layer in the rearview mirror image generation model for data preprocessing, and then inputs the preprocessed data into the multimodal fusion layer in the rearview mirror image generation model for data synchronization, feature extraction and feature fusion to obtain multimodal features. Then, the on-board device can input the multimodal features into the data preprocessing layer in the rearview mirror image generation model for data synchronization, feature extraction and feature fusion to obtain multimodal features. The encoder module is input into the rearview mirror image generation model. The modular hybrid feature extraction layer and the lightweight adaptive feature dimensionality reduction layer in the encoder module perform feature encoding and dimensionality reduction on the multimodal features to obtain dimensionality reduction features. The vehicle-mounted device can input the dimensionality reduction features into the decoder module in the rearview mirror image generation model. The decoupling generation network in the decoder module can decouple the dimensionality reduction features to obtain rearview mirror image data and warning data. The warning information fusion layer in the decoder module can superimpose and fuse the rearview mirror image data and the warning data to obtain a rearview mirror image containing warning information, and display the rearview mirror image containing warning information on the display interface corresponding to the target vehicle, thereby realizing an efficient redundancy mechanism for electronic rearview mirror failures or failures.

[0180] It is worth noting that the above rearview mirror image generation method first performs multimodal fusion and feature extraction on the millimeter-wave radar data and camera data, then decodes the two separately in the decoding stage, and then fuses and superimposes the rearview mirror image and warning information, that is, the data processing method of "fusion first, decoupling later, and then superposition". This data processing method is a precise strategy based on a deep understanding of system complexity and data characteristics. The "fusion first" strategy allows engineers to fully utilize the complementarity of millimeter-wave radar and camera data in the early stages of data processing. Millimeter-wave radar can provide accurate distance, speed and angle information, while the camera can provide rich texture and color information. By fusing these two types of data, engineers can obtain a more comprehensive and accurate environmental perception model to provide strong support for subsequent processing; however, fusion alone is not enough to meet all needs. Different sensor data have differences in characteristics and processing methods. If they are directly decoded and fused, it may cause mutual interference and confusion between data, thereby affecting the accuracy and robustness of the system. Therefore, the "post-decoupling" step is crucial. By decoupling the fused data, we can ensure that each sensor data can be processed independently and accurately, thereby avoiding mutual interference between data; finally, the "re-superposition" step is to recombine the decoupled data to form a more complete and accurate environmental perception result. By superimposing and fusing the precise spatial information of the millimeter-wave radar and the rich visual information of the camera, a fused warning information containing both precise location information and rich detail information is obtained. This fused warning information can not only help the system more accurately identify obstacles, traffic participants, and blind spot detection, but also provide more reliable information support for subsequent decision-making and planning.

[0181] In this embodiment, the existing sensor resources that have been integrated into the intelligent driving platform are used to achieve effective replacement and seamless connection of the electronic rearview mirror function through innovative algorithms and hardware fusion technology. Specifically, by integrating the data of the side view camera, rear view camera, and rear millimeter wave radar, an image with the same viewing angle as the electronic rearview mirror can be generated, thereby providing the driver with alternative visual information when the electronic rearview mirror fails. This design not only avoids the introduction of additional hardware and reduces costs, but also makes full use of the performance advantages of existing sensors to achieve the redundant function of the electronic rearview mirror.

[0182] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0183] Based on the same inventive concept, the embodiment of the present application also provides a rearview mirror image generating device for implementing the rearview mirror image generating method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more embodiments of the rearview mirror image generating device provided below can refer to the limitations of the rearview mirror image generating method above, and will not be repeated here.

[0184] In an exemplary embodiment, Figure 5 As shown, a rearview mirror image generation device is provided, including: an acquisition module 502, a fusion module 504, an encoding module 506 and a decoding module 508, wherein:

[0185] An acquisition module 502 is used to acquire first image data captured by a side view camera of a target vehicle, second image data captured by a rear view camera, and point cloud data collected by a rear radar;

[0186] A fusion module 504 is used to input the first image data, the second image data and the point cloud data into a pre-trained rearview mirror image generation model, and fuse the image features corresponding to the first image data, the image features corresponding to the second image data and the point cloud features of the point cloud data through the rearview mirror image generation model to obtain multimodal feature data;

[0187] An encoding module 506, configured to input the multimodal feature data into an encoder module in the rearview mirror image generation model, obtain encoding feature data matching the multimodal feature data, and perform dimensionality reduction processing on the encoding feature data to obtain dimensionality reduction feature data;

[0188] The decoding module 508 is used to input the reduced-dimensional feature data into a decoder module in the rearview mirror image generation model, and decouple the reduced-dimensional feature data through the decoder module to generate a rearview mirror image of the target car.

[0189] In one of the exemplary embodiments, the fusion module 504 is specifically used to fuse the image features corresponding to the first image data and the image features corresponding to the second image data to obtain fused image features; and to fuse the fused image features with the point cloud features of the point cloud data to obtain the multimodal feature data.

[0190] In one of the exemplary embodiments, the fusion module 504 is specifically used to normalize the image features corresponding to the first image data and the image features corresponding to the second image data to obtain a first attention weight corresponding to the first image data and a second attention weight corresponding to the second image data; based on the first attention weight and the second attention weight, perform weighted fusion on the image features corresponding to the first image data and the image features corresponding to the second image data to obtain the fused image features.

[0191] In one of the exemplary embodiments, the fusion module 504 is specifically used to normalize the fused image feature data and the point cloud features to obtain a third attention weight corresponding to the fused data mapping feature and a fourth attention weight corresponding to the point cloud feature; based on the third attention weight and the fourth attention weight, the fused image feature and the point cloud feature are weightedly fused to obtain the multimodal feature data.

[0192] In one of the exemplary embodiments, the acquisition module 502 is specifically used to perform convolution processing on the multimodal feature data, and normalize the convolution processing result to obtain normalized feature data; and use a preset activation function to map the normalized feature data to the encoded feature data.

[0193] In one of the exemplary embodiments, the encoding module 506 is used to decompose a sparse matrix from the encoding feature data using a sparse factor; and perform dimensionality reduction processing on the encoding feature data according to the sparse matrix to obtain the reduced dimensionality feature data.

[0194] In one of the exemplary embodiments, the decoder module includes a pre-trained residual network, and the decoding module 508 is specifically used to map the reduced-dimensional feature data to the target space to obtain mapping features; up-sample the mapping features to obtain sampling data; input the encoded feature data and the sampling data into the pre-trained residual network to obtain rearview mirror image data for the target car; and generate the rearview mirror image based on the rearview mirror image data.

[0195] In one of the exemplary embodiments, the decoding module 508 is specifically used to generate warning data for the target vehicle based on the first image data, the second image data and the point cloud data; fuse the rearview mirror image data and the warning data to obtain target image data; and generate the rearview mirror image based on the target image data.

[0196] In one of the exemplary embodiments, the above-mentioned device also includes a training module, which is specifically used to obtain first sample image data taken by a side-view camera of a sample car, second sample image data taken by a rear-view camera, and sample point cloud data collected by a rear-side radar, and obtain a rearview mirror reference image corresponding to the sample car; input the first sample image data, the second sample image data, and the sample point cloud data into a rearview mirror image generation model to be trained to obtain the rearview mirror image of the sample car; and train the rearview mirror image generation model to be trained based on the rearview mirror reference image and the rearview mirror image of the sample car.

[0197] Each module in the above rearview mirror image generation device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0198] In an exemplary embodiment, a vehicle-mounted device is provided. The vehicle-mounted device may be a server, and its internal structure diagram may be as follows: Figure 6 As shown. The vehicle-mounted device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the vehicle-mounted device is used to provide computing and control capabilities. The memory of the vehicle-mounted device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the vehicle-mounted device is used to store first image data, second image data, point cloud data, etc. The input / output interface of the vehicle-mounted device is used to exchange information between the processor and an external device. The communication interface of the vehicle-mounted device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for generating a rearview mirror image is implemented.

[0199] Those skilled in the art will understand that Figure 6The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the vehicle-mounted equipment to which the scheme of the present application is applied. The specific vehicle-mounted equipment may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0200] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0201] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0202] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0203] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for generating a rearview mirror image, characterized in that: The method comprises: Acquire first image data captured by a side-view camera of the target vehicle, second image data captured by a rear-view camera, and point cloud data collected by a rear-side radar; Inputting the first image data, the second image data, and the point cloud data into a pre-trained rearview mirror image generation model, and fusing image features corresponding to the first image data, image features corresponding to the second image data, and point cloud features of the point cloud data through the rearview mirror image generation model to obtain multimodal feature data; Inputting the multimodal feature data into an encoder module in the rearview mirror image generation model, obtaining encoding feature data matching the multimodal feature data, and decomposing a sparse matrix from the encoding feature data using a sparse factor; performing dimensionality reduction processing on the encoding feature data according to the sparse matrix to obtain reduced-dimensionality feature data; The reduced-dimensional feature data is input into a decoder module in the rearview mirror image generation model, and the reduced-dimensional feature data is mapped to a target space through the decoder module to obtain mapping features; the mapping features are upsampled to obtain sampling data; the encoded feature data and the sampling data are input into a residual network pre-trained in the decoder module to obtain rearview mirror image data for the target vehicle; and the rearview mirror image of the target vehicle is generated according to the rearview mirror image data.

2. The method according to claim 1, characterized in that: The fusing the image features corresponding to the first image data, the image features corresponding to the second image data, and the point cloud features of the point cloud data to obtain multimodal feature data includes: fusing the image features corresponding to the first image data and the image features corresponding to the second image data to obtain fused image features; The fused image features and the point cloud features of the point cloud data are fused to obtain the multimodal feature data.

3. The method according to claim 2, characterized in that The fusing the image features corresponding to the first image data and the image features corresponding to the second image data to obtain the fused image features includes: Normalizing the image features corresponding to the first image data and the image features corresponding to the second image data to obtain a first attention weight corresponding to the first image data and a second attention weight corresponding to the second image data; According to the first attention weight and the second attention weight, the image features corresponding to the first image data and the image features corresponding to the second image data are weightedly fused to obtain the fused image features.

4. The method according to claim 2, characterized in that: The fusing the fused image features and the point cloud features of the point cloud data to obtain the multimodal feature data includes: Normalizing the fused image features and the point cloud features to obtain a third attention weight corresponding to the fused image features and a fourth attention weight corresponding to the point cloud features; The fused image features and the point cloud features are weightedly fused according to the third attention weight and the fourth attention weight to obtain the multimodal feature data.

5. The method according to claim 1, characterized in that: The acquiring the coded feature data matching the multimodal feature data comprises: Performing convolution processing on the multimodal feature data, and normalizing the convolution processing result to obtain normalized feature data; The normalized feature data is mapped to the encoded feature data using a preset activation function.

6. The method according to claim 1, characterized in that The step of generating the rearview mirror image of the target vehicle according to the rearview mirror image data comprises: generating warning data for the target vehicle according to the first image data, the second image data and the point cloud data; Fusing the rearview mirror image data and the warning data to obtain target image data; The rearview mirror image is generated according to the target image data.

7. The method according to claim 1, characterized in that The method further comprises: Acquire first sample image data taken by a side-view camera of a sample car, second sample image data taken by a rear-view camera, and sample point cloud data collected by a rear-side radar, and acquire a rear-view mirror reference image corresponding to the sample car; Inputting the first sample image data, the second sample image data and the sample point cloud data into a rearview mirror image generation model to be trained to obtain a rearview mirror image of the sample car; The rearview mirror image generation model to be trained is trained according to the rearview mirror reference image and the rearview mirror image of the sample car.

8. A rearview mirror image generating device, characterized in that: The device comprises: An acquisition module, used to acquire first image data taken by a side-view camera of a target vehicle, second image data taken by a rear-view camera, and point cloud data collected by a rear-side radar; a fusion module, configured to input the first image data, the second image data, and the point cloud data into a pre-trained rearview mirror image generation model, and fuse image features corresponding to the first image data, image features corresponding to the second image data, and point cloud features of the point cloud data through the rearview mirror image generation model to obtain multimodal feature data; An encoding module, used for inputting the multimodal feature data into an encoder module in the rearview mirror image generation model, obtaining encoding feature data matching the multimodal feature data, and decomposing a sparse matrix from the encoding feature data using a sparse factor; performing dimensionality reduction processing on the encoding feature data according to the sparse matrix to obtain reduced-dimensionality feature data; A decoding module is used to input the reduced-dimensional feature data into a decoder module in the rearview mirror image generation model, map the reduced-dimensional feature data to a target space through the decoder module to obtain mapping features; upsample the mapping features to obtain sampling data; input the encoded feature data and the sampling data into a pre-trained residual network in the decoder module to obtain rearview mirror image data for the target vehicle; and generate a rearview mirror image of the target vehicle based on the rearview mirror image data.

Images