Scene recognition method and system for all-weather autonomous driving

Abstract

The application relates to a scene recognition method and system for all-weather automatic driving, which obtains image information of a driving environment, divides the image information into a day scene and a night scene based on a trained recognition model, performs image enhancement when the image information is the night scene, and performs depth estimation by using the day scene image and the enhanced night scene image to realize scene recognition; during the training of the recognition model, an original day scene image is obtained, a night scene image is generated based on a recurrent generative adversarial network, and the original day scene image and the night scene image are mixed to form an all-day data set; the all-day data set is used to distinguish the day scene image and the night scene image by using a classification decision maker; the day scene image is used to realize scene recognition by using a depth estimation model; and the night scene image is used to realize scene recognition by using a depth estimation model after image enhancement.

Description

Technical Field

[0001] This invention relates to the field of visual perception technology for autonomous driving, specifically to a scene recognition method and system for all-weather autonomous driving. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Autonomous driving requires vehicles to accurately identify and perceive their surroundings, typically relying on LiDAR, vision-based solutions, or a combination of both. Vision-based solutions are less expensive, so current technologies generally utilize image information acquired through visual means (such as cameras) and process it with algorithms to obtain scene information for autonomous driving. To identify scene information from images, existing technologies have proposed a series of algorithms, such as supervised depth estimation: using real ground depth maps as supervision, a supervised deep learning network can extract features from the input image and learn the relationship between depth and RGB values, thereby identifying scene information.

[0004] Nighttime lighting conditions are typically poor, reducing the visibility of targets such as roads, vehicles, and pedestrians. This can make it difficult for algorithms to accurately perceive and identify objects in a scene, thus affecting the accuracy of depth estimation. To address this issue, existing techniques use Generative Adversarial Networks (GANs) to adapt nighttime features to daytime features, or reduce irregularities at night through image augmentation and GAN-based regularization.

[0005] While these algorithms can solve depth estimation problems in different scenarios and achieve good results, these models are often complex and bloated, resulting in long computation times. They require powerful hardware and are difficult to configure in edge devices for autonomous driving.

[0006] At the same time, most existing models can only be applied to scene recognition in a single daytime or nighttime environment, and there is a lack of models that can be applied to all weather conditions. Summary of the Invention

[0007] To address the technical problems described in the background section, this invention provides a scene recognition method and system for all-weather autonomous driving, offering a lightweight and efficient depth estimation model that can be used in all weather conditions and is suitable for running on edge devices. This model should effectively utilize image information to improve the accuracy and stability of depth estimation, while maintaining its simplicity and efficiency.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] The first aspect of the present invention provides a scene recognition method for all-weather autonomous driving, comprising the following steps:

[0010] Image information of the driving environment is acquired and divided into daytime and nighttime scenes based on the trained recognition model. When the image information is nighttime, image enhancement is performed. Depth estimation is performed using the images of the daytime scene and the enhanced nighttime scene to achieve scene recognition.

[0011] During training, the recognition model acquires original daytime scene images, generates nighttime scene images based on a recurrent generative adversarial network, and mixes them with the original daytime scene images to form a full-day dataset. The full-day dataset uses a classification decision engine to distinguish between daytime and nighttime scene images. Daytime scene images are used to achieve scene recognition using a depth estimation model, while nighttime scene images are used to achieve scene recognition after image enhancement processing using a depth estimation model.

[0012] Furthermore, the original daytime scene image is obtained, and a nighttime scene image is generated based on a recurrent generative adversarial network, specifically as follows:

[0013] L GAN (G,D,a,b)=E b [(D(b)-1) 2 ]+E a [D(G(a)) 2 ];

[0014] L cyc =E[||G B (G A (a))-a||1]+E b [||G A (G B (b))-b||1];

[0015] L CycleGAN =L GAN (G A D B ,a,b)+L GAN (G B D A ,b,a)+λL cyc ;

[0016] Recurrent Generative Adversarial Networks (GANs) include generators (G) A D A ) and discriminator (G B D B ), where the translator between domain A and domain B is G. A A→B and G B B→A, D A Trained to distinguish real imagesa and translation image G B (b), D B Trained to distinguish between images b and G A (a) Training is performed using adversarial loss and cycle consistency loss, by stimulating G. A and G B They are inverse mappings of each other, making G B (G A (a))≈a and G A (G B (b))≈b.

[0017] Furthermore, the full-day dataset utilizes a classification decision engine to distinguish between daytime and nighttime scene images, specifically:

[0018] Get the input image X V Extracting visual features Z V =g(X) V The linear layer connects image features to words and embeds them in a space, and the projection matrix W is used to project visual features Z. V Convert to language embedding tag H q Image-text pairs are formed for training.

[0019] Furthermore, the training process for the classification decision machine includes the following steps:

[0020] For each image X V Generate multi-turn dialogue data Where T is the total number of rounds;

[0021] The generated multi-turn dialogue data forms a sequence, and the answers in the sequence are regarded as the responses of the classification decision-maker. The target answer X is generated by calculation. a The probability is determined in the current prediction label x. i The instruction and response markers from all previous rounds.

[0022] Furthermore, the instructions in round t. Considered as:

[0023]

[0024] Among them, X V For the input image, This is the generated multi-turn dialogue data.

[0025] Furthermore, for a sequence of length L, the target answer X is generated using the following formula. a The probability of:

[0026]

[0027] Where θ is a trainable parameter, X instruct,<i and X a,<i These are the instruction flags and response flags in all rounds preceding the current prediction flag xi.

[0028] Furthermore, the depth estimation model is a hybrid CNN and Transformer architecture, which implicitly performs alternative connections through summation operations and introduces linear attention instead of cross-covariance attention.

[0029] A second aspect of the present invention provides a scene recognition system for all-weather autonomous driving, comprising:

[0030] The scene recognition module is configured to: acquire image information of the driving environment, divide it into daytime scene and nighttime scene based on the trained recognition model, perform image enhancement when the image information is nighttime, and perform depth estimation using the daytime scene image and the enhanced nighttime scene image to achieve scene recognition;

[0031] During training, the recognition model acquires original daytime scene images, generates nighttime scene images based on a recurrent generative adversarial network, and mixes them with the original daytime scene images to form a full-day dataset. The full-day dataset uses a classification decision engine to distinguish between daytime and nighttime scene images. Daytime scene images are used to achieve scene recognition using a depth estimation model, while nighttime scene images are used to achieve scene recognition after image enhancement processing using a depth estimation model.

[0032] A third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the above-described all-weather autonomous driving scene recognition method.

[0033] A fourth aspect of the present invention provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps in the above-described all-weather autonomous driving scene recognition method.

[0034] Compared with existing technologies, one or more of the above technical solutions have the following beneficial effects:

[0035] By utilizing daytime scene information with good lighting conditions to generate corresponding nighttime scene information, and mixing them, an all-weather dataset is obtained. The ground truth of the generated dataset is consistent with the original daytime scene information, which facilitates verification and testing. It can also simplify the structure of subsequent depth estimation models, thereby reducing computational costs. After the model is trained, it can quickly distinguish between daytime and nighttime scenes, perform rapid depth estimation for daytime scenes, and perform image enhancement before depth estimation for nighttime scenes, thus achieving all-weather driving scene recognition. This is beneficial for deploying the algorithm in edge devices. Attached Figure Description

[0036] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0037] Figure 1 A schematic diagram illustrating the scene recognition process for all-weather autonomous driving provided in one or more embodiments of the present invention;

[0038] Figure 2 A schematic diagram of raw data during scene recognition provided in one or more embodiments of the present invention;

[0039] Figure 3 A schematic diagram of nighttime data generated during scene recognition as provided in one or more embodiments of the present invention;

[0040] Figure 4 A schematic diagram illustrating the effect of image enhancement during scene recognition provided in one or more embodiments of the present invention;

[0041] Figure 5 A schematic diagram of a depth estimation model provided for one or more embodiments of the present invention;

[0042] Figure 6 This is a schematic diagram comparing the effects of several models provided in one or more embodiments of the present invention. Detailed Implementation

[0043] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0044] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0045] The following examples illustrate a scene recognition method and system for all-weather autonomous driving, presenting a lightweight and efficient depth estimation model that can be used in all weather conditions and is suitable for running on edge devices. This model should effectively utilize image information to improve the accuracy and stability of depth estimation, while maintaining its simplicity and efficiency.

[0046] Example 1:

[0047] like Figures 1-6 As shown, the scene recognition method for all-weather autonomous driving includes the following steps:

[0048] Image information of the driving environment is acquired and divided into daytime and nighttime scenes based on the trained recognition model. When the image information is nighttime, image enhancement is performed. Depth estimation is performed using the images of the daytime scene and the enhanced nighttime scene to achieve scene recognition.

[0049] During training, the recognition model acquires original daytime scene images, generates nighttime scene images based on a recurrent generative adversarial network, and mixes them with the original daytime scene images to form a full-day dataset. The full-day dataset uses a classification decision engine to distinguish between daytime and nighttime scene images. Daytime scene images are used to achieve scene recognition using a depth estimation model, while nighttime scene images are used to achieve scene recognition after image enhancement processing using a depth estimation model.

[0050] This embodiment, after quickly distinguishing between daytime and nighttime scenes, performs rapid depth estimation for daytime scenes and image enhancement for nighttime scenes, achieving all-day depth estimation and greatly improving the efficiency and accuracy of depth estimation.

[0051] First, achieve all-day autonomous driving scene depth estimation.

[0052] like Figure 1 As shown, the steps for depth estimation in all-day autonomous driving scenarios are as follows: generate nighttime scenes using Generative Adversarial Networks (GANs) and mix them with daytime scenes to obtain an all-day dataset; fine-tune the multimodal large model to obtain a classifier that can distinguish between day and night; perform image augmentation on the classified nighttime dataset; and improve the depth estimation model to better apply it to autonomous driving scenarios.

[0053] In this embodiment, the original data, the data generated using a generative adversarial network, and the image enhancement effect of the nighttime data are respectively shown as follows: Figures 2-4 As shown.

[0054] 2. Create a full-day dataset

[0055] A nighttime dataset was generated from images in the KITTI dataset using a Cycle Generative Adversarial Network (Cycle GAN). The theoretical basis for this can be expressed as follows:

[0056] L GAN (G,D,a,b)=E b [(D(b)-1) 2 ]+E a [D(G(a)) 2 (1)

[0057] Lcyc =E[||G B (G A (a))-a||1]+E b [||G A (G B (b))-b||1] (2)

[0058] L CycleGAN =L GAN (G A D B ,a,b)+L GAN (G B D A ,b,a)+λL cyc (3)

[0059] CycleGAN consists of two pairs of generator and discriminator networks, i.e., (G in the formula) A D A ) and (G B D B ), where the translator between domain A and domain B is G. A A→B and G B B→A. D A Trained to distinguish between real image a and translated image G B (b), while D B Trained to distinguish between images b and G A (a) The system is trained using adversarial loss and cycle consistency loss. Cycle consistency loss is a method for normalizing the highly unconstrained problem of unidirectional translated images by stimulating G. A and G B They are inverse mappings of each other, making G B (G A (a))≈a and G A (G B (b))≈b.

[0060] 3. Training the classification decision machine

[0061] Fine-tuning the Large Language and Vision Assistant (LLaVA) for an input image X V Considering pre-trained language images to a pre-trained visual encoder, visual features Z are provided. V =g(X) V The model uses a simple linear layer to connect image features to words and embeddings in the space. The model applies a trainable projection matrix W to Z... V Convert to language embedding tag H qThe tag has the same dimension as the word embedding space in the language model:

[0062] H V =W·Z V Z V =g(X) V (4)

[0063] H V It is a visual representation sequence.

[0064] The training process is as follows: For each image XV, generate multi-turn dialogue data. Where T is the total number of rounds. Organize them into a sequence, treating all answers as responses from the assistant, and assigning the instruction for round t. Considered as:

[0065]

[0066] To ensure that the multimodal instruction sequences follow a uniform format, the model uses the original autoregressive training objective of LLM and adjusts it for instructions on the prediction labels.

[0067] Specifically, for a sequence of length L, the probability of generating the target answer Xa is calculated using the following formula:

[0068]

[0069] Where θ is a trainable parameter, X instruct,<i and X a,<i These are the instruction flags and response flags in all rounds preceding the current prediction flag xi.

[0070] A dataset was selected, consisting of half daytime and half nighttime images. Each image was labeled to create image-text pairs. This dataset was then fed into the model for fine-tuning, resulting in the classification decision module. Its robustness was verified through validation experiments on different datasets. The experimental results are shown in Table 1.

[0071] Table 1. Accuracy of the Classification Decision Module

[0072]

[0073] As can be seen, the accuracy is high when using the "KITTI+ to generate nighttime images" method in this embodiment.

[0074] 4. Nighttime image enhancement

[0075] A cooperative low-light image enhancement algorithm based on implicit neural representations is used to unify different degradation factors of real-world scenes with a controllable fitting function, exhibiting good robustness. Three sets of comparative experiments were conducted, using generated nighttime verification images as input, with predicted maximum distances of 80 meters, 60 meters, and 40 meters, respectively. Comparative experiments were performed using a generated dataset. In this embodiment, nighttime scenes were generated from the verification set of the KITTI dataset, and three sets of experiments were conducted using this dataset. The results are shown in Table 2.

[0076] Table 2 Comparison of Image Enhancement Effects

[0077] Dataset Distance / meter AbsRel SqRel RMSE RMSElog night 80 0.445 4.828 10.800 0.533 Image enhancement 80 0.306 2.708 7.915 0.390 night 60 0.437 4.327 9.533 0.517 Image enhancement 60 0.303 2.509 7.080 0.383 night 40 0.411 3.285 7.359 0.478 Image enhancement 40 0.293 2.055 5.637 0.364

[0078] 5. Improved lightweight depth estimation model

[0079] The Lite-Mono model is improved by adopting a hybrid CNN and Transformer architecture. It introduces a structural reparameterization method to replace the connection (Concat), which saves a lot of inference time. The information fusion process of the structural reparameterization method is implicitly performed through the addition operation, rather than leaving it to other convolutional layers.

[0080] Feature reuse via concatenation can be represented as:

[0081] y = Cat([x, Φ1(x), ..., Φ s-1 (x)]) (7)

[0082] Cat is a join operation that simply preserves the existing feature map and leaves the information processing to other operation steps.

[0083] Feature reuse through structural reparameterization can be represented as:

[0084] y = Add([x, Φ1(x), ..., Φ s-1 (x)])=Φ * (x) (8)

[0085] Unlike concatenation, summation also plays a role in feature fusion. It achieves feature reuse through structural reparameterization, which significantly reduces the complexity of the model. It is used in the first stage of the encoder, namely the liteweight_stem1 and liteweight_stem2 modules.

[0086] Introducing linear attention to replace cross-covariance attention, the simplified formula for the linear attention mechanism is:

[0087]

[0088] The above equation can be written in vector form as follows:

[0089]

[0090] In this embodiment, it is referred to as the Linear Global Feature Extraction module (LGFE).

[0091] The above process first uses a Recurrent Generative Adversarial Network (CycleGAN) to generate nighttime scenes from the KITTI dataset and mixes them with daytime scenes to obtain an all-weather dataset. The ground truth values ​​of the generated dataset are consistent with the original KITTI dataset, facilitating verification and testing. Then, the LL aVA model is fine-tuned to generate a classifier capable of quickly classifying daytime and nighttime scenes, simultaneously deciding whether image augmentation is needed: no image augmentation is required for daytime scenes, but image augmentation is needed for nighttime scenes. The Lite-Mono model is improved to obtain a lightweight depth estimation model more suitable for edge devices, which is then used to estimate the depth of the previously processed images.

[0092] The depth estimation model in this embodiment is more lightweight, significantly reducing computational load and making it more suitable for edge devices such as autonomous driving. The model introduces linear attention instead of cross-covariance attention. Linear attention is a simplified attention mechanism that obtains attention weights by calculating the dot product of the query and key, thus avoiding complex matrix operations. This computational approach has relatively low time complexity, thus achieving faster computation speed. Furthermore, it utilizes the method proposed by RepGhost to generate and fuse different feature maps through reparameterization, eliminating inefficient concat operations and saving significant inference time. The information fusion process is implicitly performed by the Add operation, rather than being left to other convolutional layers. These methods greatly improve the computational speed of the depth estimation model while reducing the demand for computational resources. Considering the stringent real-time requirements of autonomous driving scenarios and the limited computational resources of edge devices, a lightweight depth model meets the needs of autonomous driving scenarios.

[0093] To verify the performance of this embodiment, the model was trained using the same training data and image segmentation method as Monodepth2, and its performance was compared with some mainstream algorithms on the KITTI dataset. The comparison results are shown in Table 3.

[0094] Table 3. Model computational cost, parameter count, and error.

[0095] method FLOPs(G) Params(M) AbsRel SqRel RMSE RMSElog GeoNet - 31.600 0.149 1.060 5.567 0.226 DDVO - 28.100 0.151 1.257 5.583 0.228 R-MSFM3 16.500 3.500 0.114 0.815 4.712 0.193 R-MSFM6 31.200 3.800 0.112 0.806 4.704 0.193 Monodepth2 8.000 14.300 0.115 0.903 4.863 0.193 Lite-Mono 5.100 3.100 0.107 0.765 4.561 0.183 This embodiment 3.797 3.014 0.129 1.122 5.277 0.211

[0096] As shown in Table 3, the improved model in this embodiment has reduced complexity, fewer parameters, and significantly improved computational efficiency. Simultaneously, the model accuracy is also high in terms of error metrics. The comparison between the method in this embodiment and the Lite-Mono method is as follows: Figure 6 As shown.

[0097] Example 2:

[0098] The all-weather autonomous driving scene recognition system includes:

[0099] The patch complementary pre-training module is configured to: acquire MR images of different modalities and divide them into scene recognition modules; the scene recognition module is configured to: acquire image information of the driving environment, divide it into daytime scenes and nighttime scenes based on the trained recognition model, perform image enhancement when the image information is nighttime, and use the daytime scene image and the enhanced nighttime scene image to perform depth estimation in order to achieve scene recognition.

[0100] During training, the recognition model acquires original daytime scene images, generates nighttime scene images based on a recurrent generative adversarial network, and mixes them with the original daytime scene images to form a full-day dataset. The full-day dataset uses a classification decision engine to distinguish between daytime and nighttime scene images. Daytime scene images are used to achieve scene recognition using a depth estimation model, while nighttime scene images are used to achieve scene recognition after image enhancement processing using a depth estimation model.

[0101] Example 3:

[0102] This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the all-weather autonomous driving scene recognition method described in Embodiment 1 above.

[0103] Example 4:

[0104] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the all-weather autonomous driving scene recognition method described in Embodiment 1 above.

[0105] The steps involved in Embodiments 2 to 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0106] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A scene recognition method for all-weather autonomous driving, characterized in that, Includes the following steps: Image information of the driving environment is acquired and divided into daytime and nighttime scenes based on the trained recognition model. When the image information is nighttime, image enhancement is performed. Depth estimation is performed using the images of the daytime scene and the enhanced nighttime scene to achieve scene recognition. During training, the recognition model acquires original daytime scene images, generates nighttime scene images based on a recurrent generative adversarial network, and mixes them with the original daytime scene images to form a full-day dataset. The full-day dataset uses a classification decision-maker to distinguish between daytime and nighttime scene images. Daytime scene images are recognized using a depth estimation model, while nighttime scene images are recognized using a depth estimation model after image enhancement. The full-day dataset uses a classification decision engine to distinguish between daytime and nighttime scene images, specifically: Get the input image Extracting visual features Linear layers are used to connect image features to words and embed them in a space, utilizing projection matrices. Visual features Convert to language embedding tags Image-text pairs are formed for training. The training process for a classification decision engine includes the following steps: For each image Generate multi-turn dialogue data ,in It is the total number of rounds; The generated multi-turn dialogue data forms a sequence, and the answers in the sequence are regarded as the responses of the classification decision-maker. The target answer is generated through calculation. The probability is determined in the current prediction label. Command and response markers from all previous rounds; No. Wheel instructions Considered as: ； in, For the input image, For the generated multi-turn dialogue data; For a sequence of length L, the target answer is generated by calculating the following formula. The probability of: in, These are trainable parameters. and These are the current prediction markers. The instruction and response markers from all previous rounds.

2. The scene recognition method for all-weather autonomous driving as described in claim 1, characterized in that, Obtain the original daytime scene image, and generate the nighttime scene image based on a recurrent generative adversarial network, specifically: ； ； ； Recurrent Generative Adversarial Networks (GANs) include generators ( G A , G B ) and discriminator ( D A , D B ), where the translator between domain A and domain B is and , Trained to distinguish real images and translated images , Trained to distinguish images and Training is performed using adversarial loss and cycle consistency loss, through incentives. and They are inverse mappings of each other, making and .

3. The scene recognition method for all-weather autonomous driving as described in claim 1, characterized in that, The depth estimation model is a hybrid CNN and Transformer architecture. It implicitly performs alternative connections through summation operations and introduces linear attention instead of cross-covariance attention.

4. A scene recognition system for all-weather autonomous driving, employing the scene recognition method for all-weather autonomous driving as described in any one of claims 1-3, characterized in that, include The scene recognition module is configured to: acquire image information of the driving environment, divide it into daytime scene and nighttime scene based on the trained recognition model, perform image enhancement when the image information is nighttime, and perform depth estimation using the daytime scene image and the enhanced nighttime scene image to achieve scene recognition; During training, the recognition model acquires original daytime scene images, generates nighttime scene images based on a recurrent generative adversarial network, and mixes them with the original daytime scene images to form a full-day dataset. The full-day dataset uses a classification decision engine to distinguish between daytime and nighttime scene images. Daytime scene images are used to achieve scene recognition using a depth estimation model, while nighttime scene images are used to achieve scene recognition after image enhancement processing using a depth estimation model.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the steps in the scene recognition method for all-weather autonomous driving as described in any one of claims 1-3.

6. A computer device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the program, it implements the steps in the scene recognition method for all-weather autonomous driving as described in any one of claims 1-3.

Images