Obstacle detection model training method, obstacle detection method, and storage medium

Abstract

The application discloses a kind of obstacle detection model training method, obstacle detection method and storage medium, electronic equipment, vehicle.The training method includes: obtaining driving image dataset, wherein the driving image dataset includes ordinary driving image and specific obstacle image under different driving conditions;The ordinary driving image and the specific obstacle image in driving image dataset are expanded, and the training image dataset is obtained;Obstacle detection model is constructed, and the training image in training image dataset is used to train the obstacle detection model, and the target obstacle detection model is obtained.The training method is rich in training image dataset, improves the generalization ability of the target obstacle detection model obtained by training, realizes the effective identification of specific obstacle in target obstacle detection model to driving image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of obstacle detection, and in particular to a training method of an obstacle detection model, an obstacle detection method, a storage medium, an electronic device and a vehicle. BACKGROUND

[0002] The related obstacle monitoring model and obstacle recognition algorithm can solve the problem of obstacle detection and recognition in most common scenarios, but when applied to an advanced driving assistance system (ADAS), it still cannot meet the user's demand for driving safety. SUMMARY

[0003] The present application aims to at least partially solve one of the technical problems in the related art. To this end, one object of the present application is to provide a training method of an obstacle detection model, which improves the generalization ability of the target obstacle detection model obtained by training and realizes effective recognition of specific obstacles in driving images by the target obstacle detection model.

[0004] A second object of the present application is to provide an obstacle detection method.

[0005] A third object of the present application is to provide a computer-readable storage medium.

[0006] A fourth object of the present application is to provide an electronic device

[0007] A fifth object of the present application is to provide a vehicle.

[0008] To achieve the above objects, the training method of an obstacle detection model according to the first aspect of the present application comprises: obtaining a driving image dataset, wherein the driving image dataset comprises ordinary driving images and specific obstacle images under different driving conditions; performing expansion processing on the ordinary driving images and the specific obstacle images in the driving image dataset to obtain a training image dataset; constructing an obstacle detection model and training the obstacle detection model using training images in the training image dataset to obtain a target obstacle detection model.

[0009] The training method of an obstacle detection model according to the present application expands the ordinary driving images and the specific obstacle images in the driving image dataset to enrich the training image dataset, so that the obstacle detection model is more effectively trained, the generalization ability of the obstacle detection model obtained by training is improved, and the target obstacle detection model effectively recognizes specific obstacles in driving images.

[0010] In addition, the method for training the obstacle detection model according to the above-mentioned embodiments of the present application can further have the following additional technical features.

[0011] According to an embodiment of the present application, the specific obstacle image dataset includes specific obstacle images collected by a monocular camera on different specific obstacles.

[0012] According to an embodiment of the present application, the augmentation processing of the normal driving image and the specific obstacle image in the driving image dataset includes: extracting a region of interest in the normal driving image to obtain a first image, wherein the region of interest includes a drivable area of the vehicle; extracting an obstacle in the specific obstacle image to obtain a second image; performing scaling processing on the second image, and synthesizing the scaled second image with the first image to obtain a third image; performing style transfer on the corresponding third image using the normal driving image, and adding the style-transferred third image to the driving image dataset.

[0013] According to an embodiment of the present application, the augmentation processing of the normal driving image and the specific obstacle image in the driving image dataset includes: inputting images in a new driving image dataset to a pre-trained generation model to obtain generated new images, wherein the pre-trained generation model includes a trained Wasserstein GAN model and a trained variational autoencoder VAE model; cleaning the generated new images based on a preset image standard, and adding images meeting the preset image standard to the new driving image dataset.

[0014] According to an embodiment of the present application, the augmentation processing of the normal driving image and the specific obstacle image in the driving image dataset includes: performing enhancement processing on images in a new driving image dataset, and the enhancement processing mode includes at least one of color jittering, adding Gaussian noise, left-right flipping, sobel operator filtering, mixup, and image stitching.

[0015] According to an embodiment of the present application, the image stitching mode includes at least one of the following: stitching a copied image of a first target image with the first target image, wherein the first target image is determined from the driving image dataset; stitching the copied image of the first target image after rotating the copied image; stitching a second target image with the first target image, wherein the second target image is an image of the same category but different resolution as the first target image selected from the driving image dataset; and stitching the first target image after rotating the first target image with the second target image.

[0016] According to one embodiment of the present application, the backbone network of the obstacle detection model adopts a first RepVGG model, the obstacle detection model is sequentially provided with a channel attention mechanism, a spatial attention mechanism and an FcaNet attention mechanism before a global average pooling layer of the RepVGG model, the obstacle detection model is trained by using training images in the training image dataset to obtain a target obstacle detection model, comprising: unsupervised training the first RepVGG model by using the training images in the training image dataset to obtain a trained first RepVGG model; replacing a backbone network of a YOLOv5 model with the trained first RepVGG model; labeling the training images in the training image dataset, and supervised training the replaced YOLOv5 model by using the labeled training images to obtain a trained YOLOv5 model, denoted as a teacher network; constructing a student network, and performing knowledge distillation on the student network by using the teacher network to obtain the target obstacle detection model, the student network adopts a second RepVGG model.

[0017] To achieve the above object, the second aspect of the present application proposes an obstacle detection method, comprising: obtaining a to-be-detected image; inputting the to-be-detected image into a target obstacle detection model to obtain an obstacle detection result of the to-be-detected image, wherein the target obstacle detection model is obtained by using the training method of the obstacle detection model proposed in the first aspect of the present application.

[0018] To achieve the above object, the third aspect of the present application proposes a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the training method of the obstacle detection model proposed in the first aspect of the present application, or implement the obstacle detection method proposed in the second aspect of the present application.

[0019] To achieve the above object, the fourth aspect of the present application proposes an electronic device, comprising a memory and a processor, and the memory stores a computer program, and the computer program is executed by the processor to implement the obstacle detection method proposed in the second aspect of the present application.

[0020] To achieve the above object, the fifth aspect of the present application proposes a vehicle, comprising a monocular camera and an electronic device proposed in the fourth aspect of the present application.

[0021] Additional aspects and advantages of the present application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 is a flowchart of a method for training an obstacle detection model according to an embodiment of the present application;

[0023] Figure 2 is a flowchart of a method for expanding a driving image dataset according to an embodiment of the present application;

[0024] Figure 3 is a flowchart of a method for expanding a driving image dataset according to another embodiment of the present application;

[0025] Figure 4 is a flowchart of a method for obtaining a target obstacle detection model according to an embodiment of the present application;

[0026] Figure 5 is a flowchart of an obstacle detection method according to an embodiment of the present application;

[0027] Figure 6 is a structural block diagram of an electronic device according to an embodiment of the present application;

[0028] Figure 7 is a schematic diagram of a vehicle according to an embodiment of the present application. DETAILED DESCRIPTION

[0029] Embodiments of the present application are described in detail below with reference to the accompanying drawings, in which like or similar elements are denoted by the same or similar reference signs, and the embodiments described below are examples intended to explain the present application and should not be understood as limiting the present application.

[0030] The method for training an obstacle detection model, the obstacle detection method, the storage medium, the electronic device, and the vehicle according to the embodiments of the present application are described in detail below in conjunction with Descriptions 1-7 and the specific embodiments.

[0031] Figure 1 is a flowchart of a method for training an obstacle detection model according to an embodiment of the present application. As shown in Figure 1 the method for training an obstacle detection model can include:

[0032] S101, obtaining a driving image dataset, wherein the driving image dataset includes ordinary driving images and specific obstacle images under different driving conditions.

[0033] Specifically, the ordinary driving images can be obtained from a driving open-source picture dataset on the Internet, or from a driving picture dataset specially collected by a test vehicle. The ordinary driving images can also be obtained from other sources. The embodiment of the present application does not limit the way of obtaining the ordinary driving images. Since the obtained ordinary driving images are driving images collected by front cameras of various vehicle models from different driving scenes, the generalization performance of the trained target obstacle detection model is improved to a certain extent, and the vehicle-mounted front camera position is robust to a certain extent.

[0034] It should be noted that when obtaining the ordinary driving images, attention should be paid to covering different driving scenes. Obtaining ordinary driving images in different driving scenes can enrich the driving image dataset, which helps to improve the richness of the training image dataset, and thus improves the generalization performance of the trained obstacle detection model to a certain extent.

[0035] In an embodiment of the present application, the specific obstacle image dataset includes specific obstacle images collected by monocular cameras for different specific obstacles.

[0036] Specifically, when obtaining specific obstacle images, the amount of specific obstacle images obtained from the driving open-source picture dataset on the Internet is small due to the rarity of specific obstacles. To improve the recognition effect of the trained target obstacle detection model on specific (rare) obstacles, when obtaining specific obstacle images with the help of the existing specific obstacle dataset, some scenes can be arranged to collect specific obstacle images, so as to focus on collecting specific obstacles (rare obstacles).

[0037] For example, if the related driving perception algorithm cannot effectively identify some movable road barriers, similar road barriers can be placed on a special test road section in a park, and then the test vehicle can be used to collect pictures from different directions. If the related driving perception algorithm cannot effectively identify some fixed walls, more picture data of the fixed object can be collected at different times, under different lighting and climate conditions, and from different driving angles.

[0038] It should be noted that the obtained specific obstacle images can be directly used after labeling, or can be labeled and used after being expanded.

[0039] In the embodiment of the present application, when obtaining the driving image dataset, the ordinary driving images and specific obstacle images collected by vehicle-mounted front monocular cameras with different configurations and different models are used to improve the generalization performance of the trained target obstacle detection model to a certain extent.

[0040] S102, the ordinary driving image and the specific obstacle image in the driving image dataset are expanded to obtain a training image dataset.

[0041] To enrich the training image dataset of the training target obstacle detection model, image synthesis, image generation, and image data enhancement processing are performed based on the obtained ordinary driving image and specific obstacle image to expand and enrich the training image dataset, so as to improve the generalization performance of the trained obstacle detection model to a certain extent.

[0042] In an embodiment of the present application, as shown in Figure 2 The expansion processing of the ordinary driving image and the specific obstacle image in the driving image dataset can include:

[0043] S21, a region of interest in the ordinary driving image is extracted to obtain a first image, wherein the region of interest includes a vehicle drivable area;

[0044] S22, an obstacle in the specific obstacle image is extracted to obtain a second image;

[0045] S23, the second image is scaled and the scaled second image is combined with the first image to obtain a third image;

[0046] S24, the third image is style transferred using the ordinary driving image, and the style transferred third image is added to the driving image dataset.

[0047] It should be noted that image synthesis refers to using existing images to synthesize new images, such as using existing ordinary driving images and specific obstacle images to synthesize new images.

[0048] Specifically, the vehicle drivable area in the ordinary driving image and the specific obstacle in the specific obstacle image are extracted respectively, and the extracted vehicle drivable area (first image) and specific obstacle area (second image) are combined to obtain a new image (third image), which is added to the training image dataset as a training image to enrich the training image dataset.

[0049] When the region of interest (vehicle drivable area, wherein the drivable area generally refers to the road surface area) is selected from the ordinary driving image, the vehicle drivable area in the ordinary driving image can be manually labeled. Due to the large amount of work and low efficiency of manually labeling the ordinary driving image, a part of the ordinary driving image can be labeled by manual labeling, and the deep dual resolution network (DDRNet) model can be trained using the part of the labeled ordinary driving image. The region of interest is generated based on the pre-trained DDRNet model to improve the labeling efficiency of the ordinary driving image.

[0050] It should be noted that when the region of interest is generated based on the trained DDRNet model, the DDRNet model trained based on the annotated ordinary driving image needs to be verified, that is, whether the region of interest generated by the trained DDRNet model is consistent with the drivable area of the vehicle. If not, the trained DDRNet model needs to be adjusted to ensure that the region of interest generated by the adjusted DDRNet model (pre-trained DDRNet model) is consistent with the drivable area of the vehicle.

[0051] It can be implemented to extract the specific obstacle in the obstacle image by matting, wherein the matting can be manually cut. After manually labeling a part, the specific obstacle image with the labeled part is used to train the DeepLabv3+ model of the deep learning segmentation network, and the pre-trained DeepLabv3+ model is used to automatically mat the unlabeled specific obstacle image.

[0052] It should be noted that when the obstacle is extracted based on the trained DeepLabv3+ model, the DeepLabv3+ model trained based on the labeled specific obstacle image needs to be verified, that is, whether the obstacle extracted by the trained DeepLabv3+ model is consistent with the obstacle in the specific obstacle image. If not, the trained DeepLabv3+ model needs to be adjusted to ensure that the obstacle extracted by the adjusted DeepLabv3+ model (pre-trained DeepLabv3+ model) is consistent with the obstacle in the specific obstacle image.

[0053] After obtaining the vehicle drivable area in the ordinary driving image and the obstacle in the specific obstacle image, in order to enrich the training image data set, the extracted obstacle region is scaled in different proportions, and the scaled obstacle picture is placed in different positions of the vehicle drivable area to obtain the synthesized new image. In order to avoid the difference between the obstacle picture inserted into the region of interest and the ordinary driving image corresponding to the region of interest in style, which leads to that the target obstacle detection model learned only recognizes the places with different styles and fails to correctly understand the real features of the obstacle, the synthesized new image is input into the trained CycleGANS model of the generative adversarial network. The CycleGANS model performs style transfer on the synthesized new image according to the style of the ordinary driving image corresponding to the region of interest, so that the synthesized new image is closer to the real scene, so that the target obstacle detection model learned understands the real features of the obstacle.

[0054] It should be noted that the specific manner of style transfer is to use one picture as a style picture, and then use an algorithm to make the picture to be transferred have the style of the aforementioned picture. This operation can make the processed synthetic data closer to the real scene.

[0055] It should be noted that the embodiment of the present application can process all or part of the ordinary driving image in the driving image data set. The embodiment of the present application can perform obstacle extraction processing on all or part of the specific obstacle image in the driving image data set. When the extracted obstacle region is synthesized with the region of interest in the ordinary driving image, one extracted obstacle can be scaled to one or more regions of interest in the ordinary driving image. The scaling ratio of the obstacle can be determined according to the ordinary driving image corresponding to the region of interest where the obstacle is to be placed, so as to be closer to the real scene.

[0056] In the embodiment of the present application, the new image (third image) obtained after the above synthesis processing is added to the driving image data set, so that the image generation and / or image data enhancement processing can be performed on the driving image data set, and the new image obtained after the synthesis processing can also be subjected to image generation and / or image data enhancement processing, thereby expanding and enriching the training image data set.

[0057] In an embodiment of the present application, as shown in Figure 3 The expansion processing of the ordinary driving image and the specific obstacle image in the driving image data set includes:

[0058] S31, input the image in the new driving image data set to the pre-trained generation model to obtain a new image after generation, wherein the pre-trained generation model includes a trained Wasserstein GAN model and a trained variational autoencoder VAE model;

[0059] S32, based on the preset image standard, the new image after generation is cleaned, and the image meeting the preset image standard is added to the new driving image data set.

[0060] It should be noted that image generation refers to generating a new picture by using an algorithm on an existing picture, such as generating a new ordinary driving image and a specific obstacle image by using an existing ordinary driving image and a specific obstacle image.

[0061] In an embodiment of the present application, a new image can be generated based on a pre-trained generative adversarial network (GAN) or a pre-trained variational autoencoder (VAE). The variational autoencoder is a structure composed of an encoder and a decoder, which is trained to minimize the reconstruction error between the encoded and decoded data and the initial data. In practice, a Wasserstein GAN model can be constructed, and the constructed Wasserstein GAN model can be trained based on a part of the image pairs in the driving image dataset, so that the trained Wasserstein GAN network can generate new image data. In practice, a VAE model can be constructed, and the constructed VAE model can be trained based on the image pairs in the driving image dataset, so that the trained VAE model can generate new image data.

[0062] In an embodiment of the present application, a new image is generated based on the images in the driving image dataset and the new image (third image) obtained by the above synthesis processing using a pre-trained generative model. Specifically, the images in the driving image dataset and the new image (third image) obtained by the above synthesis processing are input into the pre-trained generative model, and the pre-trained generative model performs generation processing on the images in the driving image dataset and the new image (third image) obtained by the above synthesis processing to obtain a new image after generation processing. Since the new image after generation processing may not meet the requirements, the new image generated by the generation model is cleaned based on a preset image standard, such as an image close to a real scene, and images meeting the preset image standard are added to the training image dataset as training images.

[0063] In an embodiment of the present application, the new image after generation processing is added to the driving image dataset, so that the image data enhancement processing can be performed on the new image after generation processing while the image data enhancement processing is performed on the driving image dataset, thereby expanding and enriching the training image dataset.

[0064] In an embodiment of the present application, the ordinary driving images and the specific obstacle images in the driving image dataset are expanded, including:

[0065] The images in the new driving image dataset are enhanced, and the enhancement processing includes at least one of color jittering, adding Gaussian noise, left-right flipping, Sobel operator filtering, mixup, and image splicing.

[0066] Image data augmentation is performed on the driving image dataset containing newly synthesized images and / or newly generated processed images. Specifically, the following image data augmentation methods can be used to enrich the training image dataset: For example, color dithering, such as adjusting the saturation, brightness, and contrast of images in the driving image dataset, can be used. Gaussian noise, such as noise whose probability density function follows a normal distribution, can be added to the images in the driving image dataset. Images in the driving image dataset can be flipped horizontally. Images in the driving image dataset can be processed using the Sobel operator (a commonly used image processing operator for edge detection) to enrich the training image dataset. The training image dataset can be enriched using mixup (merging samples and labels in the same way to obtain new training samples). Image stitching can be performed on the images in the driving image dataset to enrich the training image dataset.

[0067] In one embodiment of the present invention, the image stitching method includes at least one of the following:

[0068] A copy of the first target image is stitched together with the first target image, which is determined from the driving image dataset.

[0069] The copied image of the first target image is rotated and then stitched together with the first target image;

[0070] The second target image is stitched together with the first target image. The second target image is an image of the same category but different resolution as the first target image, selected from the driving image dataset.

[0071] The first target image is rotated and then stitched together with the second target image.

[0072] To maximize sample richness, image stitching methods may include: determining a first target image from a driving image dataset, copying the first target image, and stitching the copy with the first target image in the left-right and up-down directions; or rotating the copy of the first target image 180 degrees clockwise and then stitching it with the first target image in the left-right and up-down directions; or selecting a second image of the same category but different resolution from the driving image dataset, stitching the second target image with the first target image in the left-right and up-down directions; or rotating the second image 180 degrees clockwise and then stitching it with the first target image in the left-right and up-down directions.

[0073] S103, construct an obstacle detection model and train the obstacle detection model using training images in the training image dataset to obtain the target obstacle detection model.

[0074] In an embodiment of the present application, the backbone network of the obstacle detection model can adopt a first RepVGG model, and the obstacle detection model is sequentially provided with a channel attention mechanism, a spatial attention mechanism and an FcaNet attention mechanism before a global average pooling layer of the RepVGG model.

[0075] In an embodiment of the present application, a first RepVGG model, such as a RepVGG_B3g4 model, can be selected as the backbone network of the obstacle detection model. To improve the feature extraction capability of the obstacle detection model on valuable areas, multiple attention mechanisms, such as a channel attention mechanism, a spatial attention mechanism and an FcaNet, can be added to the first RepVGG model. For example, the channel attention mechanism, the spatial attention mechanism and the FcaNet attention mechanism are sequentially added in this order before the global average pooling layer of the first RepVGG model.

[0076] In an embodiment of the present application, as shown in Figure 4 training the obstacle detection model using the training images in the training image dataset to obtain a target obstacle detection model, comprising:

[0077] S41, unsupervised training the first RepVGG model using the training images in the training image dataset to obtain a trained first RepVGG model;

[0078] S42, replacing the backbone network of the YOLOv5 model with the trained first RepVGG model;

[0079] S43, labeling the training images in the training image dataset, and supervised training the replaced YOLOv5 model using the labeled training images to obtain a trained YOLOv5 model, denoted as a teacher network;

[0080] S44, constructing a student network, and performing knowledge distillation on the student network using the teacher network to obtain the target obstacle detection model, wherein the student network adopts a second RepVGG model.

[0081] Specifically, in the process of training the first RepVGG model, the idea of contrastive learning can be used to improve the feature extraction capability of the first RepVGG model using the SimCLv2 method in unsupervised learning. Add a detection head to the first RepVGG model trained in an unsupervised manner, so that the target obstacle detection model trained can not only detect obstacles, but also detect traffic signs, lane lines, vehicles, etc. Specifically, replace the backbone network of the YOLOv5 model with the trained first RepVGG model, wherein the YOLOv5 model is a backbone network with a target detection head (such as traffic signs, lane lines, vehicles), and the YOLOv5 model is supervised trained using labeled training images. It should be noted that the training image data in the training image data set contains labeled obstacle, traffic sign, vehicle and other target annotation information that needs to be detected. Thus, the trained YOLOv5 model can detect obstacles, traffic signs, vehicles, etc. in the image.

[0082] To deploy the trained YOLOv5 model to the vehicle-mounted hardware, a lightweight deep learning network, i.e. a student network, can be designed based on the second RepVGG model, such as the RepVGG_B1g2 model, and the student network is knowledge distilled from the trained YOLOv5 model as a teacher network, i.e. the student network is accelerated by model pruning, weight quantization, and tensor decomposition, so that the trained student network is as close as possible to the teacher network in recognition ability. The obtained student network is deployed in the vehicle-mounted hardware to perform actual image recognition. It should be noted that the trained student network can not only recognize obstacles, but also recognize traffic signs, vehicles, etc.

[0083] The embodiment of the present application considers not only the shortcomings of current deep learning models, such as large calculation amount and high requirements for vehicle-mounted hardware, but also the fact that adding attention mechanisms can make the model focus on more meaningful areas. Therefore, on the one hand, multiple attention mechanisms are added to improve the recognition ability of the obstacle detection model, and on the other hand, a knowledge distillation method is used to ensure the recognition accuracy of the target obstacle detection model as much as possible under the premise of greatly compressing the parameter amount of the obstacle detection model, so as to obtain a lightweight obstacle detection model that can be deployed on a vehicle. The trained target obstacle detection model can apply the detection result to the ADAS auxiliary driving system in actual application, and effectively recognize specific (uncommon) obstacles.

[0084] The training method of the obstacle detection model of the embodiment of the present application obtains ordinary driving images by means of an existing driving open source picture data set, and collects specific obstacle images by using a plurality of different configurations and models of vehicle-mounted front monocular cameras, so as to enrich the driving image data set. In order to enrich the training image data set and improve the generalization performance of the trained obstacle detection model, the ordinary driving images and the specific obstacle images in the driving image data set are expanded in the manner of image synthesis, image generation, image data enhancement and the like, a plurality of expansion training image data methods are adopted to improve the training data end, the richness of the training data end sample is improved, and the obstacle detection model is more effectively trained.

[0085] The obstacle detection model constructed in the embodiment of the present application uses CNN (Convolutional Neural Network) as a backbone network, and uses contrast learning and the like to perform unsupervised training on the constructed obstacle detection model. Unsupervised training can help the obstacle detection model to perform more effective feature extraction, to a certain extent, solve the problem that the specific obstacle data is less and the obstacle detection model is difficult to perform effective feature extraction, and improve the recognition accuracy and recall rate of the obstacle detection model for uncommon obstacles. On the basis of the YOLOv5 model, the backbone network thereof is changed, and a plurality of detection heads are added, so that it can not only recognize the obstacles in front of the vehicle, but also be applicable to a plurality of tasks including lane line detection and traffic sign detection. The YOLOv5 model obtained by supervised training is simplified by using a knowledge distillation method, to obtain an obstacle detection model that can be deployed on a vehicle, to realize more effective recognition of specific (uncommon) obstacles in the driving process based on a monocular camera.

[0086] The training method of the obstacle detection model of the embodiment of the present application expands the ordinary driving images and the specific obstacle images collected based on the monocular camera in the driving image data set, to enrich the training image data set, so that the obstacle detection model is more effectively trained, the generalization ability of the trained obstacle detection model is improved, and the target obstacle detection model effectively recognizes the specific obstacles in the driving image collected based on the monocular camera.

[0087] The present application provides an obstacle detection method.

[0088] Figure 6 is a structural block diagram of an electronic device according to an embodiment of the present application. As shown in Figure 6 The obstacle detection method can include:

[0089] S201, acquiring an image to be detected;

[0090] Specifically, the driving image can be collected based on the front-mounted monocular camera on the vehicle as the to-be-detected image.

[0091] In S202, the to-be-detected image is input into the target obstacle detection model to obtain an obstacle detection result of the to-be-detected image, wherein the target obstacle detection model is obtained by using the training method of the obstacle detection model as described above.

[0092] Specifically, the to-be-detected image is input into the target obstacle detection model, and the target obstacle detection model identifies the obstacles, lane lines, traffic signs, vehicles, etc. in the to-be-detected image and outputs a detection result. The detection result can be displayed on the screen of the vehicle terminal, such as the screen of the central control screen, in the form of a target detection frame diagram.

[0093] The obstacle detection method of the embodiment of the present application uses the target obstacle detection model obtained by training to detect the driving image in the driving process, and the detection result can be applied to the ADAS system to achieve effective identification of specific (uncommon) obstacles.

[0094] The present application provides a computer-readable storage medium.

[0095] In this embodiment, the computer-readable storage medium stores a computer program, and the computer program is executed by the processor to implement the training method of the obstacle detection model as described above, or to implement the obstacle detection method as described above.

[0096] The present application provides an electronic device.

[0097] In this embodiment, the electronic device includes a memory and a processor, and the memory stores a computer program which is executed by the processor to implement the obstacle detection method as described above.

[0098] Figure 6 is a structural block diagram of the electronic device of an embodiment of the present application. As shown in Figure 6 The electronic device 500 includes a processor 501 and a memory 503. The processor 501 and the memory 503 are connected, for example, through a bus 502. Optionally, the electronic device 500 can also include a transceiver 504. It should be noted that in actual applications, the transceiver 504 is not limited to one, and the structure of the electronic device 500 does not constitute a limitation on the embodiments of the present application.

[0099] The processor 501 can be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic device, transistor logic device, hardware component, or any combination thereof. It can implement or execute various exemplary logical blocks, modules and circuits described in conjunction with the present disclosure. The processor 501 can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0100] The bus 502 can include a path for transmitting information between the above-mentioned components. The bus 502 can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus 502 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 In the figure, only one thick line is used to represent, but it does not mean that there is only one bus or one type of bus.

[0101] The memory 503 is used to store a computer program corresponding to the obstacle detection method of the above-mentioned embodiments of the present application, which is controlled and executed by the processor 501. The processor 501 is used to execute the computer program stored in the memory 503 to realize the content shown in the above-mentioned method embodiments. Figure 6 The electronic device 500 shown is only an example, and should not limit the functions and use range of the embodiments of the present application.

[0102] The present application provides a vehicle.

[0103] Figure 7 The figure is a schematic view of a vehicle according to an embodiment of the present application. Figure 7 As shown, the vehicle 1000 can include a monocular camera 600 and an electronic device 500 as described above.

[0104] Specifically, the monocular camera 600 at the front end of the vehicle and the electronic device 500 as described above are used to detect the driving image collected by the monocular camera 600 during driving. The detection result output by the electronic device 500 can be applied to the ADAS system to realize effective identification of specific (uncommon) obstacles and improve driving safety.

[0105] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0106] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0107] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0108] In the description of the application, it should be understood that the orientation or positional relationship indicated by the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", "axial", "radial", "circumferential" and the like is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the application and simplifying the description, and does not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the application.

[0109] In addition, the terms "first", "second" are only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the application, the meaning of "a plurality of" is at least two, such as two, three, etc., unless otherwise explicitly specified and limited.

[0110] In the present application, unless otherwise explicitly specified and limited, the terms "mounting", "connecting", "connecting", "fixing" and the like should be understood broadly, for example, it can be fixedly connected, or it can be detachably connected, or it can be integrated; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium; it can be the internal communication of two elements or the interaction relationship between two elements, unless otherwise explicitly limited. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0111] In the present application, unless otherwise explicitly specified and limited, the first feature "on" or "under" the second feature can be that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, the first feature "above", "above" and "above" the second feature can be that the first feature is directly above or obliquely above the second feature, or only indicates that the horizontal height of the first feature is higher than that of the second feature. The first feature "below", "below" and "below" the second feature can be that the first feature is directly below or obliquely below the second feature, or only indicates that the horizontal height of the first feature is less than that of the second feature.

[0112] Although the embodiments of the application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as a limitation on the application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the application.

Claims

1. A training method for an obstacle detection model, characterized in that, The training method includes: Obtain a driving image dataset, which includes ordinary driving images and images of specific obstacles under different driving conditions; The ordinary driving images and specific obstacle images in the driving image dataset are augmented to obtain a training image dataset. An obstacle detection model is constructed, and the obstacle detection model is trained using training images from the training image dataset to obtain a target obstacle detection model; The backbone network of the obstacle detection model adopts the first RepVGG model. Before the global average pooling layer of the RepVGG model, the obstacle detection model sequentially employs a channel attention mechanism, a spatial attention mechanism, and an FcaNet attention mechanism. The obstacle detection model is trained using training images from the training image dataset to obtain a target obstacle detection model, including: The first RepVGG model is trained unsupervised using the training images in the training image dataset to obtain the trained first RepVGG model. Replace the backbone network of the YOLOv5 model with the trained first RepVGG model; The training images in the training image dataset are labeled, and the replaced YOLOv5 model is trained in a supervised manner using the labeled training images to obtain the trained YOLOv5 model, which is denoted as the teacher network. A student network is constructed, and the student network is subjected to knowledge distillation using the teacher network to obtain the target obstacle detection model. The student network adopts the second RepVGG model.

2. The training method for the obstacle detection model according to claim 1, characterized in that, The specific obstacle image dataset includes specific obstacle images collected by a monocular camera for different specific obstacles.

3. The training method for the obstacle detection model according to claim 1, characterized in that, The image data dataset is augmented with ordinary driving images and images of specific obstacles, including: Extract the region of interest from the ordinary driving image to obtain a first image, wherein the region of interest includes the drivable area of ​​the vehicle; Extract the obstacles from the specific obstacle image to obtain a second image; The second image is scaled, and the scaled second image is combined with the first image to obtain a third image; The style transfer of the corresponding third image is performed using the ordinary driving image, and the style-transferred third image is added to the driving image dataset.

4. The training method for the obstacle detection model according to claim 3, characterized in that, The image data dataset is augmented with ordinary driving images and images of specific obstacles, including: Images from a new driving image dataset are input into a pre-trained generative model to obtain new generated images. The pre-trained generative model includes a trained Wasserstein GAN model and a trained variational autoencoder (VAE) model. The generated new images are cleaned based on preset image standards, and images that meet the preset image standards are added to the new driving image dataset.

5. The training method for the obstacle detection model according to claim 3 or 4, characterized in that, The image data dataset is augmented with ordinary driving images and images of specific obstacles, including: Enhancement processing is performed on images in the new driving image dataset. Enhancement processing methods include at least one of the following: color dithering, adding Gaussian noise, horizontal flipping, Sobel operator filtering, Mixup, and image stitching.

6. The training method for the obstacle detection model according to claim 5, characterized in that, Image stitching methods include at least one of the following: A copy of the first target image is stitched together with the first target image, which is determined from the driving image dataset; The copied image of the first target image is rotated and then stitched together with the first target image; The second target image is stitched together with the first target image, wherein the second target image is an image of the same category but different resolution selected from the driving image dataset as the first target image; The first target image is rotated and then stitched together with the second target image.

7. An obstacle detection method, characterized in that, The method includes: Acquire the image to be detected; The image to be detected is input into the target obstacle detection model to obtain the obstacle detection result of the image to be detected, wherein the target obstacle detection model is obtained using the training method of the obstacle detection model as described in any one of claims 1-6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the training method of the obstacle detection model as described in any one of claims 1-6, or implements the obstacle detection method as described in claim 7.

9. An electronic device, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the computer program is executed by the processor, it implements the obstacle detection method as described in claim 7.

10. A vehicle, characterized in that, Includes a monocular camera and the electronic device as described in claim 9.

Images