Training method of obstacle detection model, obstacle detection method and storage medium
By expanding the driving image data set and building an obstacle detection model with attention mechanism, the shortcomings of the obstacle detection model in the prior art in identifying specific obstacles are solved, efficient identification of unusual obstacles is achieved, and driving is improved Safety.
Patent Information
- Application Number
- CN202310478355.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-04-26
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2043-04-26
AI Technical Summary
Existing obstacle detection models are difficult to meet driving safety needs in advanced driving assistance systems, especially when identifying specific uncommon obstacles.
By acquiring and expanding driving image data sets, including ordinary driving images and specific obstacle images, using image synthesis, image generation and data augmentation technologies, an obstacle detection model with channel and spatial attention mechanisms is constructed, and the knowledge distillation method is optimized Model, improve the generalization ability and recognition accuracy of the model.
The obstacle detection model has significantly improved the recognition ability of specific obstacles, enhanced the recognition accuracy and recall of uncommon obstacles, and can be effectively applied to advanced driving assistance systems.
Smart Images

Figure CN118918403A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of obstacle detection, and in particular to a training method for an obstacle detection model, an obstacle detection method and storage medium, electronic equipment, and a vehicle. Background Art
[0002] Related obstacle monitoring models and obstacle recognition algorithms can solve obstacle detection and recognition problems in most common scenarios. However, when applied to Advanced Driving Assistance System (ADAS), they still cannot meet users' needs for driving safety. Summary of the invention
[0003] The present invention aims to solve one of the technical problems in the related art at least to a certain extent. To this end, one purpose of the present invention is to propose a training method for an obstacle detection model, improve the generalization ability of the trained target obstacle detection model, and realize the effective recognition of specific obstacles in driving images by the target obstacle detection model.
[0004] The second objective of the present invention is to provide an obstacle detection method.
[0005] A third object of the present invention is to provide a computer-readable storage medium.
[0006] A fourth object of the present invention is to provide an electronic device
[0007] A fifth object of the present invention is to provide a vehicle.
[0008] To achieve the above-mentioned purpose, a first aspect of the present invention proposes a training method for an obstacle detection model, the training method comprising: obtaining a driving image dataset, wherein the driving image dataset comprises ordinary driving images and specific obstacle images under different driving conditions; performing expansion processing on the ordinary driving images and specific obstacle images in the driving image dataset to obtain a training image dataset; constructing an obstacle detection model, and training the obstacle detection model using the training images in the training image dataset to obtain a target obstacle detection model.
[0009] According to the obstacle detection model training method of the embodiment of the present invention, the ordinary driving images and specific obstacle images in the driving image dataset are expanded to enrich the training image dataset, so that the obstacle detection model can be trained more effectively, the generalization ability of the trained obstacle detection model is improved, and the target obstacle detection model can effectively identify specific obstacles in the driving images.
[0010] In addition, the obstacle detection model training method proposed in the above embodiment of the present invention may also have the following additional technical features:
[0011] According to an embodiment of the present invention, the specific obstacle image data set includes specific obstacle images captured by a monocular camera of different specific obstacles.
[0012] According to one embodiment of the present invention, the ordinary driving images and specific obstacle images in the driving image dataset are expanded, including: extracting the area of interest in the ordinary driving image to obtain a first image, wherein the area of interest includes a vehicle drivable area; extracting obstacles in the specific obstacle image to obtain a second image; scaling the second image, and synthesizing the scaled second image with the first image to obtain a third image; and performing style transfer on the corresponding third image using the ordinary driving image, and adding the style transferred third image to the driving image dataset.
[0013] According to one embodiment of the present invention, ordinary driving images and specific obstacle images in the driving image dataset are expanded, including: inputting images in a new driving image dataset into a pre-trained generation model to obtain generated new images, wherein the pre-trained generation model includes a trained WassersteinGAN model and a trained variational autoencoder VAE model; cleaning the generated new images based on preset image standards, and adding images that meet the preset image standards to the new driving image dataset.
[0014] According to one embodiment of the present invention, ordinary driving images and specific obstacle images in the driving image data set are expanded, including: enhancing the images in the new driving image data set, and the enhancement processing method includes at least one of color jittering, adding Gaussian noise, left-right flipping, Sobel operator filtering, mixup, and image stitching.
[0015] According to one embodiment of the present invention, the image stitching method includes at least one of the following: stitching a copy image of a first target image with the first target image, the first target image being determined from the driving image data set; rotating the copy image of the first target image and stitching it with the first target image; stitching a second target image with the first target image, the second target image being an image of the same category but different resolution as the first target image selected from the driving image data set; rotating the first target image and stitching it with the second target image.
[0016] According to an embodiment of the present invention, the backbone network of the obstacle detection model adopts a first RepVGG model, and the obstacle detection model is sequentially provided with a channel attention mechanism, a spatial attention mechanism, and an FcaNet attention mechanism before the global average pooling layer of the RepVGG model. The obstacle detection model is trained using the training images in the training image data set to obtain a target obstacle detection model, including: performing unsupervised training on the first RepVGG model using the training images in the training image data set to obtain a trained first RepVGG model; replacing the backbone network of the YOLOv5 model with the trained first RepVGG model; annotating the training images in the training image data set, and performing supervised training on the replaced YOLOv5 model using the annotated training images to obtain a trained YOLOv5 model, which is recorded as a teacher network; constructing a student network, and performing knowledge distillation on the student network using the teacher network to obtain the target obstacle detection model, and the student network adopts a second RepVGG model.
[0017] To achieve the above-mentioned purpose, the second aspect of the present invention proposes an obstacle detection method, which includes: acquiring an image to be detected; inputting the image to be detected into a target obstacle detection model to obtain an obstacle detection result of the image to be detected, wherein the target obstacle detection model is obtained using the obstacle detection model training method proposed in the first aspect of the present invention.
[0018] To achieve the above-mentioned objectives, the third aspect of the present invention proposes a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the obstacle detection model training method proposed in the first aspect of the present invention is implemented, or the obstacle detection method proposed in the second aspect of the present invention is implemented.
[0019] To achieve the above-mentioned purpose, the fourth aspect of the present invention proposes an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the obstacle detection method proposed in the second aspect of the present invention is implemented.
[0020] To achieve the above-mentioned purpose, a fifth embodiment of the present invention proposes a vehicle, comprising a monocular camera and an electronic device as proposed in the fourth embodiment of the present invention.
[0021] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Figure 1 is a flow chart of a method for training an obstacle detection model according to an embodiment of the present invention;
[0023] Figure 2 is a flow chart of expanding a driving image data set according to a specific embodiment of the present invention;
[0024] Figure 3 is a flow chart of expanding a driving image data set according to another specific embodiment of the present invention;
[0025] Figure 4 is a flow chart of obtaining a target obstacle detection model according to an embodiment of the present invention;
[0026] Figure 5 is a flow chart of an obstacle detection method according to an embodiment of the present invention;
[0027] Figure 6 is a structural block diagram of an electronic device according to an embodiment of the present invention;
[0028] Figure 7 is a schematic diagram of a vehicle according to an embodiment of the present invention. DETAILED DESCRIPTION
[0029] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.
[0030] The following describes in detail the obstacle detection model training method, obstacle detection method and storage medium, electronic device, and vehicle of the embodiments of the present invention in combination with instructions 1-7 and specific implementation methods.
[0031] Figure 1 FIG. 1 is a flow chart of a method for training an obstacle detection model according to an embodiment of the present invention. Figure 1 As shown, the training method of the obstacle detection model may include:
[0032] S101, obtaining a driving image dataset, wherein the driving image dataset includes common driving images and specific obstacle images under different driving conditions.
[0033] Specifically, when acquiring ordinary driving images, they can be acquired from an open source driving image dataset on the Internet, or from a driving image dataset specially collected by a test vehicle. Ordinary driving images can also be acquired from other means. The embodiment of the present invention does not limit the means for acquiring ordinary driving images. Since the acquired ordinary driving images are driving images collected by front cameras of various different types of vehicles from different driving scenes, the generalization performance of the trained target obstacle detection model is improved to a certain extent, and it has a certain degree of robustness to the position of the on-board front camera.
[0034] It should be noted that when acquiring ordinary driving images, it is necessary to ensure that the acquired ordinary driving images cover different driving scenes. Acquiring ordinary driving images in different driving scenes to enrich the driving image dataset will help improve the richness of the training image dataset, and thus improve the generalization performance of the trained obstacle detection model to a certain extent.
[0035] In one embodiment of the present invention, the specific obstacle image data set includes specific obstacle images captured by a monocular camera of different specific obstacles.
[0036] Specifically, when acquiring specific obstacle images, since specific obstacles are uncommon, the amount of specific obstacle images obtained from the open source driving image dataset on the Internet is relatively small. In order to improve the recognition effect of the trained target obstacle detection model on specific (uncommon) obstacles, when acquiring specific obstacle images with the help of the existing specific obstacle dataset, some scenes can also be arranged on site to collect specific obstacle images, so as to focus on collecting specific obstacles (uncommon obstacles).
[0037] For example, if the relevant driving perception algorithm has difficulty effectively identifying some movable road posts, similar road posts can be placed on the special test road section of the park, and then test vehicles can be used to collect pictures from different directions. If the relevant driving perception algorithm has difficulty effectively identifying some fixed walls, more picture data can be collected for this fixed object from different driving perspectives under different time periods, different lighting and climate conditions.
[0038] It should be noted that the acquired specific obstacle image can be used directly after being annotated, or can be annotated and used after the specific obstacle image is expanded.
[0039] In an embodiment of the present invention, when acquiring a driving image dataset, ordinary driving images and specific obstacle images collected by vehicle-mounted front monocular cameras of different configurations and models are used to improve the generalization performance of the trained target obstacle detection model to a certain extent.
[0040] S102, performing expansion processing on common driving images and specific obstacle images in the driving image dataset to obtain a training image dataset.
[0041] In order to enrich the training image dataset for training the target obstacle detection model, image synthesis, image generation, and image data enhancement processing are performed based on the above-mentioned ordinary driving images and specific obstacle images to expand and enrich the training image dataset, so as to improve the generalization performance of the trained obstacle detection model to a certain extent.
[0042] In one embodiment of the present invention, Figure 2 As shown, the expansion processing of the common driving images and the specific obstacle images in the driving image dataset may include:
[0043] S21, extracting a region of interest from a common driving image to obtain a first image, wherein the region of interest includes a vehicle drivable area;
[0044] S22, extracting obstacles from the specific obstacle image to obtain a second image;
[0045] S23, scaling the second image, and synthesizing the scaled second image with the first image to obtain a third image;
[0046] S24, performing style transfer on the corresponding third image using the common driving image, and adding the third image after style transfer to the driving image dataset.
[0047] It should be noted that image synthesis refers to synthesizing a new image using an existing image, such as synthesizing a new image using an existing ordinary driving image and a specific obstacle image.
[0048] Specifically, the vehicle drivable area in the general driving image and the specific obstacles in the specific obstacle image are extracted respectively, and the extracted vehicle drivable area (first image) and the specific obstacle area (second image) are synthesized, and the synthesized new image (third image) is added to the training image dataset to enrich the training image dataset as a training image.
[0049] It is feasible that when selecting a region of interest (a drivable area of a vehicle, where the drivable area generally refers to a road surface area) from an ordinary driving image, the drivable area of the vehicle in the ordinary driving image can be manually annotated. Since the workload of manually annotating ordinary driving images is large and the efficiency is low, a portion of the ordinary driving images can be manually annotated, and the deep dual-resolution network (DDRNet) model can be trained using the annotated ordinary driving images. The region of interest is generated based on the pre-trained DDRNet model to improve the annotation efficiency of ordinary driving images.
[0050] It should be noted that when generating the region of interest based on the trained DDRNet model, it is necessary to verify the DDRNet model trained based on the annotated ordinary driving images, that is, to determine whether the region of interest generated by the trained DDRNet model is consistent with the vehicle's drivable area. If not, the trained DDRNet model needs to be adjusted to ensure that the region of interest generated by the adjusted DDRNet model (pre-trained DDRNet model) is consistent with the vehicle's drivable area.
[0051] It is feasible to extract obstacles from specific obstacle images by cutting out images, wherein the cutting out can be done manually. Alternatively, after a portion is manually annotated, the annotated specific obstacle images are used to train the deep learning segmentation network DeepLabv3+ model, and the pre-trained DeepLabv3+ model is used to automatically cut out the unannotated specific obstacle images.
[0052] It should be noted that when extracting obstacles based on the trained DeepLabv3+ model, it is necessary to verify the DeepLabv3+ model trained based on the annotated specific obstacle image, that is, to determine whether the obstacles extracted by the trained DeepLabv3+ model match the obstacles in the specific obstacle image. If not, it is necessary to continue to adjust the trained DeepLabv3+ model to ensure that the obstacles extracted by the adjusted DeepLabv3+ model (pre-trained DeepLabv3+ model) match the obstacles in the specific obstacle image.
[0053] After obtaining the vehicle drivable area in the ordinary driving image and the obstacles in the specific obstacle image, in order to enrich the training image dataset, the extracted obstacle area is scaled at different ratios, and the scaled obstacle images are placed at different positions in the vehicle drivable area to obtain a synthesized new image. In order to avoid the obstacle image inserted into the region of interest being too different in style from the ordinary driving image corresponding to the region of interest, resulting in the trained target obstacle detection model only learning to recognize the places with different styles but failing to correctly understand the real characteristics of the obstacles, the synthesized new image is input into the trained generative adversarial network CycleGANS model. The CycleGANS model transfers the style of the synthesized new image according to the style of the ordinary driving image corresponding to the region of interest, so that the synthesized new image is closer to the real scene, so that the trained target obstacle detection model can understand the real characteristics of the obstacles.
[0054] It should be noted that the specific method of style transfer is to use a picture as a style picture, and then use an algorithm to make the picture to be transferred have the style of the aforementioned picture. This operation can make the processed synthetic data closer to the real scene.
[0055] It should be noted that the embodiment of the present invention can perform region of interest processing on all or part of the ordinary driving images in the driving image data set. The embodiment of the present invention can perform obstacle extraction processing on all or part of the specific obstacle images in the driving image data set. Among them, when the extracted obstacle area is synthesized with the region of interest in the ordinary driving image, the extracted obstacle can be scaled to the region of interest in one or more ordinary driving images. The scaling ratio of the obstacle can be determined according to the number of ordinary driving images corresponding to the region of interest to be placed, so as to make it closer to the real scene.
[0056] In an embodiment of the present invention, the new image (third image) obtained after the above-mentioned synthesis processing is added to the driving image dataset to perform image generation and / or image data enhancement processing on the driving image dataset. At the same time, the new image obtained after the synthesis processing can be subjected to image generation and / or image data enhancement processing to expand and enrich the training image dataset.
[0057] In one embodiment of the present invention, Figure 3 As shown, the common driving images and specific obstacle images in the driving image dataset are expanded and processed, including:
[0058] S31, inputting an image in a new driving image dataset into a pre-trained generative model to obtain a generated new image, wherein the pre-trained generative model includes a trained WassersteinGAN model and a trained variational autoencoder VAE model;
[0059] S32, cleaning the generated new images based on preset image standards, and adding images that meet the preset image standards into a new driving image data set.
[0060] It should be noted that image generation refers to generating new images using algorithms from existing images, such as using existing ordinary driving images and specific obstacle images to generate new ordinary driving images and specific obstacle images.
[0061] In an embodiment of the present invention, a new image can be generated based on a pre-trained generative adversarial network (GAN) and a pre-trained variational autoencoder (VAE). The variational autoencoder is a structure composed of an encoder and a decoder, which is trained to minimize the reconstruction error between the encoded and decoded data and the initial data. It is feasible to construct a WassersteinGAN model, and train the constructed WassersteinGAN model based on some images in the driving image dataset, so that new image data can be generated using the trained WassersteinGAN network. It is feasible to construct a VAE model, and train the constructed VAE model based on images in the driving image dataset, so that the trained VAE model can generate new image data.
[0062] In an embodiment of the present invention, a pre-trained generative model is used to generate a new image based on the images in the driving image data set and the new image (third image) obtained by the above-mentioned synthesis process. Specifically, the images in the driving image data set and the new image (third image) obtained by the above-mentioned synthesis process are input into the pre-trained generative model, and the pre-trained generative model performs generative processing on the images in the driving image data set and the new image (third image) obtained by the above-mentioned synthesis process to obtain a new image after the generation process. Since the new image after the generation process may not meet the requirements, based on a preset image standard, such as taking an image close to a real scene as the preset image standard, the new image generated by the generation model is cleaned, and the image that meets the preset image standard is added to the training image data set as a training image.
[0063] In an embodiment of the present invention, the new image generated and processed as above is added to the driving image dataset to perform image data enhancement processing on the driving image dataset. At the same time, the new image generated and processed can be enhanced to expand and enrich the training image dataset.
[0064] In one embodiment of the present invention, the common driving images and the specific obstacle images in the driving image data set are expanded, including:
[0065] The images in the new driving image data set are enhanced, and the enhancement processing method includes at least one of color jittering, adding Gaussian noise, left-right flipping, Sobel operator filtering, mixup, and image splicing.
[0066] Image data enhancement processing is performed on the driving image dataset of the new images obtained after the newly added synthesis processing and / or the new images after the generation processing. Specifically, the following image data enhancement methods can be used to enrich the training image dataset. Exemplarily, color jitter is used, such as adjusting the saturation, brightness, and contrast of the images in the driving image dataset to enrich the training image dataset. Gaussian noise is added to the images in the driving image dataset, such as a type of noise whose probability density function obeys a normal distribution to enrich the training image dataset. The images in the driving image dataset are flipped left and right. The images in the driving image dataset are processed by filtering with the sobel operator (an image processing operator often used for edge detection) to enrich the training image dataset. Mixup (mixing samples and labels in the same way to obtain new training samples) is used to enrich the training image dataset. The images in the driving image dataset are enriched by image splicing.
[0067] In one embodiment of the present invention, the image stitching method includes at least one of the following:
[0068] splicing the duplicate image of the first target image with the first target image, the first target image being determined from the driving image data set;
[0069] Rotating the copy image of the first target image and then splicing it with the first target image;
[0070] splicing the second target image with the first target image, wherein the second target image is an image of the same category as the first target image but with a different resolution selected from the driving image dataset;
[0071] After the first target image is rotated, it is spliced with the second target image.
[0072] In order to maximize the sample richness, the image stitching method may include determining a first target image from a driving image data set, copying the first target image, and stitching the copy of the first target image with the first target image in the left-right direction and the top-bottom direction. After rotating the copy of the first target image by 180 degrees clockwise, it is stitched with the first target image in the left-right direction and the top-bottom direction. A second image of the same category but different resolution as the first target image is selected from the driving image data set, and the second target image is stitched with the first target image in the left-right direction and the top-bottom direction. After rotating the second image by 180 degrees clockwise, it is stitched with the first target image in the left-right direction and the top-bottom direction.
[0073] S103, constructing an obstacle detection model, and training the obstacle detection model using training images in the training image data set to obtain a target obstacle detection model.
[0074] In one embodiment of the present invention, the backbone network of the obstacle detection model may adopt the first RepVGG model, and the obstacle detection model is sequentially provided with a channel attention mechanism, a spatial attention mechanism and an FcaNet attention mechanism before the global average pooling layer of the RepVGG model.
[0075] In an embodiment of the present invention, the first RepVGG model, such as the RepVGG_B3g4 model, can be selected as the backbone network of the obstacle detection model. In order to improve the feature extraction capability of the obstacle detection model for valuable areas, multiple attention mechanisms such as a channel attention mechanism, a spatial attention mechanism, and an FcaNet can be added to the first RepVGG model. For example, before the global average pooling layer of the first RepVGG model, the channel attention mechanism, the spatial attention mechanism, and the FcaNet attention mechanism are added in order.
[0076] In one embodiment of the present invention, Figure 4 As shown, the obstacle detection model is trained using the training images in the training image dataset to obtain a target obstacle detection model, including:
[0077] S41, performing unsupervised training on the first RepVGG model using the training images in the training image data set to obtain a trained first RepVGG model;
[0078] S42, replacing the backbone network of the YOLOv5 model with the trained first RepVGG model;
[0079] S43, annotating the training images in the training image data set, and using the annotated training images to perform supervised training on the replaced YOLOv5 model to obtain a trained YOLOv5 model, which is recorded as a teacher network;
[0080] S44, construct a student network, and use the teacher network to perform knowledge distillation on the student network to obtain a target obstacle detection model. The student network adopts the second RepVGG model.
[0081] Specifically, in the process of training the first RepVGG model, the idea of contrastive learning can be adopted, and the SimCLv2 method in unsupervised learning can be used to improve the feature extraction ability of the first RepVGG model. A detection head is added to the unsupervised trained first RepVGG model, so that the trained target obstacle detection model can detect traffic signs, lane lines, vehicles, etc. in addition to obstacles. Specifically, the trained first RepVGG model replaces the backbone network of the YOLOv5 model, wherein the YOLOv5 model is a backbone network with added target detection heads (such as traffic signs, lane lines, and vehicles), and the YOLOv5 model is supervisedly trained using labeled training images. It should be noted that the training image data in the training image data set contains labeled target annotation information such as obstacles, traffic signs, and vehicles that need to be detected. Thereby, the trained YOLOv5 model can detect obstacles, traffic signs, vehicles, etc. in the image.
[0082] In order to deploy the trained YOLOv5 model to the vehicle hardware, a lightweight deep learning network, namely the student network, can be improved and designed based on the second RepVGG model, such as the RepVGG_B1g2 model. The trained YOLOv5 model is used as the teacher network to perform knowledge distillation on the student network, that is, the student network is accelerated by model pruning, weight quantization, and tensor decomposition, so that the trained student network is as close to the teacher network as possible in recognition ability. The obtained student network is deployed in the vehicle hardware for actual image recognition. It should be noted that the trained student network can not only identify obstacles, but also traffic signs, traffic signs, vehicles, etc.
[0083] In the design of the obstacle detection model, the embodiments of the present invention take into account the shortcomings of the current deep learning model, such as large amount of calculation and high requirements for vehicle-mounted hardware, and also consider that adding an attention mechanism can make the model focus on more meaningful areas. Therefore, on the one hand, multiple attention mechanisms are added to improve the recognition ability of the obstacle detection model, and on the other hand, the knowledge distillation method is used to ensure the recognition accuracy of the target obstacle detection model as much as possible under the premise of greatly compressing the number of obstacle detection model parameters, so as to obtain a lightweight obstacle detection model that can be deployed on the vehicle, so that the target obstacle detection model obtained by training can be used in the ADAS assisted driving system in actual application, so as to achieve effective recognition of specific (uncommon) obstacles.
[0084] The obstacle detection model training method of the embodiment of the present invention obtains ordinary driving images with the help of existing open source driving image datasets, and uses a variety of vehicle-mounted front monocular cameras of different configurations and models to collect targeted specific obstacle images to enrich the driving image dataset. In order to enrich the training image dataset and improve the generalization performance of the trained obstacle detection model, the ordinary driving images and specific obstacle images in the driving image dataset are expanded by image synthesis, image generation, image data enhancement, etc., and a variety of methods for amplifying training image data are used to improve the training data end, improve the richness of the training data end samples, and enable the obstacle detection model to be trained more effectively.
[0085] The obstacle detection model constructed by the embodiment of the present invention adopts CNN (Convolutional Neural Network) as the backbone network, and uses contrastive learning and other methods to perform unsupervised training on the constructed obstacle detection model. Unsupervised training can help the obstacle detection model to extract features more effectively, and to a certain extent solve the problem that the obstacle detection model is difficult to extract features effectively due to the small amount of specific obstacle data, and improve the recognition accuracy and recall rate of the obstacle detection model for uncommon obstacles. On the basis of the YOLOv5 model, its backbone network is changed, and multiple detection heads are added, so that it can not only identify obstacles in front of the vehicle, but also be applicable to multiple tasks including lane line detection and traffic sign detection. The YOLOv5 model obtained by supervised training is simplified using the knowledge distillation method to obtain a model for obstacle detection that can be deployed on the vehicle, so as to achieve more effective recognition of specific (uncommon) obstacles in the driving process based on a monocular camera.
[0086] The obstacle detection model training method of the embodiment of the present invention performs expansion processing on the common driving images and specific obstacle images collected by a monocular camera in the driving image dataset to enrich the training image dataset, so that the obstacle detection model can be trained more effectively, and the generalization ability of the obstacle detection model obtained by training is improved, so as to realize the effective recognition of specific obstacles in the driving images collected by the monocular camera by the target obstacle detection model.
[0087] The invention provides an obstacle detection method.
[0088] Figure 6 1 is a block diagram of an electronic device according to an embodiment of the present invention. Figure 6 As shown, the obstacle detection method may include:
[0089] S201, obtaining an image to be detected;
[0090] Specifically, a vehicle-mounted front monocular camera may be used to collect driving images as images to be detected.
[0091] S202, inputting the image to be detected into a target obstacle detection model to obtain an obstacle detection result of the image to be detected, wherein the target obstacle detection model is obtained by using the training method of the obstacle detection model as described above.
[0092] Specifically, the image to be detected is input into the target obstacle detection model, which identifies obstacles, lane lines, traffic signs, vehicles, etc. in the image to be detected and outputs the detection result. The detection result can be displayed on the vehicle terminal, such as the screen of the central control screen, in the form of a target detection block diagram.
[0093] The obstacle detection method of the embodiment of the present invention uses the trained target obstacle detection model to detect the driving image during driving, and the detection result can be applied to the ADAS system to achieve effective recognition of specific (uncommon) obstacles.
[0094] The present invention provides a computer-readable storage medium.
[0095] In this embodiment, a computer program is stored on a computer-readable storage medium. When the computer program is executed by a processor, the training method of the obstacle detection model described above is implemented, or the obstacle detection method described above is implemented.
[0096] The invention provides an electronic device.
[0097] In this embodiment, the electronic device includes a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, the obstacle detection method as described above is implemented.
[0098] Figure 6 1 is a block diagram of an electronic device according to an embodiment of the present invention. Figure 6 As shown, the electronic device 500 includes: a processor 501 and a memory 503. The processor 501 and the memory 503 are connected, such as through a bus 502. Optionally, the electronic device 500 may also include a transceiver 504. It should be noted that in actual applications, the transceiver 504 is not limited to one, and the structure of the electronic device 500 does not constitute a limitation on the embodiments of the present invention.
[0099] Processor 501 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of the present invention. Processor 501 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.
[0100] The bus 502 may include a path to transmit information between the above components. The bus 502 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 502 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0101] The memory 503 is used to store a computer program corresponding to the obstacle detection method of the above embodiment of the present invention, and the computer program is controlled and executed by the processor 501. The processor 501 is used to execute the computer program stored in the memory 503 to implement the content shown in the above method embodiment. Figure 6 The electronic device 500 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.
[0102] The present invention provides a vehicle.
[0103] Figure 7 is a schematic diagram of a vehicle according to an embodiment of the present invention. Figure 7 As shown, the vehicle 1000 may include a monocular camera 600 and an electronic device 500 as described above.
[0104] Specifically, the monocular camera 600 at the front end of the vehicle and the above-mentioned electronic device 500 deployed in the vehicle are used to detect the driving images collected by the monocular camera 600 during driving. The detection results output by the electronic device 500 can be applied to the ADAS system to achieve effective recognition of specific (uncommon) obstacles and improve driving safety.
[0105] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.
[0106] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0107] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.
[0108] In the description of the present invention, it is to be understood that the terms “center”, “longitudinal”, “lateral”, “length”, “width”, “thickness”, “up”, “down”, “front”, “back”, “left”, “right”, “vertical”, “horizontal”, “top”, “bottom”, “inside”, “outside”, “clockwise”, “counterclockwise”, “axial”, “radial”, “circumferential”, etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.
[0109] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.
[0110] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.
[0111] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may mean that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, a first feature being "above", "above" or "above" a second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the first feature is higher in level than the second feature. A first feature being "below", "below" or "below" a second feature may mean that the first feature is directly below or obliquely below the second feature, or simply means that the first feature is lower in level than the second feature.
[0112] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.
Claims
1. A training method for an obstacle detection model, characterized in that: The training method comprises: Acquire a driving image dataset, wherein the driving image dataset includes common driving images and specific obstacle images under different driving conditions; Performing expansion processing on the common driving images and the specific obstacle images in the driving image dataset to obtain a training image dataset; An obstacle detection model is constructed, and the obstacle detection model is trained using the training images in the training image data set to obtain a target obstacle detection model.
2. The obstacle detection model training method according to claim 1, characterized in that: The specific obstacle image data set includes specific obstacle images captured by a monocular camera of different specific obstacles.
3. The obstacle detection model training method according to claim 1, characterized in that: The common driving images and the specific obstacle images in the driving image data set are expanded, including: Extracting a region of interest from the ordinary driving image to obtain a first image, wherein the region of interest includes a vehicle drivable area; Extracting obstacles from the specific obstacle image to obtain a second image; Scaling the second image, and synthesizing the scaled second image with the first image to obtain a third image; The style of the corresponding third image is transferred using the common driving image, and the third image after the style transfer is added to the driving image dataset.
4. The obstacle detection model training method according to claim 3, characterized in that: The common driving images and the specific obstacle images in the driving image data set are expanded, including: Input the images in the new driving image dataset into a pre-trained generative model to obtain a generated new image, wherein the pre-trained generative model includes a trained WassersteinGAN model and a trained variational autoencoder VAE model; The generated new images are cleaned based on preset image standards, and images that meet the preset image standards are added to the new driving image data set.
5. The method for training an obstacle detection model according to claim 3 or 4, characterized in that: The common driving images and the specific obstacle images in the driving image data set are expanded, including: The images in the new driving image data set are enhanced, and the enhancement processing method includes at least one of color jittering, adding Gaussian noise, left-right flipping, Sobel operator filtering, mixup, and image splicing.
6. The method for training an obstacle detection model according to claim 5, characterized in that: The image stitching method includes at least one of the following: splicing a copy of a first target image with the first target image, the first target image being determined from the driving image dataset; Rotating the copy image of the first target image and then splicing it with the first target image; splicing a second target image with the first target image, wherein the second target image is an image of the same category as the first target image but with a different resolution selected from the driving image dataset; The first target image is rotated and then spliced with the second target image.
7. The obstacle detection model training method according to claim 1, characterized in that: The backbone network of the obstacle detection model adopts the first RepVGG model. The obstacle detection model is sequentially provided with a channel attention mechanism, a spatial attention mechanism and an FcaNet attention mechanism before the global average pooling layer of the RepVGG model. The obstacle detection model is trained using the training images in the training image data set to obtain a target obstacle detection model, including: Performing unsupervised training on the first RepVGG model using the training images in the training image dataset to obtain a trained first RepVGG model; Replace the backbone network of the YOLOv5 model with the trained first RepVGG model; Annotate the training images in the training image data set, and use the annotated training images to perform supervised training on the replaced YOLOv5 model to obtain a trained YOLOv5 model, which is recorded as a teacher network; A student network is constructed, and knowledge distillation is performed on the student network using the teacher network to obtain the target obstacle detection model, wherein the student network adopts the second RepVGG model.
8. An obstacle detection method, characterized in that: The method comprises: Acquire the image to be detected; The image to be detected is input into a target obstacle detection model to obtain an obstacle detection result of the image to be detected, wherein the target obstacle detection model is obtained by using the obstacle detection model training method described in any one of claims 1 to 7.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the training method of the obstacle detection model according to any one of claims 1 to 7 is implemented, or the obstacle detection method according to claim 8 is implemented.
10. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the computer program is executed by the processor, the obstacle detection method according to claim 8 is implemented.
11. A vehicle, characterized in that: The invention comprises a monocular camera and an electronic device as claimed in claim 10.
Citation Information
Patent Citations
Zero-sample image classification method based on semantic expansion
CN112380374A
Obstacle recognition method, obstacle recognition device and electronic equipment
CN113128386A
Driving area detection method, device and equipment and storage medium
CN114663397A
Target recognition model training method and device, electronic equipment and storage medium
CN115471810A
Improved yolov5s network model-based cylindrical roller surface detection method and system
CN115937651A
Cited By
Computing device and method for training artificial intelligence model
TWI929854B
