Automatic parking obstacle detection method and device

Abstract

The invention discloses an automatic parking obstacle detection method and device. A semantic segmentation model of each scene is trained by collecting multi-path look-around fisheye camera image sets of different scenes, so that a semantic segmentation model library is generated, and when parking is performed, a target semantic segmentation model is matched in the semantic segmentation model library based on a driving scene, so that the adaptability and accuracy of the semantic segmentation model in different scenes are improved, and the parking experience is improved. And the four fisheye images are subjected to semantic segmentation based on the four fisheye images and then spliced into a bird's-eye view angle, so that the problem that the bird's-eye view angle easily causes loss or distortion of obstacle information is avoided, an accurate travelable region boundary is provided, and the obstacle detection accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an automatic parking obstacle detection method and device. Background Art

[0002] Surround vision is a key technology for achieving safe navigation and obstacle avoidance in intelligent systems such as autonomous driving, automatic parking, and drones. Accurately detecting obstacles in the surrounding environment is crucial to ensuring the safety and reliability of the system.

[0003] In the current automatic parking system, obstacle detection is performed based on a bird's-eye view. From a bird's-eye view, since the on-board surround view camera is installed at a low position, when the surround view camera image is converted to a bird's-eye view, it is easy for obstacle information to be blocked and obscured, resulting in the loss or distortion of obstacle information and a decrease in detection accuracy. In addition, from a bird's-eye view, a large amount of target category information will be lost, resulting in the inability to distinguish obstacle categories, which is not conducive to the downstream decision-making and planning module of the obstacle perception module in the automatic parking system to identify dynamic obstacles. Summary of the invention

[0004] The present invention provides an automatic parking obstacle detection method and device, so as to reduce the target category information loss rate and improve the obstacle detection accuracy under the bird's eye view.

[0005] In order to solve the above technical problems, an embodiment of the present invention provides an automatic parking obstacle detection method, comprising:

[0006] Collecting a multi-channel surround fisheye camera image set of each scene, annotating the multi-channel surround fisheye camera image set based on obstacle types and scene requirements, and generating training data for each scene;

[0007] Based on the training data of each scene, the initial semantic segmentation model is trained respectively to obtain a semantic segmentation model library;

[0008] Matching a target semantic segmentation model based on the driving scene, and processing the real-time fisheye image based on the target semantic segmentation model to obtain a plurality of segmented images;

[0009] The segmented image is converted to generate a bird's-eye view semantic grayscale image, and the bird's-eye view semantic grayscale image is superimposed and spliced ​​based on a preset splicing rule to generate a panoramic bird's-eye view semantic grayscale image, and obstacles in the driving scene are identified based on the panoramic bird's-eye view semantic grayscale image.

[0010] The present invention collects a multi-channel surround fisheye camera image set of different scenes to train the semantic segmentation model of each scene, thereby generating a semantic segmentation model library. When parking, the target semantic segmentation model is matched in the semantic segmentation model library based on the driving scene, so as to improve the adaptability and accuracy of the semantic segmentation model in different scenes, thereby improving the accuracy of obstacle detection. The semantic segmentation is performed based on four fisheye images and then spliced ​​into a bird's-eye view, so as to avoid the problem that the bird's-eye view easily causes the loss or distortion of obstacle information, provide an accurate drivable area boundary, and improve the accuracy of obstacle detection.

[0011] Furthermore, the multi-channel surround fisheye camera image set of each scene is collected, and the multi-channel surround fisheye camera image set is annotated based on obstacle types and scene requirements to generate training data for each scene, including:

[0012] Collecting multiple surround-view fisheye camera images based on parking scenes to generate a multiple surround-view fisheye camera image set; the parking scenes include underground marked parking lots, ground marked parking lots, and unmarked parking areas;

[0013] Segmenting and labeling obstacles in the multi-path surround fisheye camera image based on obstacle types;

[0014] Based on the scene requirements, the obstacles in each multi-channel surround fisheye camera image are numbered to generate training data for each parking scene.

[0015] Furthermore, the target semantic segmentation model is matched based on the driving scene, and the real-time fisheye image is processed based on the target semantic segmentation model to obtain a plurality of segmented images, including:

[0016] Obtain image data from the vehicle's four-way fisheye camera, and pair the image data at the same time based on timestamp information to obtain real-time fisheye images;

[0017] A target semantic segmentation model is matched in the semantic segmentation model library based on the driving scene, and the real-time fisheye image is segmented based on the target semantic segmentation model to obtain a plurality of segmented images.

[0018] Furthermore, the converting of the segmented image to generate a bird's-eye view semantic grayscale image, and superimposing and splicing the bird's-eye view semantic grayscale image based on a preset splicing rule to generate a panoramic bird's-eye view semantic grayscale image, includes:

[0019] Constructing a grayscale mapping table based on the obstacle numbers, and performing grayscale processing on the segmented image based on the grayscale mapping table to generate four-way semantic grayscale images;

[0020] Acquire the internal reference data and external reference data of the four fisheye cameras, and convert the four semantic grayscale images into bird's-eye view semantic grayscale images based on the internal reference data and the external reference data;

[0021] The semantic grayscale images of the bird's-eye view at the same moment are superimposed and spliced ​​in the order of front view, rear view, left view and right view to generate a panoramic bird's-eye view semantic grayscale image; and for the splicing overlapping area of ​​the semantic grayscale images of the bird's-eye view, the image pixel values ​​of the upper layer are used to replace the image pixel values ​​of the lower layer.

[0022] Furthermore, the identifying obstacles in the driving scene based on the panoramic bird's-eye view semantic grayscale image includes:

[0023] Acquire a conversion relationship between pixel coordinates and physical coordinates, and convert the panoramic bird's-eye view semantic grayscale image into a raster map point cloud based on the conversion relationship;

[0024] An automatic parking control route is planned based on obstacle identifications in the grid map point cloud.

[0025] In a second aspect, the present invention provides an automatic parking obstacle detection device, comprising: an image acquisition module, a model training module, a target segmentation module and a recognition control module;

[0026] The image acquisition module is used to collect a multi-channel surround fisheye camera image set of each scene, annotate the multi-channel surround fisheye camera image set based on obstacle types and scene requirements, and generate training data for each scene;

[0027] The model training module is used to train the initial semantic segmentation model based on the training data of each scene to obtain a semantic segmentation model library;

[0028] The target segmentation module is used to match the target semantic segmentation model based on the driving scene, and process the real-time fisheye image based on the target semantic segmentation model to obtain a plurality of segmented images;

[0029] The recognition control module is used to convert the segmented image to generate a bird's-eye view semantic grayscale image, and to overlay and splice the bird's-eye view semantic grayscale image based on a preset splicing rule to generate a panoramic bird's-eye view semantic grayscale image, and to identify obstacles in the driving scene based on the panoramic bird's-eye view semantic grayscale image.

[0030] Furthermore, the image acquisition module is used to:

[0031] Collecting multiple surround-view fisheye camera images based on parking scenes to generate a multiple surround-view fisheye camera image set; the parking scenes include underground marked parking lots, ground marked parking lots, and unmarked parking areas;

[0032] Segmenting and labeling obstacles in the multi-path surround fisheye camera image based on obstacle types;

[0033] Based on the scene requirements, the obstacles in each multi-channel surround fisheye camera image are numbered to generate training data for each parking scene.

[0034] Furthermore, the target segmentation module is used to:

[0035] Obtain image data from the vehicle's four-way fisheye camera, and pair the image data at the same time based on timestamp information to obtain real-time fisheye images;

[0036] A target semantic segmentation model is matched in the semantic segmentation model library based on the driving scene, and the real-time fisheye image is segmented based on the target semantic segmentation model to obtain a plurality of segmented images.

[0037] Furthermore, the identification control module is used to:

[0038] Constructing a grayscale mapping table based on the obstacle numbers, and performing grayscale processing on the segmented image based on the grayscale mapping table to generate four-way semantic grayscale images;

[0039] Acquire the internal reference data and external reference data of the four fisheye cameras, and convert the four semantic grayscale images into bird's-eye view semantic grayscale images based on the internal reference data and the external reference data;

[0040] The semantic grayscale images of the bird's-eye view at the same moment are superimposed and spliced ​​in the order of front view, rear view, left view and right view to generate a panoramic bird's-eye view semantic grayscale image; and for the splicing overlapping area of ​​the semantic grayscale images of the bird's-eye view, the image pixel values ​​of the upper layer are used to replace the image pixel values ​​of the lower layer.

[0041] Furthermore, the identification control module is used to:

[0042] Acquire a conversion relationship between pixel coordinates and physical coordinates, and convert the panoramic bird's-eye view semantic grayscale image into a raster map point cloud based on the conversion relationship;

[0043] An automatic parking control route is planned based on obstacle identifications in the grid map point cloud. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A schematic diagram of a flow chart of an automatic parking obstacle detection method provided by an embodiment of the present invention;

[0045] Figure 2 A real-time fisheye image of a four-way surround-view fisheye camera provided by an embodiment of the present invention;

[0046] Figure 3 A real-time fisheye image of a four-way surround-view fisheye camera provided by an embodiment of the present invention;

[0047] Figure 4 A semantic grayscale image from a bird's-eye view provided by an embodiment of the present invention;

[0048] Figure 5 A schematic diagram of a two-dimensional grid map provided by an embodiment of the present invention;

[0049] Figure 6 A schematic diagram of a process of obstacle detection during parking provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0051] The terms "first" and "second" and the like in the specification, claims and drawings of this application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices.

[0052] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0053] Example 1

[0054] See also Figure 1 , Figure 1 The present invention provides a flow chart of an automatic parking obstacle detection method according to an embodiment of the present invention. The present invention provides an automatic parking obstacle detection method, including steps 101 to 104, which are as follows:

[0055] Step 101: collecting a multi-channel surround fisheye camera image set of each scene, annotating the multi-channel surround fisheye camera image set based on obstacle types and scene requirements, and generating training data for each scene;

[0056] In this embodiment, the multi-channel surround fisheye camera image set of each scene is collected, and the multi-channel surround fisheye camera image set is annotated based on obstacle types and scene requirements to generate training data for each scene, including:

[0057] Collecting multiple surround-view fisheye camera images based on parking scenes to generate a multiple surround-view fisheye camera image set; the parking scenes include underground marked parking lots, ground marked parking lots, and unmarked parking areas;

[0058] Segmenting and labeling obstacles in the multi-path surround fisheye camera image based on obstacle types;

[0059] Based on the scene requirements, the obstacles in each multi-channel surround fisheye camera image are numbered to generate training data for each parking scene.

[0060] In this embodiment, image data of the four-way fisheye camera of the vehicle for developing the automatic parking function in various scenarios are collected to generate a multi-way surround fisheye camera image set, wherein the scenarios include common scenarios of users and the use of the automatic parking function, such as: underground marked parking lots, ground marked parking lots, and unmarked parking areas.

[0061] In this embodiment, obstacles are marked for each image in the multi-channel surround fisheye camera image set. The images are first segmented and marked based on the obstacle type, where the obstacle types include the vehicle itself, static obstacles and dynamic obstacles. Static obstacles include curbs, green plants and charging piles, and dynamic obstacles include cars, bicycles and pedestrians.

[0062] In this embodiment, static obstacles and dynamic obstacles in each multi-channel surround fisheye camera image are numbered based on scene requirements to generate training data for each scene. For example, a car is obstacle 1, a pedestrian is obstacle 2, and a curb is obstacle 3.

[0063] In this embodiment, each pixel in each image is labeled and numbered based on a labeling tool, and the labeling tools include: Label box, VGG Image Annotator (VIA), etc.

[0064] In this embodiment, multi-channel surround fisheye camera images are collected based on different weather conditions and light conditions to ensure that the multi-channel surround fisheye camera image set contains images under different lighting conditions and different weather conditions, such as daytime images, night images, rainy day images, snowy day images, sunny day images, foggy day images, etc.

[0065] Step 102: training the initial semantic segmentation model based on the training data of each scene to obtain a semantic segmentation model library;

[0066] In this embodiment, after obtaining the training data for each scene, it also includes increasing the diversity of the training data in each scene by rotating, scaling, flipping, and color space transformation operations on the images in the training data, preventing model overfitting, and improving the robustness of the model.

[0067] In this embodiment, for the training data of each scenario, the data set is divided into a training set, a validation set, and a test set based on a preset ratio.

[0068] In this embodiment, an initial semantic segmentation model is constructed based on deep learning, and the initial semantic segmentation model is trained based on the training data of each scene to obtain a trained semantic segmentation model corresponding to each scene, and the weight parameters of the corresponding semantic segmentation model are recorded to generate a semantic segmentation model library.

[0069] In this embodiment, an initial semantic segmentation model is constructed based on the DeepLab framework, the model parameters are initialized, and the initial semantic segmentation model is trained based on the training data of each scene. The semantic segmentation model is evaluated based on indicators such as IoU (intersection over union), precision, and recall until the preset conditions are met. The trained semantic segmentation model is output, the semantic segmentation model corresponding to each scene is obtained, and the corresponding weight parameters are recorded to generate a semantic segmentation model library.

[0070] Step 103: matching a target semantic segmentation model based on the driving scene, and processing the real-time fisheye image based on the target semantic segmentation model to obtain a plurality of segmented images;

[0071] In this embodiment, the target semantic segmentation model is matched based on the driving scene, and the real-time fisheye image is processed based on the target semantic segmentation model to obtain a plurality of segmented images, including:

[0072] Obtain image data from the vehicle's four-way fisheye camera, and pair the image data at the same time based on timestamp information to obtain real-time fisheye images;

[0073] A target semantic segmentation model is matched in the semantic segmentation model library based on the driving scene, and the real-time fisheye image is segmented based on the target semantic segmentation model to obtain a plurality of segmented images.

[0074] Please refer to Figure 2 , Figure 2 A real-time fisheye image of a four-way surround-view fisheye camera provided by an embodiment of the present invention.

[0075] In this embodiment, a real-time fisheye image of a vehicle and a vehicle driving scene are obtained, a parking scene is matched in a semantic segmentation model library based on the driving scene, and a corresponding target semantic segmentation model is obtained.

[0076] In this embodiment, the fisheye camera installed on the vehicle is used to obtain image information of the surrounding environment in real time, and the most appropriate semantic segmentation model is selected from the semantic segmentation model library according to the current driving scene, so as to ensure the best segmentation effect of the real-time fisheye image in a specific scene and improve the accuracy and robustness of obstacle detection.

[0077] In this embodiment, the real-time fisheye image is processed based on the target semantic segmentation model to obtain a number of segmented images, and the semantic information of each target in the image, such as vehicles, pedestrians, roads, parking spaces, etc., is obtained.

[0078] Step 104: Overlay and splice the segmented images to generate a panoramic bird's-eye view semantic grayscale image, and identify obstacles in the driving scene based on the panoramic bird's-eye view semantic grayscale image.

[0079] Please refer to Figure 3 , Figure 3 A real-time fisheye image of a four-way surround-view fisheye camera provided by an embodiment of the present invention.

[0080] In this embodiment, the segmented image is converted to generate a bird's-eye view semantic grayscale image, and the bird's-eye view semantic grayscale image is superimposed and spliced ​​based on a preset splicing rule to generate a panoramic bird's-eye view semantic grayscale image, including:

[0081] Constructing a grayscale mapping table based on the obstacle numbers, and performing grayscale processing on the segmented image based on the grayscale mapping table to generate four-way semantic grayscale images;

[0082] Acquire the internal reference data and external reference data of the four fisheye cameras, and convert the four semantic grayscale images into bird's-eye view semantic grayscale images based on the internal reference data and the external reference data;

[0083] The semantic grayscale images of the bird's-eye view at the same moment are superimposed and spliced ​​in the order of front view, rear view, left view and right view to generate a panoramic bird's-eye view semantic grayscale image; and for the splicing overlapping area of ​​the semantic grayscale images of the bird's-eye view, the image pixel values ​​of the upper layer are used to replace the image pixel values ​​of the lower layer.

[0084] In this embodiment, a grayscale mapping table is constructed based on the obstacle number, and each obstacle number is assigned a grayscale value between 0 and 255, where 0 represents black and 255 represents white. Thus, the grayscale value is adjusted according to the importance of the obstacle type or the need for visual recognition to generate a grayscale mapping table.

[0085] As a specific example of an embodiment of the present invention, pixel value 0 represents the vehicle, pixel value 1 represents obstacle 1, pixel value 2 represents obstacle 2, and pixel value 3 represents obstacle 3.

[0086] In this embodiment, after acquiring four semantic grayscale images, the internal and external parameter data of the camera are read from the vehicle-side memory, and the four semantic grayscale images are converted into a bird's-eye view respectively.

[0087] Please refer to Figure 4 , Figure 4 A semantic grayscale image from a bird's-eye view is provided in an embodiment of the present invention.

[0088] In this embodiment, the timestamp information of the real-time fisheye image is obtained, and based on the timestamp information and the preset stitching rules, the semantic grayscale images of the bird's-eye view at the same moment are stitched to generate a panoramic bird's-eye view semantic grayscale map, wherein the stitching rules include superimposing and stitching the semantic grayscale images of the bird's-eye view at the same moment, and when the overlapping areas are stitched, the pixel values ​​of the image in the lower layer will be replaced by the pixel values ​​of the image in the upper layer, and the stitching order is front view, rear view, left view and right view, and finally a panoramic bird's-eye view semantic grayscale map is generated.

[0089] In this embodiment, when the semantic grayscale images from the bird's-eye view are superimposed and spliced ​​based on the front view, rear view, left view and right view, the image pixel values ​​of the upper layer are used to replace the image pixel values ​​of the lower layer in the overlapping area, thereby avoiding global traversal and fusion of the image overlap areas, thereby improving the splicing efficiency.

[0090] In this embodiment, the step of identifying obstacles in the driving scene based on the panoramic bird's-eye view semantic grayscale image includes:

[0091] Acquire a conversion relationship between pixel coordinates and physical coordinates, and convert the panoramic bird's-eye view semantic grayscale image into a raster map point cloud based on the conversion relationship;

[0092] An automatic parking control route is planned based on obstacle identifications in the grid map point cloud.

[0093] Please refer to Figure 5 , Figure 5 A schematic diagram of a two-dimensional grid map provided by an embodiment of the present invention.

[0094] In this embodiment, after generating the panoramic bird's-eye view semantic grayscale image, the panoramic bird's-eye view semantic grayscale image is converted into a two-dimensional grid map point cloud based on the conversion relationship between pixel coordinates and physical coordinates, and output to the planning control module, so that the planning control module plans the automatic parking control route according to the obstacle identification in the grid map point cloud to complete the automatic parking.

[0095] Please refer to Figure 6 , Figure 6 A schematic diagram of a process of obstacle detection during parking provided by an embodiment of the present invention.

[0096] In this embodiment, image data from the vehicle's four fisheye cameras are acquired in real time during the parking process, the images collected at the same time are paired using timestamp information, and the parameter weights of the corresponding semantic segmentation model are matched based on the driving scene, so that the paired images are input into the semantic segmentation model respectively.

[0097] In this embodiment, the semantic segmentation results of the four fisheye camera images are processed and converted into semantic grayscale images, and the internal and external parameter data of the camera are read from the vehicle-side memory to convert the four semantic grayscale images into a bird's-eye view.

[0098] In this embodiment, the four semantic grayscale images of the bird's-eye view at the same time are superimposed and spliced. When the overlapping areas are spliced, the image pixel values ​​in the lower layer will be replaced by the image pixel values ​​in the upper layer. The splicing order is front view, rear view, left view and right view, and finally a panoramic bird's-eye view semantic grayscale image is generated. Finally, based on the conversion relationship between pixel coordinates and physical coordinates, the panoramic bird's-eye view semantic grayscale image is converted into a two-dimensional raster map point cloud, and output to the planning control module, so that the planning control module performs automatic parking according to the two-dimensional raster map point cloud.

[0099] The present invention collects a multi-channel surround fisheye camera image set of different scenes to train the semantic segmentation model of each scene, thereby generating a semantic segmentation model library. When parking, the target semantic segmentation model is matched in the semantic segmentation model library based on the driving scene, so as to improve the adaptability and accuracy of the semantic segmentation model in different scenes, thereby improving the accuracy of obstacle detection. The semantic segmentation is performed based on four fisheye images and then spliced ​​into a bird's-eye view, so as to avoid the problem that the bird's-eye view easily causes the loss or distortion of obstacle information, provide an accurate drivable area boundary, and improve the accuracy of obstacle detection.

[0100] The embodiment of the present invention also provides an automatic parking obstacle detection device, including: an image acquisition module, a model training module, a target segmentation module and a recognition control module;

[0101] The image acquisition module is used to collect a multi-channel surround fisheye camera image set of each scene, annotate the multi-channel surround fisheye camera image set based on obstacle types and scene requirements, and generate training data for each scene;

[0102] The model training module is used to train the initial semantic segmentation model based on the training data of each scene to obtain a semantic segmentation model library;

[0103] The target segmentation module is used to match the target semantic segmentation model based on the driving scene, and process the real-time fisheye image based on the target semantic segmentation model to obtain a plurality of segmented images;

[0104] The recognition control module is used to convert the segmented image to generate a bird's-eye view semantic grayscale image, and to overlay and splice the bird's-eye view semantic grayscale image based on a preset splicing rule to generate a panoramic bird's-eye view semantic grayscale image, and to identify obstacles in the driving scene based on the panoramic bird's-eye view semantic grayscale image.

[0105] In this embodiment, the image acquisition module is used to:

[0106] Collecting multiple surround-view fisheye camera images based on parking scenes to generate a multiple surround-view fisheye camera image set; the parking scenes include underground marked parking lots, ground marked parking lots, and unmarked parking areas;

[0107] Segmenting and labeling obstacles in the multi-path surround fisheye camera image based on obstacle types;

[0108] Based on the scene requirements, the obstacles in each multi-channel surround fisheye camera image are numbered to generate training data for each parking scene.

[0109] In this embodiment, the target segmentation module is used to:

[0110] Obtain image data from the vehicle's four-way fisheye camera, and pair the image data at the same time based on timestamp information to obtain real-time fisheye images;

[0111] A target semantic segmentation model is matched in the semantic segmentation model library based on the driving scene, and the real-time fisheye image is segmented based on the target semantic segmentation model to obtain a plurality of segmented images.

[0112] In this embodiment, the identification control module is used to:

[0113] Constructing a grayscale mapping table based on the obstacle numbers, and performing grayscale processing on the segmented image based on the grayscale mapping table to generate four-way semantic grayscale images;

[0114] Acquire the internal reference data and external reference data of the four fisheye cameras, and convert the four semantic grayscale images into bird's-eye view semantic grayscale images based on the internal reference data and the external reference data;

[0115] The semantic grayscale images of the bird's-eye view at the same moment are superimposed and spliced ​​in the order of front view, rear view, left view and right view to generate a panoramic bird's-eye view semantic grayscale image; and for the splicing overlapping area of ​​the semantic grayscale images of the bird's-eye view, the image pixel values ​​of the upper layer are used to replace the image pixel values ​​of the lower layer.

[0116] In this embodiment, the identification control module is used to:

[0117] Acquire a conversion relationship between pixel coordinates and physical coordinates, and convert the panoramic bird's-eye view semantic grayscale image into a raster map point cloud based on the conversion relationship;

[0118] An automatic parking control route is planned based on obstacle identifications in the grid map point cloud.

[0119] The present invention collects a multi-channel surround fisheye camera image set of different scenes to train the semantic segmentation model of each scene, thereby generating a semantic segmentation model library. When parking, the target semantic segmentation model is matched in the semantic segmentation model library based on the driving scene, so as to improve the adaptability and accuracy of the semantic segmentation model in different scenes, thereby improving the accuracy of obstacle detection. The semantic segmentation is performed based on four fisheye images and then spliced ​​into a bird's-eye view, so as to avoid the problem that the bird's-eye view easily causes the loss or distortion of obstacle information, provide an accurate drivable area boundary, and improve the accuracy of obstacle detection.

[0120] In an embodiment of the present invention, a terminal device is also provided, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, and the processor implements the above-mentioned automatic parking obstacle detection method when executing the computer program.

[0121] In an embodiment of the present invention, a computer-readable storage medium is further provided. The computer-readable storage medium includes a stored computer program. When the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the above-mentioned automatic parking obstacle detection method.

[0122] Exemplarily, the computer program may be divided into one or more modules, one or more modules are stored in a memory and executed by a processor to implement the present invention. One or more modules may be a series of computer program instruction segments capable of implementing specific functions, and the instruction segments are used to describe the execution process of the computer program in a terminal device.

[0123] The terminal device may be a computing device such as a desktop computer, a notebook, a PDA, and a cloud server. The terminal device may include, but is not limited to, a processor, a memory, and a display. Those skilled in the art will appreciate that the above components are merely examples of the terminal device and do not constitute a limitation on the terminal device. The terminal device may include more or fewer components than the components, or may combine certain components, or different components. For example, the terminal device may also include input and output devices, network access devices, buses, etc.

[0124] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the terminal device, and uses various interfaces and lines to connect various parts of the entire terminal device.

[0125] The memory can be used to store computer programs and / or modules. The processor realizes various functions of the terminal device by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, a text conversion function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, text message data, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Med ia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0126] Among them, if the module based on the processing of the multi-device access platform is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), electric carrier signal, telecommunication signal and software distribution medium, etc. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0127] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. An automatic parking obstacle detection method, characterized in that: include: Collecting a multi-channel surround fisheye camera image set of each scene, annotating the multi-channel surround fisheye camera image set based on obstacle types and scene requirements, and generating training data for each scene; Based on the training data of each scene, the initial semantic segmentation model is trained respectively to obtain a semantic segmentation model library; Matching a target semantic segmentation model based on the driving scene, and processing the real-time fisheye image based on the target semantic segmentation model to obtain a plurality of segmented images; The segmented image is converted to generate a bird's-eye view semantic grayscale image, and the bird's-eye view semantic grayscale image is superimposed and spliced ​​based on a preset splicing rule to generate a panoramic bird's-eye view semantic grayscale image, and obstacles in the driving scene are identified based on the panoramic bird's-eye view semantic grayscale image.

2. The automatic parking obstacle detection method according to claim 1, characterized in that: The collecting of multi-channel surround fisheye camera image sets of each scene, labeling the multi-channel surround fisheye camera image sets based on obstacle types and scene requirements, and generating training data for each scene, includes: Collecting multiple surround-view fisheye camera images based on parking scenes to generate a multiple surround-view fisheye camera image set; the parking scenes include underground marked parking lots, ground marked parking lots, and unmarked parking areas; Segmenting and labeling obstacles in the multi-path surround fisheye camera image based on obstacle types; Based on the scene requirements, the obstacles in each multi-channel surround fisheye camera image are numbered to generate training data for each parking scene.

3. The automatic parking obstacle detection method according to claim 2, characterized in that: The target semantic segmentation model is matched based on the driving scene, and the real-time fisheye image is processed based on the target semantic segmentation model to obtain a plurality of segmented images, including: Obtain image data from the vehicle's four-way fisheye camera, and pair the image data at the same time based on timestamp information to obtain real-time fisheye images; A target semantic segmentation model is matched in the semantic segmentation model library based on the driving scene, and the real-time fisheye image is segmented based on the target semantic segmentation model to obtain a plurality of segmented images.

4. The automatic parking obstacle detection method according to claim 3, characterized in that: The step of converting the segmented image to generate a bird's-eye view semantic grayscale image, and superimposing and splicing the bird's-eye view semantic grayscale image based on a preset splicing rule to generate a panoramic bird's-eye view semantic grayscale image includes: Constructing a grayscale mapping table based on the obstacle numbers, and performing grayscale processing on the segmented image based on the grayscale mapping table to generate four-way semantic grayscale images; Acquire the internal reference data and external reference data of the four fisheye cameras, and convert the four semantic grayscale images into bird's-eye view semantic grayscale images based on the internal reference data and the external reference data; The semantic grayscale images of the bird's-eye view at the same moment are superimposed and spliced ​​in the order of front view, rear view, left view and right view to generate a panoramic bird's-eye view semantic grayscale image; and for the splicing overlapping area of ​​the semantic grayscale images of the bird's-eye view, the image pixel values ​​of the upper layer are used to replace the image pixel values ​​of the lower layer.

5. The automatic parking obstacle detection method according to claim 4, characterized in that: The method of identifying obstacles in a driving scene based on the panoramic bird's-eye view semantic grayscale image includes: Acquire a conversion relationship between pixel coordinates and physical coordinates, and convert the panoramic bird's-eye view semantic grayscale image into a raster map point cloud based on the conversion relationship; An automatic parking control route is planned based on obstacle identifications in the grid map point cloud.

6. An automatic parking obstacle detection device, characterized in that: include: Image acquisition module, model training module, target segmentation module and recognition control module; The image acquisition module is used to collect a multi-channel surround fisheye camera image set of each scene, annotate the multi-channel surround fisheye camera image set based on obstacle types and scene requirements, and generate training data for each scene; The model training module is used to train the initial semantic segmentation model based on the training data of each scene to obtain a semantic segmentation model library; The target segmentation module is used to match the target semantic segmentation model based on the driving scene, and process the real-time fisheye image based on the target semantic segmentation model to obtain a plurality of segmented images; The recognition control module is used to convert the segmented image to generate a bird's-eye view semantic grayscale image, and to overlay and splice the bird's-eye view semantic grayscale image based on a preset splicing rule to generate a panoramic bird's-eye view semantic grayscale image, and to identify obstacles in the driving scene based on the panoramic bird's-eye view semantic grayscale image.

7. The automatic parking obstacle detection device according to claim 6, characterized in that: The image acquisition module is used for: Collecting multiple surround-view fisheye camera images based on parking scenes to generate a multiple surround-view fisheye camera image set; the parking scenes include underground marked parking lots, ground marked parking lots, and unmarked parking areas; Segmenting and labeling obstacles in the multi-path surround fisheye camera image based on obstacle types; Based on the scene requirements, the obstacles in each multi-channel surround fisheye camera image are numbered to generate training data for each parking scene.

8. The automatic parking obstacle detection device according to claim 7, characterized in that: The target segmentation module is used for: Obtain image data from the vehicle's four-way fisheye camera, and pair the image data at the same time based on timestamp information to obtain real-time fisheye images; A target semantic segmentation model is matched in the semantic segmentation model library based on the driving scene, and the real-time fisheye image is segmented based on the target semantic segmentation model to obtain a plurality of segmented images.

9. The automatic parking obstacle detection device according to claim 8, characterized in that: The identification control module is used to: Constructing a grayscale mapping table based on the obstacle numbers, and performing grayscale processing on the segmented image based on the grayscale mapping table to generate four-way semantic grayscale images; Acquire the internal reference data and external reference data of the four fisheye cameras, and convert the four semantic grayscale images into bird's-eye view semantic grayscale images based on the internal reference data and the external reference data; The semantic grayscale images of the bird's-eye view at the same moment are superimposed and spliced ​​in the order of front view, rear view, left view and right view to generate a panoramic bird's-eye view semantic grayscale image; and for the splicing overlapping area of ​​the semantic grayscale images of the bird's-eye view, the image pixel values ​​of the upper layer are used to replace the image pixel values ​​of the lower layer.

10. The automatic parking obstacle detection device according to claim 9, characterized in that: The identification control module is used to: Acquire a conversion relationship between pixel coordinates and physical coordinates, and convert the panoramic bird's-eye view semantic grayscale image into a raster map point cloud based on the conversion relationship; An automatic parking control route is planned based on obstacle identifications in the grid map point cloud.

Images