Obstacle detection method and device

Abstract

The present disclosure provides an obstacle detection method and device, which relates to the field of artificial intelligence technology, specifically the technical fields of computer vision, image processing and deep learning, and can be applied to autonomous driving scenarios. A specific implementation of the method includes: detecting foreground obstacles in the previous frame image taken by the vehicle's camera; filtering out adjacent foreground obstacle pairs from the foreground obstacles as candidate under-segmented obstacle pairs; predicting the position of the candidate under-segmented obstacles in the current frame image based on the speed of the candidate under-segmented obstacles in the candidate under-segmented obstacle pairs; and detecting the under-segmented obstacles from the current frame image based on the position of the candidate under-segmented obstacles. This implementation can efficiently and accurately detect various types of under-segmented obstacles by judging the continuity of obstacles in previous and subsequent frames.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, specifically computer vision, image processing, deep learning and other technical fields, and can be applied in autonomous driving scenarios. Background Art

[0002] Self-driving cars, also known as driverless cars, computer-driven cars, or wheeled mobile robots, are intelligent cars that are unmanned by computer systems. Self-driving cars rely on artificial intelligence, visual computing, radar, monitoring devices, and global positioning systems to work together, allowing computers to automatically and safely operate motor vehicles without any active human operation.

[0003] In autonomous driving scenarios, autonomous vehicles need to detect obstacles in the surrounding environment. However, the perception model sometimes detects multiple obstacles as the same one, making the vehicle unsafe to drive and causing the safety officer to take over in interactive scenarios. Summary of the invention

[0004] The embodiments of the present disclosure provide an obstacle detection method, apparatus, device, storage medium, and program product.

[0005] In a first aspect, an embodiment of the present disclosure proposes an obstacle detection method, including: detecting a foreground obstacle in a previous frame image taken by a camera of a vehicle; filtering out adjacent foreground obstacle pairs from the foreground obstacles as candidate under-segmented obstacle pairs; predicting a position of the candidate under-segmented obstacle in a current frame image based on the speed of the candidate under-segmented obstacle in the candidate under-segmented obstacle pair; and detecting the under-segmented obstacle from the current frame image based on the position of the candidate under-segmented obstacle.

[0006] In a second aspect, an embodiment of the present disclosure proposes an obstacle detection device, comprising: a first detection module, configured to detect a foreground obstacle in a previous frame image taken by a camera of a vehicle; a filtering module, configured to filter out adjacent foreground obstacle pairs from the foreground obstacles as candidate under-segmented obstacle pairs; a prediction module, configured to predict the position of the candidate under-segmented obstacle in the current frame image based on the speed of the candidate under-segmented obstacle in the candidate under-segmented obstacle pair; and a second detection module, configured to detect the under-segmented obstacle from the current frame image based on the position of the candidate under-segmented obstacle.

[0007] In a third aspect, an embodiment of the present disclosure proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in the first aspect.

[0008] In a fourth aspect, an embodiment of the present disclosure proposes a non-transitory computer-readable storage medium storing computer instructions, where the computer instructions are used to cause a computer to execute the method described in the first aspect.

[0009] In a fifth aspect, an embodiment of the present disclosure proposes a computer program product, including a computer program, which implements the method described in the first aspect when executed by a processor.

[0010] The embodiments of the present disclosure provide an obstacle detection method, which can efficiently and accurately detect various types of under-segmented obstacles by judging the continuity of obstacles in previous and next frames.

[0011] The key or important features of the embodiments of the present disclosure are not intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Other features, purposes and advantages of the present disclosure will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings. The drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure. Among them:

[0013] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0014] Figure 2 is a flow chart of an embodiment of an obstacle detection method according to the present disclosure;

[0015] Figure 3 is a flow chart of another embodiment of the obstacle detection method according to the present disclosure;

[0016] Figure 4 is a flow chart of another embodiment of the obstacle detection method according to the present disclosure;

[0017] Figure 5 is a flowchart of an obstacle detection method that can implement an embodiment of the present disclosure;

[0018] Figure 6 is a structural schematic diagram of an embodiment of an obstacle detection device according to the present disclosure;

[0019] Figure 7 is a block diagram of an electronic device used to implement the obstacle detection method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0020] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0021] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0022] Figure 1 An exemplary system architecture 100 is shown in which the present application may be applied.

[0023] like Figure 1 As shown, the system architecture 100 may include a camera 101, a network 102, and a server 103. The network 102 is used to provide a medium for a communication link between the camera 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links, or optical fiber cables.

[0024] The camera 101 may be installed on the autonomous driving vehicle and interact with the server 103 via the network 102 to receive or send messages, etc. The camera 101 may collect a sequence of image frames around the autonomous driving vehicle and send them to the server 103 for processing.

[0025] The server 103 may be an onboard server or a cloud server of the autonomous driving vehicle. The server 103 may analyze and process the received image frame sequence, etc., and generate processing results (such as under-segmented obstacles) for subsequent control of the autonomous driving vehicle.

[0026] It should be noted that the server 103 can be hardware or software. When the server 103 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server 103 is software, it can be implemented as multiple software or software modules (for example, for providing distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0027] It should be noted that the obstacle detection method or perception model training method provided in the embodiments of the present application is generally executed by the server 103 , and accordingly, the obstacle detection device or perception model training device is generally arranged in the server 103 .

[0028] It should be understood that Figure 1The number of cameras, networks and servers in the embodiment is only for illustration. Any number of cameras, networks and servers may be provided as required.

[0029] Continue to refer Figure 2 , which shows a process 200 of an embodiment of an obstacle detection method according to the present disclosure. The obstacle detection method comprises the following steps:

[0030] Step 201 , detecting foreground obstacles in the last frame of image captured by a camera of the vehicle.

[0031] In this embodiment, the execution subject of the obstacle detection method (for example Figure 1 The server 103 shown in the figure can obtain the image frame sequence captured by the camera of the vehicle. For the previous frame image of the current frame image, the above-mentioned execution subject can detect the foreground obstacle in the previous frame image.

[0032] The foreground obstacle may be an obstacle in the foreground of the previous frame image. The foreground obstacle is usually close to the vehicle, and the vehicle may collide with the foreground obstacle if it continues to travel along the planned trajectory.

[0033] Usually, all obstacles in the previous frame image are detected and converted to the vehicle coordinate system. Obstacles that fall within the observed obstacle range are foreground obstacles. Obstacles that fall outside the observed obstacle range are rear obstacles and need to be filtered. Among them, the observed obstacle range can be determined by the distance from the vehicle and the distance from the planned trajectory of the vehicle. For example, an obstacle whose distance from the vehicle is less than a distance threshold (such as 50 meters) and whose vertical distance from the planned trajectory of the vehicle is less than the observation threshold will fall into the observed obstacle range.

[0034] Step 202: filter out mutually adjacent foreground obstacle pairs from the foreground obstacles as candidate under-segmentation obstacle pairs.

[0035] In this embodiment, the execution entity may filter out mutually adjacent foreground obstacle pairs from the foreground obstacles as candidate under-segmentation obstacle pairs.

[0036] A candidate under-segmented obstacle pair is also called a suspected under-segmented obstacle pair, which can include two adjacent foreground obstacles. These two adjacent foreground obstacles may be under-segmented and detected as the same obstacle. Usually, under-segmentation occurs between two obstacles that are close to each other. In order to improve the accuracy of under-segmentation, the bounding boxes of the two obstacles intersect. The bounding box can be the smallest cuboid that contains the obstacle and has any direction relative to the coordinate axis.

[0037] In some embodiments, a bounding box of each foreground obstacle is determined. If the bounding boxes of two foreground obstacles intersect, the two foreground obstacles are used as candidate under-segmentation obstacle pairs. By selecting obstacles with two intersecting bounding boxes as candidate under-segmentation obstacle pairs, the accuracy of under-segmentation characterization is improved.

[0038] Step 203 : predicting the position of the candidate under-segmented obstacle in the current frame image based on the speed of the candidate under-segmented obstacle in the candidate under-segmented obstacle pair.

[0039] In this embodiment, the execution subject may predict the position of the candidate under-segmented obstacle in the current frame image based on the speed of the candidate under-segmented obstacle in the candidate under-segmented obstacle pair.

[0040] Generally, the distance of the candidate under-segmented obstacle can be obtained by calculating the product of the speed of the candidate under-segmented obstacle and the time interval between two frames of images. The position of the candidate under-segmented obstacle in the previous frame of image is added by the distance to obtain the position of the candidate under-segmented obstacle in the current frame of image.

[0041] Step 204 : Detect under-segmented obstacles from the current frame image based on the positions of the candidate under-segmented obstacles.

[0042] In this embodiment, the execution subject may detect under-segmented obstacles from the current frame image based on the positions of the candidate under-segmented obstacles.

[0043] Usually, the under-segmented obstacle is near the location of the candidate under-segmented obstacle, has the same category as the candidate under-segmented obstacle, and has a length and width greater than those of the candidate under-segmented obstacle.

[0044] In some embodiments, for two candidate under-segmented obstacles in a candidate under-segmented obstacle pair, within two preset ranges centered on the position of the candidate under-segmented obstacle, an obstacle of the same category as one of the candidate under-segmented obstacles and having a length and width greater than the candidate under-segmented obstacle of the category is searched. If there is only one obstacle that meets the conditions, the obstacle is regarded as an under-segmented obstacle. The preset range is centered on the position of the candidate under-segmented obstacle in the current frame image, and the sum of the width of the candidate under-segmented obstacle and the preset value is the radius.

[0045] The embodiments of the present disclosure provide an obstacle detection method, which can efficiently and accurately detect various types of under-segmented obstacles by judging the continuity of obstacles in previous and next frames.

[0046] Further references Figure 3 , which shows a process 300 of an embodiment of an obstacle detection method according to the present disclosure. The obstacle detection method comprises the following steps:

[0047] Step 301: Detect obstacles in the previous frame of image.

[0048] In this embodiment, the execution subject of the obstacle detection method (for example Figure 1 The server 103 shown in the figure can obtain the image frame sequence taken by the camera of the vehicle. For the previous frame image of the current frame image, the above-mentioned execution subject can detect all obstacles in the previous frame image.

[0049] Usually, by inputting the previous frame of image into the perception model, all obstacles in the previous frame of image can be detected.

[0050] Step 302 : Select obstacles whose distance from the vehicle is less than a distance threshold and whose vertical distance from the planned trajectory of the vehicle is less than an observation threshold from the obstacles as foreground obstacles.

[0051] In this embodiment, the execution subject may select, from among the obstacles, an obstacle whose distance to the vehicle is less than a distance threshold and whose vertical distance to the planned trajectory of the vehicle is less than an observation threshold as a foreground obstacle.

[0052] The foreground obstacle may be an obstacle in the foreground of the previous frame image. The foreground obstacle is usually close to the vehicle, and the vehicle may collide with the foreground obstacle if it continues to travel along the planned trajectory.

[0053] Usually, all obstacles in the previous frame image are detected and converted to the vehicle coordinate system. Obstacles that fall within the observed obstacle range are foreground obstacles. Obstacles that fall outside the observed obstacle range are rear obstacles and need to be filtered. Among them, the observed obstacle range can be determined by the distance from the vehicle and the distance from the planned trajectory of the vehicle. For example, an obstacle whose distance from the vehicle is less than a distance threshold (such as 50 meters) and whose vertical distance from the planned trajectory of the vehicle is less than the observation threshold will fall into the observed obstacle range. Among them, the position of the obstacle is pos obs , the predicted trajectory of the obstacle and the vehicle is S = {pos1,pos2,...,pos n}, the vertical distance between the obstacle and the planned trajectory of the vehicle is min{dist(pos obs ,pos i ),i=0,1,...,n}. The observation threshold can be equal to the sum of the vehicle width, the obstacle width and the preset value, and its formula can be as follows:

[0054] L 观测阈值 = l 车宽 +l 障碍物宽度 +l 阈值 .

[0055] Among them, L 观测阈值 is the observation threshold, l车宽 is the vehicle length, l 障碍物宽度 is the obstacle width, l 阈值 It is a preset value, which can be an empirical value, such as 1 meter.

[0056] Step 303: calculate the distance between the foreground obstacles, and filter out the foreground obstacle pairs whose distance is less than the sum of the width of the two foreground obstacles and a preset value as the foreground obstacle pairs close to each other.

[0057] In this embodiment, the execution body may calculate the distance between two foreground obstacles, and filter out foreground obstacle pairs whose distance is less than the sum of the width of the two foreground obstacles and a preset value as foreground obstacle pairs close to each other.

[0058] Usually, under-segmentation occurs between two obstacles that are close to each other.

[0059] Among them, the distance formula between foreground obstacles is as follows:

[0060]

[0061] Among them, (x i ,y i ) is the coordinate of the i-th foreground obstacle, (x j ,y j ) is the coordinate of the jth foreground obstacle, if d ij <l 障碍物i宽度 +l 障碍物j宽度 +l 阈值 , indicating that the two foreground obstacles are close to each other, l 障碍物i宽度 is the width of the i-th foreground obstacle, l 障碍物j宽度 is the width of the jth foreground obstacle, l 阈值 It is a preset value, which can be an empirical value, such as 1 meter.

[0062] Step 304 : Filter out foreground obstacle pairs whose bounding boxes intersect from the foreground obstacle pairs that are close to each other.

[0063] In this embodiment, the execution entity may filter out foreground obstacle pairs whose bounding boxes intersect from foreground obstacle pairs that are close to each other.

[0064] Usually, under-segmentation occurs between two obstacles that are close to each other. In order to improve the accuracy of under-segmentation, the bounding boxes of the two obstacles are intersected. The bounding box can be the smallest cuboid that contains the obstacle and has any direction relative to the coordinate axis.

[0065] Step 305 : predicting the position of the candidate under-segmented obstacle in the current frame image based on the speed of the candidate under-segmented obstacle in the candidate under-segmented obstacle pair.

[0066] In this embodiment, the execution subject may predict the position of the candidate under-segmented obstacle in the current frame image based on the speed of the candidate under-segmented obstacle in the candidate under-segmented obstacle pair.

[0067] In this embodiment, the execution subject may predict the position of the candidate under-segmented obstacle in the current frame image based on the speed of the candidate under-segmented obstacle in the candidate under-segmented obstacle pair.

[0068] Generally, the distance of the candidate under-segmented obstacle can be obtained by calculating the product of the speed of the candidate under-segmented obstacle and the time interval between two frames of images. The position of the candidate under-segmented obstacle in the previous frame of image is added by the distance to obtain the position of the candidate under-segmented obstacle in the current frame of image.

[0069] The calculation formula for the position of the candidate under-segmented obstacle in the current frame image can be as follows:

[0070] x′=x+v x ;

[0071] y′=y+v y .

[0072] Among them, (x′, y′) is the position of the candidate under-segmented obstacle in the current frame image, (x, y) is the position of the candidate under-segmented obstacle in the previous frame image, and v x is the velocity of the candidate under-segmented obstacle in the X-axis direction, v y is the velocity of the candidate under-segmented obstacle in the Y-axis direction.

[0073] Step 306 : within a preset range centered on the position of the candidate under-segmented obstacle, search for an obstacle of the same category as the candidate under-segmented obstacle pair and with a greater length and width than the candidate under-segmented obstacle pair as the under-segmented obstacle.

[0074] In this embodiment, the execution subject may search for an obstacle of the same category as the candidate under-segmentation obstacle pair and having a greater length and width than the candidate under-segmentation obstacle pair within a preset range centered on the position of the candidate under-segmentation obstacle as the under-segmentation obstacle.

[0075] Usually, the under-segmented obstacle is near the location of the candidate under-segmented obstacle, has the same category as the candidate under-segmented obstacle, and has a length and width greater than those of the candidate under-segmented obstacle.

[0076] For two candidate under-segmented obstacles in a candidate under-segmented obstacle pair, within two preset ranges centered on the position of the candidate under-segmented obstacle, search for obstacles of the same category as one of the candidate under-segmented obstacles and with a length and width greater than that of the candidate under-segmented obstacles of the same category. If there is only one obstacle that meets the conditions, the obstacle is regarded as an under-segmented obstacle. The preset range is centered on the position (x′, y′) of the candidate under-segmented obstacle in the current frame image, and the sum of the width of the candidate under-segmented obstacle and the preset value is the radius r.

[0077] The radius r calculation formula can be as follows:

[0078] r=l 障碍物宽度 +l 阈值 .

[0079] l 障碍物宽度 is the obstacle width, l 阈值 It is a preset value, which can be an empirical value, such as 1 meter.

[0080] Generally, if the maximum value of the ratio of the width of the under-segmented obstacle to the maximum width of the candidate under-segmented obstacle pair, and the maximum value of the ratio of the length of the under-segmented obstacle to the maximum length of the candidate under-segmented obstacle pair, is greater than the change amplitude threshold, it means that the length and width of the obstacle are significantly larger than the length and width of the corresponding candidate under-segmented obstacle.

[0081] Among them, if the obstacle length of the current frame image is l cur , with a width of w cur , the maximum length of the candidate under-segmented obstacle pair in the previous frame is l max , the maximum width is w max , then the width variation is c w =w cur / w max , the length variation is c l = l cur / l max If max(c w ,c l )>c 阈值 , then the candidate under-segmented obstacle pair appears under-segmented in the current frame.

[0082] The disclosed embodiments provide an obstacle detection method, which combines the intersection of obstacle frames and the continuity judgment of obstacles in previous and next frames to further improve the detection efficiency and accuracy of under-segmented obstacles.

[0083] Further references Figure 4 , which shows a process 400 of another embodiment of the obstacle detection method according to the present disclosure. The obstacle detection method comprises the following steps:

[0084] Step 401 , detecting foreground obstacles in the last frame of image captured by the camera of the vehicle.

[0085] Step 402: Filter out mutually adjacent foreground obstacle pairs from the foreground obstacles as candidate under-segmentation obstacle pairs.

[0086] Step 403 : predicting the position of the candidate under-segmented obstacle in the current frame image based on the speed of the candidate under-segmented obstacle in the candidate under-segmented obstacle pair.

[0087] Step 404 : Detect under-segmented obstacles from the current frame image based on the positions of the candidate under-segmented obstacles.

[0088] In this embodiment, the specific operations of steps 401-404 are already described in Figure 2 In the illustrated embodiment, steps 201 - 204 are described in detail and will not be repeated here.

[0089] Step 405: labeling information of under-segmented obstacles in the current frame image.

[0090] In this embodiment, the execution subject of the obstacle detection method (for example Figure 1 The server 103 shown in the figure can annotate information of the under-segmented obstacle in the current frame image, wherein the annotated information may include but is not limited to: the category, size and position of the under-segmented obstacle.

[0091] Step 406, taking the current frame image as input and the information as output, performs model training to obtain a perception model.

[0092] In this embodiment, the execution subject can input the current frame image into the model and output the prediction result. Based on the prediction result and the annotated information, the loss is calculated. Based on the loss, the parameters are adjusted until the loss is small enough and the model converges to obtain the perception model. The perception model can be used to detect obstacles in the image, including under-segmented obstacles.

[0093] The disclosed embodiment provides an obstacle detection method, which can efficiently and accurately detect various types of under-segmented obstacles by judging the continuity of obstacles in previous and next frames. Under-segmented obstacles are discovered from massive data by means of data mining, thereby promoting iterative optimization of perception models and improving driving experience.

[0094] For ease of understanding, Figure 5 A flowchart of the obstacle detection method according to the embodiment of the present disclosure is shown.

[0095] like Figure 5As shown, when the vehicle is in autonomous driving mode, the deployed perception model can be used to obtain vehicle-side information, including but not limited to real-time location information, real-time perception reporting, real-time trajectory prediction, and real-time vehicle condition. Then, perform the following steps:

[0096] Step 501, reporting message records based on vehicle-side information perception.

[0097] Step 502, determine whether there are obstacles with two adjacent bounding boxes intersecting in the previous frame. If yes, execute step 504; if not, execute step 503.

[0098] Step 503, ignore.

[0099] Step 504, determine whether there is only one obstacle around the promotion position of the current frame. If so, execute step 505; if not, execute step 503.

[0100] Step 505: Calculate the length and width changes of the obstacle relative to the two obstacles in the previous frame.

[0101] Step 506, determine whether the length and width change is greater than a threshold. If yes, execute step 507; if not, execute step 503.

[0102] Step 507, analyzing the reasons for under-segmentation.

[0103] Step 508: data labeling.

[0104] Step 509: model training.

[0105] Step 510, model deployment.

[0106] The deployed perception model can be used to obtain vehicle-side information and return to continue executing step 501, thereby driving continuous iteration of the perception model.

[0107] Further references Figure 6 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of an obstacle detection device. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0108] like Figure 6As shown, the obstacle detection device 600 of this embodiment may include: a first detection module 601, a filtering module 602, a prediction module 603 and a second detection module 604. The first detection module 601 is configured to detect foreground obstacles in the previous frame image captured by the camera of the vehicle; the filtering module 602 is configured to filter out adjacent foreground obstacle pairs from the foreground obstacles as candidate under-segmented obstacle pairs; the prediction module 603 is configured to predict the position of the candidate under-segmented obstacles in the current frame image based on the speed of the candidate under-segmented obstacles in the candidate under-segmented obstacle pairs; the second detection module 604 is configured to detect under-segmented obstacles from the current frame image based on the position of the candidate under-segmented obstacles.

[0109] In the present embodiment, in the obstacle detection device 600, the specific processing of the first detection module 601, the filtering module 602, the prediction module 603 and the second detection module 604 and the technical effects thereof can be referred to in Figure 2 The relevant descriptions of steps 201 - 204 in the corresponding embodiment are not repeated here.

[0110] In some optional implementations of this embodiment, the first detection module 601 is further configured to: detect obstacles in the previous frame image; select obstacles from the obstacles whose distance from the vehicle is less than a distance threshold and whose vertical distance from the planned trajectory of the vehicle is less than an observation threshold as foreground obstacles.

[0111] In some optional implementations of this embodiment, the observation threshold is equal to the sum of the vehicle width, the obstacle width and a preset value.

[0112] In some optional implementations of this embodiment, the filtering module 602 includes: a filtering submodule configured to filter out foreground obstacle pairs whose bounding boxes intersect from the foreground obstacles.

[0113] In some optional implementations of this embodiment, the filtering submodule is further configured to: calculate the distance between foreground obstacles, filter out foreground obstacle pairs whose distance is less than the sum of the width of the two foreground obstacles and a preset value, as foreground obstacle pairs close to each other; and filter out foreground obstacle pairs whose bounding boxes intersect from the foreground obstacle pairs close to each other.

[0114] In some optional implementations of this embodiment, the second detection module 604 is further configured to: within a preset range centered on the position of the candidate under-segmented obstacle, search for an obstacle that is of the same category as the candidate under-segmented obstacle pair and has a length and width greater than the candidate under-segmented obstacle pair as an under-segmented obstacle.

[0115] In some optional implementations of this embodiment, the radius of the preset range is equal to the sum of the width of the candidate under-segmented obstacle and a preset value.

[0116] In some optional implementations of this embodiment, a maximum value of a ratio of a width of the under-segmented obstacle to a maximum width of the candidate under-segmented obstacle pair and a maximum value of a ratio of a length of the under-segmented obstacle to a maximum length of the candidate under-segmented obstacle pair is greater than a change amplitude threshold.

[0117] In some optional implementations of this embodiment, the obstacle detection device 600 also includes: a labeling module, configured to label information of under-segmented obstacles in the current frame image, wherein the information includes at least one of the category, size and position of the under-segmented obstacles; and a training module, configured to take the current frame image as input and the information as output, perform model training, and obtain a perception model.

[0118] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0119] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0120] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0121] like Figure 7 As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0122] A number of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0123] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as the obstacle detection method. For example, in some embodiments, the obstacle detection method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the obstacle detection method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the obstacle detection method in any other appropriate manner (e.g., by means of firmware).

[0124] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0125] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0126] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0127] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0128] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0129] A computer system may include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises through computer programs running on respective computers and having a client-server relationship to each other. The server may be a cloud server, a server in a distributed system, or a server combined with a blockchain.

[0130] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions provided by this disclosure can be achieved, and this document does not limit this.

[0131] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An obstacle detection method, comprising: Detect foreground obstacles in the last frame of image captured by the vehicle's camera; Filtering mutually adjacent foreground obstacle pairs from the foreground obstacles as candidate under-segmentation obstacle pairs; Predicting a position of the candidate under-segmented obstacle in the candidate under-segmented obstacle pair in the current frame image based on a speed of the candidate under-segmented obstacle in the candidate under-segmented obstacle pair; An obstacle satisfying a condition is searched within a preset range centered on the position of the candidate under-segmented obstacle in the candidate under-segmented obstacle pair. If there is only one obstacle satisfying the condition, the obstacle satisfying the condition is used as the under-segmented obstacle, wherein the condition includes being of the same category as a candidate under-segmented obstacle in the candidate under-segmented obstacle pair and having a length and width greater than that of the candidate under-segmented obstacle of the same category.

2. The method according to claim 1, wherein: The foreground obstacle in the previous frame image captured by the camera of the detection vehicle includes: Detecting obstacles in the previous frame of image; An obstacle whose distance from the vehicle is less than a distance threshold and whose vertical distance from the planned trajectory of the vehicle is less than an observation threshold is selected from the obstacles as the foreground obstacle.

3. The method according to claim 2, wherein: The observation threshold is equal to the sum of the vehicle width, the obstacle width and a preset value.

4. The method according to claim 1, wherein: The filtering out mutually adjacent foreground obstacle pairs from the foreground obstacles as candidate under-segmentation obstacle pairs includes: Foreground obstacle pairs whose bounding boxes intersect with each other are filtered out from the foreground obstacles.

5. The method according to claim 4, wherein: The filtering out foreground obstacle pairs whose bounding boxes intersect from the foreground obstacles comprises: Calculating the distance between the foreground obstacles, filtering out foreground obstacle pairs whose distance is less than the sum of the width of the two foreground obstacles and a preset value as foreground obstacle pairs close to each other; The foreground obstacle pairs whose bounding boxes intersect are filtered out from the foreground obstacle pairs that are close to each other.

6. The method according to claim 1, wherein: The radius of the preset range is equal to the sum of the width of the candidate under-segmented obstacle in the candidate under-segmented obstacle pair and a preset value.

7. The method according to claim 1, wherein: The method further comprises: If the maximum value of the ratio of the width of the obstacle to the maximum width of the candidate under-segmented obstacle pair and the maximum value of the ratio of the length of the obstacle to the maximum length of the candidate under-segmented obstacle pair is greater than the change amplitude threshold, it is determined that the candidate under-segmented obstacle pair is under-segmented in the current frame image.

8. The method according to any one of claims 1 to 7, wherein: The method further comprises: Labeling information of the under-segmented obstacle in the current frame image, wherein the information includes at least one of a category, a size, and a position of the under-segmented obstacle: The current frame image is taken as input, and the information is taken as output, and model training is performed to obtain a perception model.

9. An obstacle detection device, comprising: A first detection module is configured to detect a foreground obstacle in a previous frame of image captured by a camera of the vehicle; A filtering module, configured to filter out mutually adjacent foreground obstacle pairs from the foreground obstacles as candidate under-segmentation obstacle pairs; A prediction module, configured to predict a position of a candidate under-segmented obstacle in the candidate under-segmented obstacle pair in a current frame image based on a speed of the candidate under-segmented obstacle in the candidate under-segmented obstacle pair; The second detection module is configured to search for an obstacle that meets the conditions within a preset range centered on the position of the candidate under-segmented obstacle in the candidate under-segmented obstacle pair, and if there is only one obstacle that meets the conditions, use the obstacle that meets the conditions as the under-segmented obstacle, wherein the conditions include being of the same category as one of the candidate under-segmented obstacles in the candidate under-segmented obstacle pair and having a length and width greater than the candidate under-segmented obstacles of the same category.

10. The device according to claim 9, wherein: The first detection module is further configured to: Detecting obstacles in the previous frame of image; An obstacle whose distance from the vehicle is less than a distance threshold and whose vertical distance from the planned trajectory of the vehicle is less than an observation threshold is selected from the obstacles as the foreground obstacle.

11. The device according to claim 10, wherein: The observation threshold is equal to the sum of the vehicle width, the obstacle width and a preset value.

12. The device according to claim 9, wherein: The filtering module comprises: The filtering submodule is configured to filter out foreground obstacle pairs whose bounding boxes intersect from the foreground obstacles.

13. The device according to claim 12, wherein: The filtering submodule is further configured to: Calculating the distance between the foreground obstacles, filtering out foreground obstacle pairs whose distance is less than the sum of the width of the two foreground obstacles and a preset value as foreground obstacle pairs close to each other; The foreground obstacle pairs whose bounding boxes intersect are filtered out from the foreground obstacle pairs that are close to each other.

14. The device according to claim 13, wherein: The radius of the preset range is equal to the sum of the width of the candidate under-segmented obstacle and a preset value.

15. The device according to claim 9, wherein: The device also includes: The determination module is configured to determine that the candidate under-segmented obstacle pair is under-segmented in the current frame image if the maximum value of the ratio of the width of the obstacle to the maximum width of the candidate under-segmented obstacle pair and the ratio of the length of the obstacle to the maximum length of the candidate under-segmented obstacle pair is greater than a change amplitude threshold.

16. The device according to any one of claims 9 to 15, wherein: The device also includes: a labeling module, configured to label information of the under-segmented obstacle in the current frame image, wherein the information includes at least one of a category, a size, and a position of the under-segmented obstacle: The training module is configured to take the current frame image as input and the information as output to perform model training and obtain a perception model.

17. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.

18. A non-transitory computer-readable storage medium storing computer instructions, the computer instructions being used to cause the computer to execute the method of any one of claims 1-8.

19. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 8.

Images