Obstacle recognition method, obstacle recognition model training method

Abstract

The application discloses an obstacle identification method and an obstacle identification model training method, and belongs to the technical field of automatic driving. The method comprises the following steps: acquiring an image of a target scene; identifying the image through a closed set identification model to obtain a first identification result, wherein the closed set identification model is used for identifying a learned known-class obstacle image, and the first identification result comprises an identified obstacle image and a class of the obstacle image; identifying the image through an open set identification model to obtain a second identification result, wherein the open set identification model is used for identifying an unknown-class obstacle image based on the learned known-class obstacle image, and the second identification result comprises an identified obstacle image; and fusing the first identification result and the second identification result to obtain a third identification result. The method improves the accuracy of the identification result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, and in particular to an obstacle identification method and an obstacle identification model training method. BACKGROUND

[0002] Automatic driving is a technology of realizing automatic driving of a vehicle on a road through a computer system. Since the actual road condition is complex, there are a large number of pedestrians and vehicles as obstacles, how to realize obstacle identification and then plan a driving route to avoid obstacles becomes a key of automatic driving.

[0003] In the related art, a camera is arranged on an automatic driving vehicle, an environment image is collected through the camera, and an obstacle in the environment image is identified through a closed set identification model. However, the closed set identification model is obtained based on a sample labeled by a person, if only a car, a pedestrian and a bicycle are labeled in the sample, the closed set identification model can only identify the car, the pedestrian and the bicycle. However, there are various obstacles on the road where the automatic driving vehicle drives, and the closed set identification model often has a missing detection problem, and the accuracy of obstacle identification is low. SUMMARY

[0004] Embodiments of the present application provide an obstacle identification method and an obstacle identification model training method, which improve the accuracy of obstacle identification. The technical solution is as follows:

[0005] On one hand, an obstacle identification method is provided, and the method comprises:

[0006] obtaining an image of a target scene;

[0007] identifying the image through a closed set identification model to obtain a first identification result, the closed set identification model is used to identify a learned obstacle image of a known category, and the first identification result comprises an identified obstacle image and a category of the obstacle image;

[0008] identifying the image through an open set identification model to obtain a second identification result, the open set identification model is used to identify an obstacle image of an unknown category based on a learned obstacle image of a known category, and the second identification result comprises an identified obstacle image;

[0009] fusing the first identification result and the second identification result to obtain a third identification result.

[0010] In a possible implementation manner, the distance between the obstacle and the road in the identified obstacle image is determined based on the second identification result, comprising:

[0011] For any obstacle image in the second identification result, a road image is identified from an image to which the obstacle image belongs;

[0012] Based on the position of the obstacle image and the road image in the image, a distance between an obstacle indicated by the obstacle image and a road indicated by the road image in the image is determined.

[0013] In a possible implementation, the identification of the road image from the image to which the obstacle image belongs includes:

[0014] Obtaining point cloud data corresponding to the image, the image and the point cloud data corresponding to a same scene; obtaining points belonging to a road surface from the point cloud data based on the height of points in the point cloud data; projecting the obtained points into the image to obtain the road image in the image; or,

[0015] Processing the image through a road segmentation model to obtain the road image in the image.

[0016] In a possible implementation, the method further includes:

[0017] For any obstacle image in the second identification result, obtaining point cloud data based on an image to which the obstacle image belongs, the image and the point cloud data corresponding to a same scene; mapping points in the point cloud data to the image, determining a depth map of the image based on the depth of points in the point cloud data, and determining an image depth of the obstacle image in the image based on the depth map of the image; or,

[0018] For any obstacle image in the second identification result, obtaining point cloud data based on an image to which the obstacle image belongs, the image and the point cloud data corresponding to a same scene; mapping points in the point cloud data to the image, determining an image depth of the obstacle image based on the depth of point cloud data corresponding to the obstacle image in the image; or,

[0019] For any obstacle image in the second identification result, determining an image depth of the obstacle image through a depth determination model.

[0020] In an aspect, a method for training an obstacle identification model is provided, and the method includes:

[0021] Obtaining a sample set, the sample set including a plurality of sample images;

[0022] In an Nth iteration of the training process, an open set identification model corresponding to an (N-1)th iteration is used to identify obstacle images of unknown classes in a sample set corresponding to the (N-1)th iteration, and the identified obstacle images are labeled as unknown classes, where the open set identification model is used to identify obstacle images of unknown classes based on learned obstacle images of known classes, and N is a positive integer greater than 1.

[0023] A closed set identification model is trained based on the labeled sample set to obtain a closed set identification model corresponding to the Nth iteration, and the closed set identification model is used to identify obstacle images of known classes.

[0024] If the training process reaches a loop termination condition, the closed set identification model corresponding to the Nth iteration is output as an obstacle identification model.

[0025] If the training process does not reach the loop termination condition, the multiple sample images in the sample set are identified by using a closed set identification model corresponding to the Nth iteration, the multiple sample images in the sample set are labeled based on the identification result to obtain a sample set corresponding to the Nth iteration, and the open set identification model corresponding to the (N-1)th iteration is trained based on the sample set corresponding to the Nth iteration to obtain the open set identification model corresponding to the Nth iteration.

[0026] In a possible implementation, the labeling of the identified obstacle images as unknown classes includes:

[0027] Obstacle images meeting an obstacle condition are filtered from the identified obstacle images, and the obstacle condition is a condition under which an object affects driving of an autonomous vehicle.

[0028] The filtered obstacle images are labeled as the unknown classes.

[0029] In a possible implementation, the filtering of the obstacle images meeting the obstacle condition from the identified obstacle images includes at least one of the following:

[0030] The obstacle condition includes a distance threshold, the distance threshold indicates that an object with a distance less than the distance threshold from a road is an obstacle affecting driving of the autonomous vehicle, and the distance of an obstacle from the road in the identified obstacle image is determined, and the obstacle image with a distance less than the distance threshold is filtered from the identified obstacle images.

[0031] The obstacle condition includes a depth threshold, the depth threshold indicates that an object affecting driving of the autonomous vehicle has a maximum image depth in a driving environment image, and the image depth of the identified obstacle image is used to filter the obstacle image with an image depth greater than the depth threshold from the identified obstacle images.

[0032] The obstacle condition includes a size threshold indicating a minimum image size of an object affecting driving of the autonomous vehicle in the driving environment image; and the identified obstacle image is filtered out from the identified obstacle image based on an image size of the identified obstacle image, and the obstacle image with an image size greater than the size threshold.

[0033] In a possible implementation, the determining the distance between the obstacle in the identified obstacle image and the road includes:

[0034] For any identified obstacle image, a road image is identified from a sample image to which the obstacle image belongs.

[0035] Based on positions of the obstacle image and the road image in the sample image, a distance between the obstacle indicated by the obstacle image and the road indicated by the road image in the sample image is determined.

[0036] In a possible implementation, the identifying the road image from the sample image to which the obstacle image belongs includes:

[0037] Point cloud data corresponding to the sample image is obtained, the sample image and the point cloud data corresponding to the same scene; points belonging to a road surface are obtained from the point cloud data based on heights of points in the point cloud data; the obtained points are projected into the sample image to obtain a road image in the sample image; or

[0038] The sample image is processed by a road segmentation model to obtain a road image in the sample image.

[0039] In a possible implementation, the method further includes:

[0040] For any identified obstacle image, point cloud data is obtained based on a sample image to which the obstacle image belongs, the sample image and the point cloud data corresponding to the same scene; points in the point cloud data are mapped into the sample image, a depth map of the sample image is determined based on depths of points in the point cloud data, and an image depth of the obstacle image in the sample image is determined based on the depth map of the sample image; or

[0041] For any identified obstacle image, point cloud data is obtained based on a sample image to which the obstacle image belongs, the sample image and the point cloud data corresponding to the same scene; points in the point cloud data are mapped into the sample image, a depth map of the sample image is determined based on depths of points in the point cloud data, and an image depth of the obstacle image in the sample image is determined based on the depth map of the sample image; or

[0042] For any identified obstacle image, an image depth of the obstacle image is determined by a depth determination model.

[0043] In a possible implementation, the method further includes:

[0044] In each iteration process, point cloud data corresponding to the sample image is acquired, and the sample image and the point cloud data correspond to a same scene;

[0045] Points in the point cloud data that are not more than a distance threshold away from a road are clustered to obtain a plurality of point clusters;

[0046] The plurality of point clusters are mapped to the sample image, and if an object image corresponding to any point cluster in the sample image is not an obstacle image identified by the open set identification model, the object image is determined as an obstacle image and is labeled as an unknown category.

[0047] In a possible implementation, the labeling of the plurality of sample images in the sample set based on the identification result, to obtain a sample set corresponding to the Nth iteration, includes:

[0048] In the sample set, the identified obstacle images are labeled based on categories to which the identified obstacle images belong;

[0049] From the identified obstacle images, an obstacle image that meets a confidence condition and belongs to the unknown category is screened out;

[0050] The screened-out obstacle image is position-reset in a target manner to obtain the sample set corresponding to the Nth iteration.

[0051] In a possible implementation, the output of the closed set identification model corresponding to the Nth iteration as the obstacle identification model if the training process reaches a loop termination condition includes:

[0052] If the number of iterations of the training process reaches a target number, the closed set identification model corresponding to the Nth iteration is output as the obstacle identification model.

[0053] In an aspect, an obstacle identification apparatus is provided, and the apparatus includes:

[0054] An acquisition module is configured to acquire an image of a target scene;

[0055] The first identification module is configured to identify the image by using a closed set identification model to obtain a first identification result, the closed set identification model is configured to identify a known-class obstacle image, and the first identification result includes the identified obstacle image and a class of the obstacle image.

[0056] The second identification module is configured to identify the image by using an open set identification model to obtain a second identification result, the open set identification model is configured to identify an unknown-class obstacle image based on the known-class obstacle image, and the second identification result includes the identified obstacle image.

[0057] The fusion module is configured to fuse the first identification result and the second identification result to obtain a third identification result.

[0058] In a possible implementation, the fusion module includes:

[0059] The screening unit is configured to screen obstacle images that meet an obstacle condition from the second identification result to obtain a fourth identification result, the obstacle condition is a condition under which an object affects driving of an autonomous vehicle.

[0060] The fusion unit is configured to fuse the first identification result and the fourth identification result to obtain the third identification result.

[0061] In a possible implementation, the obstacle condition includes a distance threshold, the distance threshold indicates that an object with a distance less than the distance threshold from a road is an obstacle that affects driving of an autonomous vehicle, and the screening unit is configured to determine distances from the road of the identified obstacle images based on the second identification result, and screen obstacle images with distances less than the distance threshold from the second identification result.

[0062] The obstacle condition includes a depth threshold, the depth threshold indicates that an object that affects driving of an autonomous vehicle has a maximum image depth in a driving environment image, and the screening unit is configured to determine image depths of the identified obstacle images based on the second identification result, and screen obstacle images with image depths greater than the depth threshold from the second identification result.

[0063] The obstacle condition includes a size threshold, the size threshold indicates that an object that affects driving of an autonomous vehicle has a minimum image size in a driving environment image, and the screening unit is configured to determine image sizes of the obstacle images in the second identification result, and screen obstacle images with image sizes greater than the size threshold from the second identification result.

[0064] In a possible implementation, the screening unit is configured to, for any obstacle image in the second identification result, identify a road image from an image to which the obstacle image belongs; and determine a distance between an obstacle indicated by the obstacle image and a road indicated by the road image in the image based on positions of the obstacle image and the road image in the image.

[0065] In a possible implementation, the screening unit is configured to obtain point cloud data corresponding to the image, the image and the point cloud data corresponding to a same scene; obtain points belonging to a road surface from the point cloud data based on heights of points in the point cloud data; project the obtained points into the image to obtain a road image in the image; or,

[0066] The screening unit is configured to process the image by using a road segmentation model to obtain a road image in the image.

[0067] In a possible implementation, the screening unit is configured to, for any obstacle image in the second identification result, obtain point cloud data based on an image to which the obstacle image belongs, the image and the point cloud data corresponding to a same scene; map points in the point cloud data to the image, determine a depth map of the image based on depths of points in the point cloud data, and determine an image depth of the obstacle image in the image based on the depth map of the image; or,

[0068] The screening unit is configured to, for any obstacle image in the second identification result, obtain point cloud data based on an image to which the obstacle image belongs, the image and the point cloud data corresponding to a same scene; map points in the point cloud data to the image, determine an image depth of the obstacle image based on depths of point cloud data corresponding to the obstacle image in the image; or,

[0069] The screening unit is configured to, for any obstacle image in the second identification result, determine an image depth of the obstacle image by using a depth determination model.

[0070] In a possible implementation, the fusion module is configured to: acquire point cloud data of the target scene; cluster points in the point cloud data that are not more than a distance threshold away from a road to obtain a plurality of point clusters; map the plurality of point clusters to the sample image to obtain object images corresponding to the plurality of point clusters respectively; determine a fifth identification result based on the object images corresponding to the plurality of point clusters respectively and the second identification result, the fifth identification result including object images in the object images corresponding to the plurality of point clusters that do not belong to the second identification result; and fuse the first identification result, the second identification result, and the fifth identification result to obtain the third identification result.

[0071] In an aspect, a device for training an obstacle identification model is provided, and the device includes:

[0072] An acquisition module configured to acquire a sample set, the sample set including a plurality of sample images;

[0073] A labeling module configured to, in an Nth iteration of a training process, identify, by an open set identification model corresponding to an (N-1)th iteration, an obstacle image of an unknown category in a sample set corresponding to the (N-1)th iteration, and label the identified obstacle image as the unknown category, the open set identification model being configured to identify an obstacle image of an unknown category based on learned obstacle images of known categories, N being a positive integer greater than 1.

[0074] A training module configured to train a closed set identification model based on the labeled sample set to obtain a closed set identification model corresponding to the Nth iteration, the closed set identification model being configured to identify obstacle images of known categories.

[0075] An output module configured to, if the training process reaches a loop termination condition, output the closed set identification model corresponding to the Nth iteration as the obstacle identification model.

[0076] The training module is further configured to, if the training process does not reach the loop termination condition, identify, by the closed set identification model corresponding to the Nth iteration, the plurality of sample images in the sample set, label the plurality of sample images in the sample set based on an identification result to obtain a sample set corresponding to the Nth iteration, train the open set identification model corresponding to the (N-1)th iteration based on the sample set corresponding to the Nth iteration to obtain the open set identification model corresponding to the Nth iteration.

[0077] In a possible implementation, the labeling module includes:

[0078] A screening unit configured to screen, from the identified obstacle images, an obstacle image that satisfies an obstacle condition, the obstacle condition being a condition under which an object affects driving of an autonomous vehicle.

[0079] The labeling unit is configured to label the screened obstacle image as the unknown category.

[0080] In a possible implementation, the screening unit is configured to perform at least one of the following:

[0081] The obstacle condition includes a distance threshold, the distance threshold indicating that an object with a distance less than the distance threshold from a road is an obstacle affecting driving of an autonomous vehicle; a distance of an obstacle from the road in the identified obstacle image is determined, and an obstacle image with a distance less than the distance threshold is screened from the identified obstacle image.

[0082] The obstacle condition includes a depth threshold, the depth threshold indicating that an object affecting driving of an autonomous vehicle has a maximum image depth in a driving environment image; an image depth of the identified obstacle image is determined, and an obstacle image with an image depth greater than the depth threshold is screened from the identified obstacle image.

[0083] The obstacle condition includes a size threshold, the size threshold indicating that an object affecting driving of an autonomous vehicle has a minimum image size in a driving environment image; an image size of the identified obstacle image is determined, and an obstacle image with an image size greater than the size threshold is screened from the identified obstacle image.

[0084] In a possible implementation, the screening unit is configured to, for any identified obstacle image, identify a road image from a sample image to which the obstacle image belongs; and determine a distance between an obstacle indicated by the obstacle image and a road indicated by the road image in the sample image based on positions of the obstacle image and the road image in the sample image.

[0085] In a possible implementation, the screening unit is configured to obtain point cloud data corresponding to the sample image, the sample image and the point cloud data corresponding to a same scene; obtain points belonging to a road surface from the point cloud data based on heights of points in the point cloud data; project the obtained points to the sample image to obtain a road image in the sample image; or

[0086] The screening unit is configured to process the sample image by using a road segmentation model to obtain a road image in the sample image.

[0087] In a possible implementation, the apparatus further includes:

[0088] determining, for any identified obstacle image, an image depth of the obstacle image based on a sample image to which the obstacle image belongs, the sample image and the point cloud data corresponding to a same scene; mapping points in the point cloud data to the sample image, determining a depth map of the sample image based on depths of the points in the point cloud data, and determining the image depth of the obstacle image based on the depth map of the sample image; or

[0089] determining, for any identified obstacle image, an image depth of the obstacle image based on a sample image to which the obstacle image belongs, the sample image and the point cloud data corresponding to a same scene; mapping points in the point cloud data to the sample image, determining a depth map of the sample image based on depths of the points in the point cloud data, and determining the image depth of the obstacle image based on the depth map of the sample image; or

[0090] The determining module is configured to determine, for any identified obstacle image, an image depth of the obstacle image by using a depth determination model.

[0091] In a possible implementation, the apparatus further includes:

[0092] The obtaining module is further configured to obtain, in each iteration process, point cloud data corresponding to the sample image, the sample image and the point cloud data corresponding to a same scene.

[0093] The clustering module is configured to cluster points in the point cloud data that are not more than a distance threshold away from the road, to obtain a plurality of point clusters.

[0094] The labeling module is further configured to map the plurality of point clusters to the sample image, and determine an object image corresponding to any point cluster as an obstacle image and label the obstacle image as an unknown category, if the object image is not an obstacle image identified by the open set recognition model.

[0095] In a possible implementation, the training module is configured to label, in the sample set, identified obstacle images based on categories to which the identified obstacle images belong; filter, from the identified obstacle images, obstacle images that satisfy a confidence condition and belong to the unknown category; and reset positions of the filtered obstacle images in a target manner, to obtain a sample set corresponding to the Nth iteration.

[0096] In a possible implementation, the output module is configured to output, if a number of iterations of the training process reaches a target number, a closed set recognition model corresponding to the Nth iteration as an obstacle recognition model.

[0097] In an aspect, a computer device is provided, which includes one or more processors and one or more memories having at least one program code stored therein, which is loaded and executed by the one or more processors to implement operations performed by the obstacle identification method according to any possible implementation described above, or to implement operations performed by the obstacle identification model training method according to any possible implementation described above.

[0098] In an aspect, a computer readable storage medium is provided, which has at least one program code stored therein, which is loaded and executed by a processor to implement operations performed by the obstacle identification method according to any possible implementation described above, or to implement operations performed by the obstacle identification model training method according to any possible implementation described above.

[0099] In an aspect, a computer program or computer program product is provided, which includes a computer program code, which, when executed by a computer, causes the computer to implement operations performed by the obstacle identification method according to any possible implementation described above, or to implement operations performed by the obstacle identification model training method according to any possible implementation described above.

[0100] The obstacle identification method provided by the embodiments of the present application considers that the obstacle images identified by the closed set identification model are not comprehensive enough, but the accuracy of the identification result is relatively high; and the obstacle images identified by the open set identification model are relatively comprehensive, but the accuracy of the identification result is relatively low. Therefore, the embodiments of the present application identify the same image by the closed set identification model and the open set identification model, fuse the two identification results as the final identification result, and improve the accuracy of the identification result.

[0101] The obstacle identification model training method provided by the embodiments of the present application mines the obstacle images of new categories in the sample set by the open set identification model, iteratively trains the open set identification model and the closed set identification model, so that the open set identification model and the closed set identification model can identify more categories of obstacle images, improves the obstacle identification capability of the open set identification model and the closed set identification model, and finally obtains the closed set identification model as the obstacle identification model, and improves the accuracy of the obstacle identification model. BRIEF DESCRIPTION OF DRAWINGS

[0102] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0103] Figure 1 is a schematic diagram of an implementation environment provided by an embodiment of the present application;

[0104] Figure 2 is a flowchart of a method for training an obstacle identification model provided by an embodiment of the present application;

[0105] Figure 3 is a flowchart of a method for training an obstacle identification model provided by an embodiment of the present application;

[0106] Figure 4 is a flowchart of a method for training an obstacle identification model provided by an embodiment of the present application;

[0107] Figure 5 is a flowchart of a method for identifying an obstacle provided by an embodiment of the present application;

[0108] Figure 6 is a structural schematic diagram of an apparatus for training an obstacle identification model provided by an embodiment of the present application;

[0109] Figure 7 is a structural schematic diagram of another apparatus for training an obstacle identification model provided by an embodiment of the present application;

[0110] Figure 8 is a structural schematic diagram of an apparatus for identifying an obstacle provided by an embodiment of the present application;

[0111] Figure 9 is a structural schematic diagram of another apparatus for identifying an obstacle provided by an embodiment of the present application;

[0112] Figure 10 is a structural schematic diagram of a terminal provided by an embodiment of the present application;

[0113] Figure 11 is a structural schematic diagram of a server provided by an embodiment of the present application. DETAILED DESCRIPTION

[0114] To make the objectives, technical solutions and advantages of the present application clearer, the following will further describe the embodiments of the present application in detail with reference to the accompanying drawings.

[0115] It can be understood that the terms “first”, “second”, etc. used in the present application can be used herein to describe various concepts, but unless specifically stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the present application, a first sample image can be referred to as a second sample image, and a second sample image can be referred to as a first sample image.

[0116] The terms "at least one", "multiple", "each", "any" used in this application include one, two or more than two, multiple includes two or more than two, and each refers to each of the corresponding multiple, any refers to any one of the multiple, for example, a plurality of sample images includes 3 sample images, and each refers to each of the 3 sample images, and any refers to any one of the 3 sample images, which can be the first, the second or the third.

[0117] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data, etc.) and signals involved in the present application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions. For example, the sample images and point cloud data involved in the present application are obtained under full authorization. And the above information and data are used in big data application scenarios after processing, and cannot be identified to any natural person or have a specific association with them.

[0118] In some embodiments, the obstacle recognition model training method provided by the embodiments of the present application is executed by a computer device. In some embodiments, the computer device is a terminal, which is any type of terminal such as a mobile phone, a computer, a tablet computer, an autonomous vehicle, etc. The autonomous vehicle includes a vehicle (e.g., a car, a truck, a bus, etc.) that travels on the ground, and can also include a vehicle (e.g., a drone, an airplane, a helicopter, etc.) that travels in the air, and can also include a vehicle (e.g., a ship, a submarine, etc.) that travels on or in water. The autonomous vehicle can or can not accommodate one or more passengers. In addition, the autonomous vehicle can be applied to the field of unmanned delivery, such as the field of express delivery, the field of take-out delivery, etc.

[0119] In other embodiments, the computer device is a server, which is a server, or a server cluster composed of several servers, or a cloud computing service center. In other embodiments, the computer device includes an autonomous vehicle and a server.

[0120] It should be noted that the embodiments of the present application do not limit the execution subject of the obstacle recognition model training method.

[0121] Figure 1 is a schematic diagram of an implementation environment provided by the embodiments of the present application, as Figure 1 shown, the implementation environment includes an autonomous vehicle 101 and a server 102, and the autonomous vehicle 101 and the server 102 are connected through a wireless or wired network.

[0122] In some embodiments, the server 102 is a server providing services for the autonomous vehicle 101. Optionally, the server 102 provides an electronic map for the autonomous vehicle 101, the server 102 is configured to update an obstacle recognition model of the autonomous vehicle, and the role of the server 102 is not limited in the embodiments of the present application.

[0123] In some embodiments, the server 102 is configured to train the obstacle recognition model, and deploy the trained obstacle recognition model to the autonomous vehicle 101.

[0124] Figure 2 FIG. 1 is a flowchart of an obstacle recognition model training method according to an embodiment of the present application. The embodiments of the present application are exemplarily described taking a computer device as an execution subject, and the embodiments include:

[0125] 201. The computer device acquires a sample set, the sample set including a plurality of sample images.

[0126] The sample image can be any image. Optionally, the sample image is an image of a road taken. In some embodiments, the sample image is an image taken by the autonomous vehicle during driving, or an image taken by a camera arranged on the road, or an image acquired through other manners, and the sample image is not limited in the embodiments of the present application.

[0127] 202. In the Nth iteration of the training process, the computer device identifies unknown-class obstacle images in the sample set corresponding to the (N-1)th iteration through an open set recognition model corresponding to the (N-1)th iteration, and labels the identified obstacle images as unknown class, the open set recognition model being configured to identify unknown-class obstacle images based on learned known-class obstacle images, N being a positive integer greater than 1.

[0128] In the embodiments of the present application, the open set recognition model corresponding to the (N-1)th iteration is trained through the sample set corresponding to the (N-1)th iteration, and thus the known-class obstacle images learned by the open set recognition model corresponding to the (N-1)th iteration are the obstacle images labeled in the sample set corresponding to the (N-1)th iteration. Through the open set recognition model corresponding to the (N-1)th iteration, the sample set corresponding to the (N-1)th iteration is identified, not only the obstacle images labeled in the sample set corresponding to the (N-1)th iteration can be identified, but also the obstacle images unlabeled in the sample set corresponding to the (N-1)th iteration can be identified, and the classes of these obstacle images are determined as unknown class.

[0129] In some embodiments, when the open set recognition model identifies the image, the open set recognition model first detects an object image from the image, determines a confidence of the object image, the confidence indicating a probability that an object in the object image is an obstacle. If the confidence of the object image is greater than a first threshold, the object image is classified into a known class, and a class that is most similar to the object image is selected from the known class as a class corresponding to the object image. If the confidence of the object image is less than the first threshold and greater than a second threshold, the object image is determined to be an obstacle image, and a class corresponding to the object image is an unknown class. The first threshold is greater than the second threshold. Embodiments of the present application do not limit the specific values of the first threshold and the second threshold. Optionally, the first threshold is 0.8, and the second threshold is 0.3.

[0130] If the open set recognition model is trained using a certain sample set, the confidence of the obstacle images labeled in the sample set in the open set recognition model will increase, even to 1. In this way, the confidence of other object images similar to the obstacle images will also increase accordingly, so that the open set recognition model determines other object images similar to the obstacle images as obstacle images of an unknown class.

[0131] For example, the open set recognition model is trained using a sample set in which three types of obstacles, i.e., a car, a pedestrian, and a bicycle, are labeled. The trained open set recognition model can not only recognize the three types of obstacles, i.e., the car, the pedestrian, and the bicycle, but also recognize other obstacles similar to the three types of obstacles, i.e., a tricycle, and determine the recognized other obstacles as an unknown class.

[0132] 203. The computer device trains a closed set recognition model based on the labeled sample set to obtain a closed set recognition model corresponding to the Nth iteration, the closed set recognition model being used to identify obstacle images of a known class.

[0133] The closed set recognition model can only identify obstacle images of a known class. For example, the closed set recognition model is trained using a sample set in which three types of obstacles, i.e., a car, a pedestrian, and a bicycle, are labeled. The trained closed set recognition model can only recognize the three types of obstacles, i.e., the car, the pedestrian, and the bicycle.

[0134] Since the open set recognition model labels the identified obstacle images of an unknown class as an unknown class, the “unknown class” can be regarded as a “known class”. Based on the labeled sample set, the closed set recognition model is trained so that the closed set recognition model can learn obstacle images of a known class and obstacle images of an unknown class labeled in the sample set.

[0135] 204. If the training process reaches a loop termination condition, the closed set recognition model corresponding to the Nth iteration is output as an obstacle recognition model.

[0136] The training process in the embodiments of the present application is a process of multiple iterations, and the training is stopped when the training process reaches a loop termination condition. Alternatively, the loop termination condition is that the number of iterations reaches a target number.

[0137] Through the N iteration process, the open set recognition model continuously mines new obstacle images in the sample set, so that the closed set recognition model corresponding to the Nth iteration can recognize more categories of obstacle images, greatly improving the obstacle recognition capability of the closed set recognition model. The closed set recognition model corresponding to the Nth iteration can be output as an obstacle recognition model, which will be put into use later.

[0138] 205、If the training process does not reach the loop termination condition, the multiple sample images in the sample set are recognized based on the closed set recognition model corresponding to the Nth iteration, the multiple sample images in the sample set are labeled based on the recognition result, the sample set corresponding to the Nth iteration is obtained, and the open set recognition model corresponding to the N-1th iteration is trained based on the sample set corresponding to the Nth iteration to obtain the open set recognition model corresponding to the Nth iteration.

[0139] If the training process does not reach the loop termination condition, the training needs to be continued. In order to enable the open set recognition model to continue to mine new categories of obstacles from the sample set, the open set recognition model needs to be retrained.

[0140] Taking the tricycle mined by the open set recognition model corresponding to the N-1th iteration as an example, the open set recognition model corresponding to the N-1th iteration may not be able to recognize all the tricycle images in the sample set, so the accuracy of the "labeled sample set" in the above step 203 is low. In order to obtain a more accurate sample set, the embodiments of the present application recognize the multiple sample images in the sample set based on the closed set recognition model corresponding to the Nth iteration, label the multiple sample images in the sample set based on the recognition result, and obtain the sample set corresponding to the Nth iteration. The open set recognition model corresponding to the N-1th iteration is trained based on the sample set corresponding to the Nth iteration to obtain the open set recognition model corresponding to the Nth iteration.

[0141] The obstacle recognition model training method provided by the embodiments of the present application mines new categories of obstacle images in the sample set through the open set recognition model, and iteratively trains the open set recognition model and the closed set recognition model, so that the open set recognition model and the closed set recognition model can recognize more categories of obstacle images, improving the obstacle recognition capability of the open set recognition model and the closed set recognition model. The finally obtained closed set recognition model is used as an obstacle recognition model, improving the accuracy of the obstacle recognition model.

[0142] Figure 3is a flowchart of an obstacle recognition model training method provided by an embodiment of the present application. Embodiments of the present application are exemplarily described taking a computer device as an execution subject, and the embodiments include:

[0143] 301. The computer device obtains a sample set, which includes a plurality of sample images.

[0144] It should be noted that the sample set can be a labeled sample set or an unlabeled sample set, and the embodiments of the present application do not limit this.

[0145] 302. In the first iteration of the training process, the computer device identifies the plurality of sample images in the sample set by using a closed set recognition model.

[0146] The closed set recognition model in the above step 302 is a trained closed set recognition model, which can recognize M types of obstacles. M is any positive integer. The closed set recognition model can be trained by using a sample set labeled with M types of obstacles. The closed set recognition model can be a cascade model or an RCNN (Region Convolutional Neural Networks) model.

[0147] By using the closed set recognition model, the plurality of sample images in the sample set can be identified, and M types of obstacles in the sample images can be recognized, and the types of the obstacles can be determined.

[0148] 303. The computer device identifies the obstacle images of unknown types in the sample set by using an open set recognition model.

[0149] The open set recognition model in the above step 303 is a trained open set recognition model, which is trained by using a sample set labeled with M types of obstacles. Therefore, the open set recognition model can mine the obstacle of an unknown type in the sample set by relying on the M types of obstacles.

[0150] The open set recognition model can be an OpenDet model, which is composed of two learners. One learner is used to learn to extract more accurate image features, and the other learner is used to learn more accurate first and second thresholds. In some embodiments, the OpenDet model can be trained by using a random gradient descent optimization method, and the learning method with an initial learning rate of 0.001, a momentum of 0.9, and a weight decay coefficient of 0.00001 is used for training. The learning rate is reduced by 10 times at the 30,000th and 50,000th training, and the total training is 60,000 times.

[0151] 304. The computer device screens the obstacle images identified by the open set identification model to obtain obstacle images satisfying an obstacle condition, the obstacle condition being a condition that an object affects driving of the autonomous vehicle.

[0152] It should be noted that the open set identification model can identify both known-class obstacle images and unknown-class obstacle images, and the "obstacle images identified by the open set identification model" in the step 304 are only unknown-class obstacle images identified by the open set identification model.

[0153] According to the description in the step 202, the open set identification model determines an object image with a confidence less than the first threshold value and greater than the second threshold value as an unknown-class obstacle image. In this way, the determined obstacle images are not accurate enough, and the determined obstacle images can include non-obstacle images. Therefore, after identifying unknown-class obstacle images in the sample set by the open set identification model, the embodiment of the present application screens the identified obstacle images to eliminate unreliable detection results of unknown-class obstacle images.

[0154] In a possible implementation, the computer device screens the obstacle images identified by the open set identification model to obtain obstacle images satisfying an obstacle condition, including at least one of the following:

[0155] (1) The obstacle condition includes a distance threshold value, the distance threshold value indicating that an object with a distance less than the distance threshold value from a road is an obstacle affecting driving of the autonomous vehicle; the computer device determines a distance between an obstacle in the identified obstacle image and the road, and screens the identified obstacle image to obtain an obstacle image with a distance less than the distance threshold value.

[0156] It should be noted that only an object on a road surface or an object close to the road surface can affect driving of the autonomous vehicle and be regarded as an obstacle, and an object far from the road surface is an object high in the air and does not affect driving of the autonomous vehicle, and therefore, the object far from the road surface is not an obstacle. The computer device screens the identified obstacle image to obtain an obstacle image of an obstacle located on the road or close to the road.

[0157] The distance threshold value can be any distance, which can be an empirical value or a value set by a technician, and the embodiment of the present application does not limit the distance threshold value.

[0158] Before determining the distance between the identified obstacle and the road in the identified obstacle image, the road in the sample image needs to be identified. In some embodiments, determining the distance between the identified obstacle and the road in the identified obstacle image comprises: for any identified obstacle image, identifying a road image from the sample image to which the obstacle image belongs; and determining the distance between the obstacle indicated by the obstacle image and the road indicated by the road image in the sample image based on the positions of the obstacle image and the road image in the sample image.

[0159] In a possible implementation, the sample set further comprises point cloud data corresponding to the sample images, and the computer device can determine the road image in the sample image based on the point cloud data. The computer device identifies the road image from the sample image to which the obstacle image belongs, comprising: obtaining the point cloud data corresponding to the sample image, the sample image and the point cloud data corresponding to the same scene; obtaining points belonging to the road surface from the point cloud data based on the height of the points in the point cloud data; and projecting the obtained points to the sample image to obtain the road image in the sample image.

[0160] In the possible implementation, the road surface can be regarded as a plane, and the computer device can cluster the point cloud on the road surface based on a clustering algorithm to obtain the point cloud road segmentation result, that is, the points belonging to the road surface in the point cloud data; and project the points belonging to the road surface to the sample image to determine the concave polygon composed of the projection points to obtain the road image in the sample image.

[0161] In another possible implementation, the computer device identifies the road image in the sample image through a road segmentation model. The computer device identifies the road image from the sample image to which the obstacle image belongs, comprising: processing the sample image through the road segmentation model to obtain the road image in the sample image.

[0162] In some embodiments, the sample set comprises point cloud data corresponding to part of the sample images, as shown in FIG. 2, the computer device can perform road segmentation on the part of the sample images based on the point cloud data corresponding to the part of the sample images to obtain the road image in the part of the sample images, train the road segmentation model based on the road image in the part of the sample images, and subsequently directly use the road segmentation model to process the sample image to obtain the road image in the sample image. Figure 4

[0163] (2) The obstacle condition comprises a depth threshold, the depth threshold indicating the maximum image depth of an object affecting the driving of the autonomous vehicle in the driving environment image; and the obstacle image with the image depth greater than the depth threshold is filtered out from the identified obstacle image based on the image depth of the identified obstacle image.

[0164] ​It should be noted that the depth of the object in the sample image can represent the distance between the object and the autonomous vehicle, and the greater the depth of the object, the farther the distance between the object and the autonomous vehicle, and the smaller the influence on the driving of the autonomous vehicle; the smaller the depth of the object, the closer the distance between the object and the autonomous vehicle, and the greater the influence on the driving of the autonomous vehicle. Moreover, small objects far away are usually difficult to detect, and the determination of unknown obstacles far away has more errors, so in the embodiments of the present application, small objects far away are filtered out.

[0165] The depth threshold value can be any value, and optionally, the depth threshold value is an empirical value; optionally, the depth threshold value is a value set by the technician, and the embodiments of the present application do not limit the depth threshold value.

[0166] In some embodiments, in order to obtain the depth of the obstacle image, the computer device determines the depth of the obstacle image based on the point cloud data. Optionally, the method further comprises: for any identified obstacle image, obtaining point cloud data based on the sample image to which the obstacle image belongs, the sample image and the point cloud data corresponding to the same scene; mapping the points in the point cloud data to the sample image, determining the depth map of the sample image based on the depth of the points in the point cloud data, and determining the image depth of the obstacle image in the sample image based on the depth map of the sample image.

[0167] Wherein, after the computer device maps the points in the point cloud data to the sample image, a traditional depth completion method (for example, the IP-Basic method) can be used to obtain the depth map of the sample image, and the depth map is a pixel-level depth map.

[0168] In other embodiments, without determining the depth map of the sample image, the image depth of the obstacle image can be directly determined based on the depth of the point cloud data corresponding to the obstacle image. The method further comprises: for any identified obstacle image, obtaining point cloud data based on the sample image to which the obstacle image belongs, the sample image and the point cloud data corresponding to the same scene; mapping the point cloud in the point cloud data to the sample image, and determining the image depth of the obstacle image based on the depth of the point cloud data corresponding to the obstacle image in the sample image.

[0169] In other embodiments, the computer device can also determine the image depth of the obstacle image based on a depth determination model. The method further comprises: for any identified obstacle image, determining the image depth of the obstacle image through the depth determination model.

[0170] Optionally, the sample set includes point cloud data corresponding to part of the sample images, such as Figure 4As shown, the computer device can determine the depth map of the sample image based on the point cloud data corresponding to the sample image, train a depth determination model based on the depth map of the sample image, and then directly determine the image depth of the obstacle image based on the depth determination model.

[0171] (3) The obstacle condition includes a size threshold, which indicates the minimum image size of the object affecting the driving of the autonomous vehicle in the driving environment image; based on the image size of the identified obstacle images, obstacle images with an image size greater than the size threshold are selected from the identified obstacle images.

[0172] It should be noted that if the image size of an obstacle is small, it may be because the obstacle itself is small or because the obstacle is far away from the autonomous vehicle. Regardless of whether the obstacle itself is small or the obstacle is far away from the autonomous vehicle, it is unlikely to affect the autonomous vehicle's driving. Therefore, obstacle images with small image sizes can be filtered out without affecting the autonomous vehicle's driving.

[0173] The size threshold can be any value. Optionally, the size threshold is an empirical value; alternatively, the size threshold is a value set by a technician. This application embodiment does not limit the size threshold.

[0174] 305. The computer equipment labels the sample set based on the recognition results of the closed set recognition model and the obstacle images that meet the obstacle conditions.

[0175] The closed-set recognition model identifies M types of known obstacles with annotation information, while images of obstacles meeting the obstacle conditions identify 1 type of unknown obstacles. Based on the closed-set recognition model's results and the images of obstacles meeting the obstacle conditions, the computer device annotates the sample set, resulting in a sample set annotated with M+1 types of obstacles.

[0176] In some embodiments, the sample set also includes point cloud data corresponding to some sample images. To improve the recall rate of unknown category obstacles, embodiments of this application can further supplement the unknown category obstacles based on the point cloud data. Figure 4 As shown, the method further includes: acquiring point cloud data corresponding to the sample image, wherein the sample image and the point cloud data correspond to the same scene; clustering the points in the point cloud data whose distance from the road does not exceed a distance threshold to obtain multiple point clusters; mapping the multiple point clusters to the sample image, wherein if the object image corresponding to any point cluster in the sample image is not an obstacle image identified by the open set recognition model, then the object image is identified as an obstacle image and labeled as an unknown category.

[0177] It should be noted that the obstacle image identified by the open set identification model includes the obstacle image of the unknown category, and also includes the obstacle image of the M known categories. The object image corresponding to the point cluster in the sample image is not the obstacle image identified by the open set identification model, which means that the object image corresponding to the point cluster in the sample image is neither the unknown category obstacle image identified by the open set identification model nor the M known category obstacle image identified by the open set identification model.

[0178] The points in the point cloud data that are not more than the distance threshold away from the road are clustered, and then the point clusters obtained by clustering are framed by the obstacle pre-frames. Subsequently, the pre-frames are compared with the pre-frames obtained by the open set identification model, and the overlapping pre-frames are filtered out.

[0179] 306、The computer device trains the closed set identification model based on the labeled sample set to obtain the closed set identification model corresponding to the first iteration.

[0180] The labeled sample set includes M known category obstacles and 1 unknown category obstacle. The closed set identification model is trained based on the labeled sample set to obtain the closed set identification model corresponding to the first iteration, which is a model for identifying M+1 known category obstacles.

[0181] 307、The computer device identifies the plurality of sample images in the sample set by the closed set identification model corresponding to the first iteration, labels the plurality of sample images in the sample set based on the identification result to obtain the sample set corresponding to the first iteration, and trains the open set identification model based on the sample set corresponding to the first iteration to obtain the open set identification model corresponding to the first iteration.

[0182] Although the open set identification model mines the 1 unknown category obstacle, the same category obstacle in the sample set may not be mined. For example, the open set identification model mines the tricycle as an unknown category obstacle, but the sample set may include part of the tricycle image that is not labeled. In order to improve the accuracy of the sample set, the sample set is labeled by the closed set identification model corresponding to the first iteration. Since the closed set identification model corresponding to the first iteration can identify M+1 known category obstacles, the M+1 known category obstacles in the sample set can be accurately identified by the closed set identification model corresponding to the first iteration.

[0183] In some embodiments, the training process of the closed set recognition model generally has the problem of class imbalance, and the open set recognition model has the problem of detection error. In order to alleviate the above problems, the computer device will also resample the unknown class obstacles. That is, from the unknown class obstacles mined from the sample set, select unknown class obstacle images with high credibility, reset the obstacles according to various methods such as random placement, straight line placement, or radial dense placement, and at the same time, use the label information of these unknown class obstacle images as training supervision information. Because the number of unknown class obstacle images is large and the combination method is various, the above method can achieve the effect of sufficient data enhancement.

[0184] In a possible implementation, the computer device labels a plurality of sample images in the sample set based on the recognition result to obtain a sample set corresponding to the first iteration, including: in the sample set, labeling the recognized obstacle images based on the categories to which the recognized obstacle images belong; from the recognized obstacle images, filtering out obstacle images that meet a confidence condition and belong to an unknown category; and resetting the filtered obstacle images according to a target manner to obtain the sample set corresponding to the first iteration.

[0185] For example, based on the confidence of the obstacle image, the obstacle images are clustered, and the obstacle images far from the cluster center are filtered out. Then, in the cluster, the target is sampled uniformly, according to the sample appearance probability, the Poisson fusion scheme is used for target placement, the natural expansion of the visual effect is achieved, and the sample set is used to train the model to ensure the accuracy and recall rate of the unknown obstacle prediction.

[0186] 308. In the Nth iteration process of the training process, the computer device identifies the obstacle images of an unknown category in the sample set corresponding to the (N-1)th iteration through the open set recognition model corresponding to the (N-1)th iteration, and labels the identified obstacle images as an unknown category. N is a positive integer greater than 1.

[0187] The above step 308 can be the second iteration process, the third iteration process, and the last iteration process of the training process.

[0188] In a possible implementation, the computer device labels the identified obstacle images as an unknown category, including: the computer device filters out obstacle images that meet an obstacle condition from the identified obstacle images, the obstacle condition being a condition in which an object affects the driving of an autonomous vehicle; and labels the filtered obstacle images as an unknown category.

[0189] The step 308 is the same as the above steps 303 and 304, and will not be described one by one here.

[0190] 309. The computer device trains the closed set recognition model based on the labeled sample set to obtain a closed set recognition model corresponding to the Nth iteration.

[0191] The step 309 is the same as the step 306 described above, and thus will not be repeated here.

[0192] 310. If the number of iterations of the training process reaches the target number, the computer device outputs the closed set recognition model corresponding to the Nth iteration as the obstacle recognition model.

[0193] The target number can be any number, for example, 5, 10, 20, 100, etc. The embodiments of the present application do not limit the target number.

[0194] 311. If the number of iterations of the training process does not reach the target number, the computer device identifies the plurality of sample images in the sample set based on the closed set recognition model corresponding to the Nth iteration, labels the plurality of sample images in the sample set based on the identification result, obtains a sample set corresponding to the Nth iteration, trains the open set recognition model corresponding to the N-1th iteration based on the sample set corresponding to the Nth iteration, and obtains an open set recognition model corresponding to the Nth iteration.

[0195] The step 311 is the same as the step 307 described above, and thus will not be repeated here.

[0196] The embodiments of the present application repeatedly perform the steps 308, 309 and 311 described above until the number of iterations of the training process reaches the target number, and then perform the step 310 to end.

[0197] The embodiments of the present application obtain reliable unknown-class obstacle detection results through the open set recognition model, mark these unknown-class obstacles as unknown classes, and then fuse the original M known-class obstacles to train an M+1 known-class obstacle closed set recognition model, so that the closed set recognition model can detect more unknown-class obstacles while ensuring the detection performance of the unknown-class obstacles. Based on the detection result of the trained known-class obstacle closed set recognition model, the open set recognition model is trained to mine new unknown-class obstacles. Through the repeated execution of the above steps, an N+1 known-class obstacle closed set recognition model and an unknown-class obstacle open set recognition model can be obtained.

[0198] The obstacle recognition model training method provided in the embodiments of the present application mines obstacle images of new categories in the sample set through the open set recognition model, iteratively trains the open set recognition model and the closed set recognition model, so that the open set recognition model and the closed set recognition model can recognize more categories of obstacle images, improves the obstacle recognition capability of the open set recognition model and the closed set recognition model, and finally obtains the closed set recognition model as the obstacle recognition model, thereby improving the accuracy of the obstacle recognition model.

[0199] It should be noted that, in the ECCV 2022 SSLAD Track 3: Corner Case Detection (which is an international competition for obstacle detection in an autonomous driving scene using large-scale unlabeled data and a small amount of labeled data), the method based on the embodiments of the present application won first place in the competition, and the overall index was much higher than that of the second place.

[0200] It should be noted that the obstacle recognition model training method provided in the embodiments of the present application mines obstacle images of new categories in the sample set through the open set recognition model, iteratively trains the open set recognition model and the closed set recognition model, mines more categories of obstacle images from the sample set, and trains the closed set recognition model through the sample set of more categories of obstacle images to obtain the obstacle recognition model. As can be seen, by combining the open set recognition model and the closed set recognition model, an accurate recognition result can be obtained. Therefore, in actual application, the obstacle recognition model trained by the obstacle recognition model training method shown in Figure 3 may be identified by the open set recognition model and the closed set recognition model. The embodiments of the present application exemplarily illustrate "obstacle recognition through the open set recognition model and the closed set recognition model" through the embodiments shown in Figure 5 .

[0201] Figure 5 is a flowchart of an obstacle recognition method provided in the embodiments of the present application. The embodiments of the present application exemplarily illustrate the case that the execution subject is an autonomous vehicle. Referring to Figure 5 , the embodiments include:

[0202] 501. The autonomous vehicle obtains an image of a target scene.

[0203] The image of the target scene can be any image taken during the driving process of the autonomous vehicle, and the target scene is not limited in the embodiments of the present application. It should be noted that the embodiments of the present application only exemplarily illustrate the case that the execution subject is an autonomous vehicle, and therefore, when the execution subject is other equipment, the target scene can also be other scenes, and the target scene is not limited in the embodiments of the present application, and can be any scene.

[0204] 502、The automatic driving vehicle identifies the image by a closed set identification model to obtain a first identification result, the closed set identification model being used to identify a learned known-class obstacle image, and the first identification result including the identified obstacle image and a class of the obstacle image.

[0205] The step 502 is the same as the step 302, and thus is not described herein.

[0206] 503、The automatic driving vehicle identifies the image by an open set identification model to obtain a second identification result, the open set identification model being used to identify an unknown-class obstacle image based on a learned known-class obstacle image, and the second identification result including the identified obstacle image.

[0207] The step 503 is the same as the step 303, and thus is not described herein.

[0208] 504、The automatic driving vehicle fuses the first identification result and the second identification result to obtain a third identification result.

[0209] The obstacle images identified by the closed set identification model are not comprehensive, but the accuracy of the identification result is high; and the obstacle images identified by the open set identification model are comprehensive, but the accuracy of the identification result is low. Therefore, the embodiments of the present application identify the same image by the closed set identification model and the open set identification model, fuse the two identification results as the final identification result, and improve the accuracy of the identification result.

[0210] In a possible implementation, the automatic driving vehicle fuses the first identification result and the second identification result to obtain a third identification result, including: determining, by the automatic driving vehicle, an obstacle image same as the obstacle image in the first identification result in the second identification result, and obtaining a confidence of the determined obstacle image; screening, by the automatic driving vehicle, the obstacle image from the second identification result based on the confidence; and fusing, by the automatic driving vehicle, the screened obstacle image and the first identification result to obtain the third identification result.

[0211] The screening of the obstacle image from the second identification result based on the confidence can be screening of an obstacle image with a confidence not less than the confidence from the second identification result; or screening of an obstacle image with a confidence not less than a confidence threshold from the second identification result, the confidence threshold being determined based on the confidence and can be less than the confidence.

[0212] In another possible implementation, the automatic driving vehicle fuses the first recognition result and the second recognition result to obtain a third recognition result, including: the automatic driving vehicle screens obstacle images that meet an obstacle condition from the second recognition result to obtain a fourth recognition result, the obstacle condition being a condition under which an object affects driving of the automatic driving vehicle; and the automatic driving vehicle fuses the first recognition result and the fourth recognition result to obtain the third recognition result.

[0213] Optionally, the automatic driving vehicle screens the obstacle images that meet the obstacle condition from the second recognition result, including at least one of the following: (1) the obstacle condition includes a distance threshold, the distance threshold indicating that an object with a distance less than the distance threshold from a road is an obstacle affecting driving of the automatic driving vehicle; based on the second recognition result, the automatic driving vehicle determines distances between the recognized obstacle images and the road, and screens the obstacle images with distances less than the distance threshold from the second recognition result; (2) the obstacle condition includes a depth threshold, the depth threshold indicating that an object affecting driving of the automatic driving vehicle has a maximum image depth in a driving environment image; based on the second recognition result, the automatic driving vehicle determines image depths of the recognized obstacle images, and screens the obstacle images with image depths greater than the depth threshold from the second recognition result; and (3) the obstacle condition includes a size threshold, the size threshold indicating that an object affecting driving of the automatic driving vehicle has a minimum image size in the driving environment image; based on image sizes of the obstacle images in the second recognition result, the automatic driving vehicle screens the obstacle images with image sizes greater than the size threshold from the second recognition result.

[0214] Optionally, the automatic driving vehicle determines distances between the recognized obstacle images and the road based on the second recognition result, including: for any obstacle image in the second recognition result, the automatic driving vehicle identifies a road image from an image to which the obstacle image belongs; and based on positions of the obstacle image and the road image in the image, the automatic driving vehicle determines a distance between an obstacle indicated by the obstacle image and a road indicated by the road image in the image.

[0215] Optionally, the automatic driving vehicle identifies the road image from the image to which the obstacle image belongs, including: obtaining point cloud data corresponding to the image, the image and the point cloud data corresponding to a same scene; based on heights of points in the point cloud data, the automatic driving vehicle obtains points belonging to a road surface from the point cloud data; and the automatic driving vehicle projects the obtained points to the image to obtain the road image in the image; or the automatic driving vehicle processes the image through a road segmentation model to obtain the road image in the image.

[0216] Optionally, the method further comprises: for any obstacle image in the second recognition result, obtaining point cloud data based on an image to which the obstacle image belongs, the image and the point cloud data corresponding to the same scene; mapping points in the point cloud data to the image, determining a depth map of the image based on depths of the points in the point cloud data, and determining an image depth of the obstacle image in the image based on the depth map of the image; or for any obstacle image in the second recognition result, obtaining point cloud data based on an image to which the obstacle image belongs, the image and the point cloud data corresponding to the same scene; mapping points in the point cloud data to the image, determining an image depth of the obstacle image based on depths of the point cloud data corresponding to the obstacle image in the image; or for any obstacle image in the second recognition result, determining an image depth of the obstacle image by using a depth determination model.

[0217] The step of obtaining the fourth recognition result can refer to the step 304, which will not be described here.

[0218] In another possible manner, the third recognition result can also be obtained based on the point cloud data. Optionally, the third recognition result is obtained by fusing the first recognition result and the second recognition result, comprising: obtaining point cloud data of the target scene; clustering points in the point cloud data that are not more than a distance threshold away from the road to obtain a plurality of point clusters; mapping the plurality of point clusters to the sample images to obtain object images corresponding to the plurality of point clusters respectively; determining a fifth recognition result based on the object images corresponding to the plurality of point clusters respectively and the second recognition result, the fifth recognition result comprising object images in the object images corresponding to the plurality of point clusters respectively that do not belong to the second recognition result; and fusing the first recognition result, the second recognition result and the fifth recognition result to obtain the third recognition result.

[0219] Figure 6 is a structural schematic diagram of a kind of obstacle recognition model training device provided by the embodiment of the application, refer to Figure 6 The device comprises:

[0220] The obtaining module 601 is configured to obtain a sample set, wherein the sample set comprises a plurality of sample images.

[0221] The labeling module 602 is configured to, in the Nth iteration process of the training process, identify obstacle images of unknown categories in a sample set corresponding to the (N-1) th iteration by using an open set recognition model corresponding to the (N-1) th iteration, and label the identified obstacle images as unknown categories, wherein the open set recognition model is used to identify obstacle images of unknown categories based on learned obstacle images of known categories, and N is a positive integer greater than 1.

[0222] The training module 603 is configured to train the closed set recognition model based on the labeled sample set, to obtain a closed set recognition model corresponding to the Nth iteration, the closed set recognition model being used to identify the obstacle image of the known category.

[0223] The output module 604 is configured to output the closed set recognition model corresponding to the Nth iteration as the obstacle recognition model if the training process reaches the loop termination condition.

[0224] The training module 603 is further configured to, if the training process does not reach the loop termination condition, identify the plurality of sample images in the sample set by using the closed set recognition model corresponding to the Nth iteration, label the plurality of sample images in the sample set based on the identification result, obtain a sample set corresponding to the Nth iteration, train the open set recognition model corresponding to the (N-1)th iteration based on the sample set corresponding to the Nth iteration, and obtain the open set recognition model corresponding to the Nth iteration.

[0225] As shown in Figure 7 In a possible implementation, the labeling module 602 includes:

[0226] The screening unit 6021 is configured to screen the obstacle images meeting the obstacle condition from the identified obstacle images, the obstacle condition being a condition that an object affects the driving of the autonomous vehicle.

[0227] The labeling unit 6022 is configured to label the screened obstacle images as the unknown category.

[0228] In a possible implementation, the screening unit 6021 is configured to at least one of the following at least one:

[0229] The obstacle condition includes a distance threshold, the distance threshold indicating that an object with a distance less than the distance threshold from the road is an obstacle affecting the driving of the autonomous vehicle; the distance of the obstacle from the road in the identified obstacle image is determined, and the obstacle image with a distance less than the distance threshold is screened from the identified obstacle image.

[0230] The obstacle condition includes a depth threshold, the depth threshold indicating that an object affecting the driving of the autonomous vehicle has a maximum image depth in the driving environment image; the image depth of the identified obstacle image is determined, and the obstacle image with an image depth greater than the depth threshold is screened from the identified obstacle image.

[0231] The obstacle condition includes a size threshold indicating a minimum image size of an object affecting driving of the autonomous vehicle in the driving environment image; and the screening unit 6021 is configured to screen, from the identified obstacle images, an obstacle image with an image size greater than the size threshold based on an image size of the identified obstacle image.

[0232] In a possible implementation, the screening unit 6021 is configured to, for any identified obstacle image, identify a road image from a sample image to which the obstacle image belongs; and determine a distance between an obstacle indicated by the obstacle image and a road indicated by the road image in the sample image based on positions of the obstacle image and the road image in the sample image.

[0233] In a possible implementation, the screening unit 6021 is configured to obtain point cloud data corresponding to the sample image, the sample image and the point cloud data corresponding to a same scene; obtain, from the point cloud data, points belonging to a road surface based on heights of points in the point cloud data; and project the obtained points into the sample image to obtain a road image in the sample image; or

[0234] The screening unit 6021 is configured to obtain a road image in the sample image by processing the sample image through a road segmentation model.

[0235] In a possible implementation, the apparatus further includes:

[0236] The determining module 605 is configured to, for any identified obstacle image, obtain point cloud data based on a sample image to which the obstacle image belongs, the sample image and the point cloud data corresponding to a same scene; map points in the point cloud data to the sample image, determine a depth map of the sample image based on depths of points in the point cloud data, and determine an image depth of the obstacle image in the sample image based on the depth map of the sample image; or

[0237] The determining module 605 is configured to, for any identified obstacle image, obtain point cloud data based on a sample image to which the obstacle image belongs, the sample image and the point cloud data corresponding to a same scene; map points in the point cloud data to the sample image, determine an image depth of the obstacle image based on a depth of point cloud data corresponding to the obstacle image in the sample image; or

[0238] The determining module 605 is configured to, for any identified obstacle image, determine an image depth of the obstacle image by a depth determination model.

[0239] In a possible implementation, the apparatus further includes:

[0240] The acquisition module 601 is further configured to acquire point cloud data corresponding to the sample image in each iteration process, the sample image and the point cloud data corresponding to the same scene.

[0241] The clustering module 606 is configured to cluster points in the point cloud data that are not more than a distance threshold from a road to obtain a plurality of point clusters.

[0242] The labeling module 602 is further configured to map the plurality of point clusters to the sample image, and if an object image corresponding to any point cluster in the sample image is not an obstacle image identified by the open set recognition model, determine the object image as an obstacle image and label it as an unknown class.

[0243] In a possible implementation, the training module 603 is configured to, in the sample set, label the identified obstacle image based on a class to which the identified obstacle image belongs; filter, from the identified obstacle image, an obstacle image that meets a confidence condition and belongs to the unknown class; and reset a position of the filtered obstacle image in a target manner to obtain a sample set corresponding to the Nth iteration.

[0244] In a possible implementation, the output module 604 is configured to, if an iteration process of the training process reaches a target number of times, output the Nth iteration corresponding closed set recognition model as a target closed set recognition model.

[0245] It should be noted that: the obstacle recognition model training apparatus provided in the above embodiments is only used for example to divide the above functional modules in training the obstacle recognition model, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the computer device is divided into different functional modules to complete all or part of the functions described above. In addition, the obstacle recognition model training apparatus and the obstacle recognition model training method provided in the above embodiments belong to the same concept, and the specific implementation process is described in detail in the method embodiments, which will not be repeated here.

[0246] Figure 8 is a structural schematic diagram of an obstacle recognition apparatus provided in the present embodiment, referring to Figure 8 The apparatus includes:

[0247] The acquisition module 801 is configured to acquire an image of a target scene.

[0248] The first recognition module 802 is used to recognize the image through a closed-set recognition model to obtain a first recognition result. The closed-set recognition model is used to recognize obstacle images of known categories learned from the model. The first recognition result includes the recognized obstacle image and the category of the obstacle image.

[0249] The second recognition module 803 is used to recognize the image through an open set recognition model to obtain a second recognition result. The open set recognition model is used to recognize an unknown category of obstacle image based on the learned known category of obstacle image. The second recognition result includes the recognized obstacle image.

[0250] The fusion module 804 is used to fuse the first recognition result and the second recognition result to obtain the third recognition result.

[0251] like Figure 9 As shown, in one possible implementation, the fusion module 804 includes:

[0252] The filtering unit 8041 is used to filter out obstacle images that meet the obstacle conditions from the second recognition result to obtain the fourth recognition result. The obstacle conditions are the conditions under which an object affects the driving of an autonomous vehicle.

[0253] The fusion unit 8042 is used to fuse the first recognition result and the fourth recognition result to obtain the third recognition result.

[0254] In one possible implementation, the obstacle condition includes a distance threshold, which indicates that objects whose distance from the road is less than the distance threshold are obstacles that affect the driving of the autonomous vehicle; the filtering unit 8041 is used to determine the distance between the obstacle and the road in the identified obstacle image based on the second recognition result, and filter out obstacle images whose distance is less than the distance threshold from the second recognition result.

[0255] The obstacle conditions include a depth threshold, which indicates the maximum image depth of an object affecting the driving of an autonomous vehicle in the driving environment image; the filtering unit 8041 is used to determine the image depth of the identified obstacle image based on the second recognition result, and filter out obstacle images with an image depth greater than the depth threshold from the second recognition result;

[0256] The obstacle conditions include a size threshold, which indicates the minimum image size of an object affecting the driving of an autonomous vehicle in the driving environment image; the filtering unit 8041 is used to filter out obstacle images whose image size is greater than the size threshold from the second recognition result based on the image size of the obstacle images in the second recognition result.

[0257] In a possible implementation, the screening unit 8041 is configured to, for any obstacle image in the second recognition result, identify a road image from an image to which the obstacle image belongs; and determine a distance between an obstacle indicated by the obstacle image and a road indicated by the road image in the image based on positions of the obstacle image and the road image in the image.

[0258] In a possible implementation, the screening unit 8041 is configured to obtain point cloud data corresponding to the image, the image and the point cloud data corresponding to a same scene; obtain, from the point cloud data, points belonging to a road surface based on heights of points in the point cloud data; project the obtained points into the image to obtain the road image in the image; or,

[0259] The screening unit 8041 is configured to process the image by using a road segmentation model to obtain the road image in the image.

[0260] In a possible implementation, the screening unit 8041 is configured to, for any obstacle image in the second recognition result, obtain point cloud data based on an image to which the obstacle image belongs, the image and the point cloud data corresponding to a same scene; map points in the point cloud data to the image, determine a depth map of the image based on depths of the points in the point cloud data, and determine an image depth of the obstacle image in the image based on the depth map of the image; or,

[0261] The screening unit 8041 is configured to, for any obstacle image in the second recognition result, obtain point cloud data based on an image to which the obstacle image belongs, the image and the point cloud data corresponding to a same scene; map points in the point cloud data to the image, and determine an image depth of the obstacle image based on depths of the point cloud data corresponding to the obstacle image in the image; or,

[0262] The screening unit 8041 is configured to, for any obstacle image in the second recognition result, determine an image depth of the obstacle image by using a depth determination model.

[0263] In a possible implementation, the fusion module 804 is configured to obtain point cloud data of a target scene; cluster points in the point cloud data that are at a distance of not more than a distance threshold from a road to obtain a plurality of point clusters; map the plurality of point clusters to a sample image to obtain object images respectively corresponding to the plurality of point clusters; determine a fifth recognition result based on the object images respectively corresponding to the plurality of point clusters and the second recognition result, the fifth recognition result including object images that do not belong to the second recognition result in the object images respectively corresponding to the plurality of point clusters; and fuse the first recognition result, the second recognition result, and the fifth recognition result to obtain a third recognition result.

[0264] It should be noted that the obstacle identification device provided in the above embodiment is only used for illustrating the division of the above functional modules when identifying the obstacle, and in actual application, the above functions can be completed by different functional modules according to the needs, that is, the internal structure of the autonomous vehicle is divided into different functional modules to complete all or part of the functions described above. In addition, the obstacle identification device and the obstacle identification method provided in the above embodiment belong to the same concept, and the specific implementation process is described in the method embodiment, which will not be repeated here.

[0265] Figure 10 is a structural block diagram of a terminal 1000 provided by an embodiment of the present application. The terminal 1000 comprises a processor 1001 and a memory 1002.

[0266] The processor 1001 can include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1001 can be implemented in at least one of a hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), a PLA (Programmable Logic Array). The processor 1001 can also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a CPU (Central Processing Unit). The coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 1001 can be integrated with a GPU (Graphics Processing Unit). The GPU is responsible for rendering and drawing the content required to be displayed on the display screen. In some embodiments, the processor 1001 can also include an AI (Artificial Intelligence) processor. The AI processor is used to process machine learning related computing operations.

[0267] The memory 1002 can include one or more computer-readable storage media, which can be non-transitory. The memory 1002 can also include a high-speed random access memory and a non-volatile memory such as one or more disk storage devices, flash storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1002 is used to store at least one program code for being executed by the processor 1001 to implement the obstacle identification model training method or the obstacle identification method provided by the method embodiment in the present application.

[0268] In some embodiments, the terminal 1000 can further optionally include a peripheral device interface 1003 and at least one peripheral device. The processor 1001, the memory 1002 and the peripheral device interface 1003 can be connected through a bus or a signal line. Each peripheral device can be connected with the peripheral device interface 1003 through a bus, a signal line or a circuit board. Specifically, the peripheral device includes at least one of a radio frequency circuit 1004, a display screen 1005, a camera 1006 and a power supply 1007.

[0269] Those skilled in the art can understand that the structure shown in the above embodiments is not a limitation on the terminal 1000, and the terminal 1000 can include more or fewer components than those shown in the figure, or combine certain components, or adopt a different component arrangement. Figure 10

[0270] Figure 11 The server 1100 provided by the embodiments of the present application can have a large difference due to different configurations or performances, and can include one or more processors (Central Processing Units, CPUs) 1101 and one or more memories 1102. The memory 1102 stores at least one program code, which is loaded and executed by the processor 1101 to implement the method provided by each method embodiment described above. Of course, the server can also have a wired or wireless network interface, a keyboard, an input and output interface and other components for realizing the functions of the device, and will not be described here.

[0271] The server 1100 is configured to execute the steps performed by the server in the above method embodiments.

[0272] In the exemplary embodiments, a computer readable storage medium, such as a memory including program code, is also provided. The program code can be executed by a processor in a computer device to complete the obstacle identification model training method or the obstacle identification method in the above embodiments. For example, the computer readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk and an optical data storage device, etc.

[0273] In the exemplary embodiments, a computer program or a computer program product is also provided. The computer program or the computer program product includes computer program code, which, when executed by a computer, causes the computer to implement the obstacle identification model training method or the obstacle identification method in the above embodiments.

[0274] ​Those skilled in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by hardware, or can be instructed to relevant hardware by program. The program can be stored in a computer readable storage medium, and the storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0275] The above only describes optional embodiments of the present application and is not used to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An obstacle recognition method, characterized in that, The method includes: Acquire images of the target scene; The image is identified using a closed-set recognition model to obtain a first recognition result. The closed-set recognition model is used to identify obstacle images of known categories that have been learned. The first recognition result includes the identified obstacle image and the category of the obstacle image. The image is identified using an open set recognition model to obtain a second recognition result. The open set recognition model is used to identify obstacle images of unknown categories based on learned obstacle images of known categories. The second recognition result includes the identified obstacle images. Obtain point cloud data of the target scene; The points in the point cloud data whose distance from the road does not exceed a distance threshold are clustered to obtain multiple point clusters; The multiple point clusters are mapped onto the sample image to obtain the object images corresponding to the multiple point clusters respectively; Based on the object images corresponding to the plurality of point clusters and the second recognition result, a fifth recognition result is determined. The fifth recognition result includes object images that do not belong to the second recognition result among the object images corresponding to the plurality of point clusters. The first identification result, the second identification result, and the fifth identification result are fused together to obtain the third identification result.

2. A method for training an obstacle recognition model, characterized in that, The method includes: Obtain a sample set, which includes multiple sample images; During the Nth iteration of the training process, the open set recognition model corresponding to the N-1th iteration identifies obstacle images of unknown categories in the sample set corresponding to the N-1th iteration, and labels the identified obstacle images as unknown categories. The open set recognition model is used to identify obstacle images of unknown categories based on obstacle images of known categories that have been learned. N is a positive integer greater than 1. A closed-set recognition model is trained based on the labeled sample set to obtain the closed-set recognition model corresponding to the Nth iteration. The closed-set recognition model is used to recognize obstacle images of known categories. If the training process reaches the loop termination condition, the closed set recognition model corresponding to the Nth iteration will be output as an obstacle recognition model. If the training process does not reach the loop termination condition, the multiple sample images in the sample set are identified using the closed-set recognition model corresponding to the Nth iteration. Based on the recognition results, the multiple sample images in the sample set are labeled to obtain the sample set corresponding to the Nth iteration. The open-set recognition model corresponding to the N-1th iteration is trained based on the sample set corresponding to the Nth iteration to obtain the open-set recognition model corresponding to the Nth iteration. During any iteration, the point cloud data corresponding to the sample image is acquired, and the sample image and the point cloud data correspond to the same scene. The points in the point cloud data whose distance from the road does not exceed a distance threshold are clustered to obtain multiple point clusters; The multiple point clusters are mapped onto the sample image. If the object image corresponding to any point cluster in the sample image is not an obstacle image identified by the open set recognition model, then the object image is determined to be an obstacle image and labeled as an unknown category.

3. An obstacle recognition device, characterized in that, The device includes: The acquisition module is used to acquire images of the target scene; The first recognition module is used to recognize the image through a closed-set recognition model to obtain a first recognition result. The closed-set recognition model is used to recognize obstacle images of known categories learned from the model. The first recognition result includes the recognized obstacle image and the category of the obstacle image. The second recognition module is used to recognize the image through an open set recognition model to obtain a second recognition result. The open set recognition model is used to recognize obstacle images of unknown categories based on learned obstacle images of known categories. The second recognition result includes the recognized obstacle images. A fusion module is used to acquire point cloud data of the target scene; The points in the point cloud data whose distance from the road does not exceed a distance threshold are clustered to obtain multiple point clusters; The multiple point clusters are mapped onto the sample image to obtain the object images corresponding to the multiple point clusters respectively; Based on the object images corresponding to the plurality of point clusters and the second recognition result, a fifth recognition result is determined. The fifth recognition result includes object images that do not belong to the second recognition result among the object images corresponding to the plurality of point clusters. The first identification result, the second identification result, and the fifth identification result are fused together to obtain the third identification result.

4. An obstacle recognition model training device, characterized in that, The device includes: The acquisition module is used to acquire a sample set, which includes multiple sample images; The annotation module is used to identify obstacle images of unknown category in the sample set corresponding to the N-1th iteration during the Nth iteration of the training process, through the open set recognition model corresponding to the N-1th iteration, and to label the identified obstacle images as unknown category. The open set recognition model is used to identify obstacle images of unknown category based on the obstacle images of known category learned, where N is a positive integer greater than 1. The training module is used to train a closed-set recognition model based on the labeled sample set to obtain the closed-set recognition model corresponding to the Nth iteration. The closed-set recognition model is used to recognize obstacle images of known categories. The output module is used to output the closed set recognition model corresponding to the Nth iteration as an obstacle recognition model if the training process reaches the loop termination condition. The training module is further configured to, if the training process does not reach the loop termination condition, identify the plurality of sample images in the sample set using the closed-set recognition model corresponding to the Nth iteration, label the plurality of sample images in the sample set based on the recognition results to obtain the sample set corresponding to the Nth iteration, and train the open-set recognition model corresponding to the (N-1)th iteration based on the sample set corresponding to the Nth iteration to obtain the open-set recognition model corresponding to the Nth iteration; during any iteration, acquire the point cloud data corresponding to the sample image, wherein the sample image and the point cloud data correspond to the same scene; The points in the point cloud data whose distance from the road does not exceed a distance threshold are clustered to obtain multiple point clusters; The multiple point clusters are mapped onto the sample image. If the object image corresponding to any point cluster in the sample image is not an obstacle image identified by the open set recognition model, then the object image is determined to be an obstacle image and labeled as an unknown category.

5. A computer device, characterized in that, The computer device includes one or more processors and one or more memories, wherein at least one piece of program code is stored in the one or more memories, and the at least one piece of program code is loaded and executed by the one or more processors to implement the operation performed by the obstacle recognition method as described in claim 1, or to implement the operation performed by the obstacle recognition model training method as described in claim 2.

6. A computer-readable storage medium, characterized in that, The storage medium stores at least one piece of program code, which is loaded and executed by a processor to implement the operation performed by the obstacle recognition method as described in claim 1, or to implement the operation performed by the obstacle recognition model training method as described in claim 2.

Images