Obstacle detection method and device for autonomous vehicle, electronic equipment, storage medium and program product

Abstract

The invention discloses an obstacle detection method and device for an automatic driving vehicle, electronic equipment, a storage medium and a program product, and belongs to the technical field of automatic driving. The obstacle detection method for the autonomous vehicle comprises the following steps: acquiring three-dimensional geometric information of a target obstacle in a vehicle driving environment; based on the three-dimensional geometric information and calibration parameters of the at least one camera, determining a plurality of image areas corresponding to the target obstacle in an image acquired by the camera; inputting a plurality of image areas corresponding to the target obstacle and a plurality of candidate types of texts into a trained visual language model, and obtaining semantic similarity between each image area output by the visual language model and each candidate type of text; and determining type information of the target obstacle based on the semantic similarity corresponding to each image area. According to the obstacle detection method of the automatic driving vehicle, the accuracy and robustness of semantic recognition of the automatic driving system on universal obstacles are improved.

Description

Obstacle detection methods, devices, electronic equipment, storage media, and software products for autonomous vehicles. Technical Field

[0001] This application belongs to the field of autonomous driving technology, and in particular relates to an obstacle detection method, device, electronic device, storage medium and program product for autonomous vehicles. Background Technology

[0002] During autonomous driving, it is necessary to detect general obstacles (in autonomous driving, this refers to general obstacles other than dynamic obstacles, such as vehicles and pedestrians, or all obstacles that affect passage). These obstacles are complex in shape, randomly distributed, and dynamically changing. Related technologies can only provide semantic types for a limited number of general obstacles. For some general obstacles in autonomous driving scenarios, related technologies cannot identify the semantic categories of such obstacles, making them unsuitable for downstream prediction or planning control scenarios and limiting their applicability. Summary of the Invention

[0003] This invention aims to address at least one of the technical problems existing in the prior art. To this end, this invention proposes an obstacle detection method, device, electronic device, storage medium, and program product for autonomous vehicles. It achieves open set detection capabilities that can dynamically expand the recognition types without retraining the model, improving the accuracy and robustness of semantic recognition of common obstacles in autonomous driving systems. It is applicable to downstream prediction or planning control scenarios, and has a wide range of applications.

[0004] In a first aspect, this application provides an obstacle detection method for autonomous vehicles, comprising: acquiring three-dimensional geometric information of a target obstacle in the vehicle's driving environment; determining multiple image regions corresponding to the target obstacle in images acquired by the camera based on the three-dimensional geometric information and calibration parameters of at least one camera; the images acquired by the camera include multi-view images and / or multi-temporal images; inputting the multiple image regions corresponding to the target obstacle and multiple candidate type texts in a dynamic semantic type library into a trained visual language model, and acquiring the semantic similarity between each image region output by the visual language model and each of the candidate type texts; and determining the type information of the target obstacle based on the semantic similarity corresponding to each image region.

[0005] The obstacle detection method for autonomous vehicles provided in this application obtains the three-dimensional position information of obstacles through geometric perception, then projects the three-dimensional information onto a camera image to extract multiple corresponding image regions. These image regions, along with candidate text from a dynamic semantic type library, are then input into a pre-trained visual language model for cross-modal semantic similarity calculation. Finally, the similarity results of all image regions are fused to determine the obstacle type. This method achieves the ability to dynamically expand the open set detection capability of the recognition type without retraining the model, improving the accuracy and robustness of the semantic recognition of general obstacles in autonomous driving systems. It is applicable to downstream prediction or planning control scenarios and has a wide range of applications.

[0006] An embodiment of the present application provides an obstacle detection method for an autonomous vehicle, wherein determining multiple image regions corresponding to the target obstacle in an image acquired by the camera based on the three-dimensional geometric information and calibration parameters of at least one camera includes: projecting the three-dimensional geometric information onto the image acquired by the camera based on the camera calibration parameters; obtaining a pixel region surrounding the target obstacle in the image acquired by the camera according to the projection result, and determining the pixel region as the image region.

[0007] An embodiment of the present application provides an obstacle detection method for an autonomous vehicle, wherein the at least one camera is a plurality of cameras with different acquisition angles, and the plurality of image regions include the image regions corresponding to the target obstacle in the images acquired by the plurality of cameras with different acquisition angles at the current acquisition time.

[0008] An embodiment of the obstacle detection method for an autonomous vehicle according to this application includes projecting the three-dimensional geometric information onto an image acquired by the camera based on the camera's calibration parameters. This includes: transforming the three-dimensional geometric information to the vehicle coordinate system at at least one historical moment based on the vehicle's position information at that historical moment; and projecting the transformed three-dimensional geometric information onto an image acquired by the camera at that historical moment based on the camera's calibration parameters. The plurality of image regions includes the image region corresponding to the target obstacle in the image acquired by the camera at the at least one historical moment.

[0009] An embodiment of the obstacle detection method for an autonomous vehicle according to this application includes inputting multiple image regions corresponding to the target obstacle and multiple candidate type texts from a dynamic semantic type library into a trained visual language model, and obtaining the semantic similarity between each image region output by the visual language model and each candidate type text, comprising: obtaining visual feature vectors of each image region based on the visual language model; obtaining text feature vectors of each candidate type text based on the text encoder of the visual language model; calculating the similarity score between each visual feature vector and each text feature vector to obtain the semantic similarity.

[0010] An embodiment of the obstacle detection method for autonomous vehicles according to this application includes determining the type information of the target obstacle based on the semantic similarity corresponding to each of the image regions, comprising: performing a fusion processing on the semantic similarity of all image regions corresponding to the same target obstacle to obtain a comprehensive matching degree of the target obstacle with respect to each of the candidate type texts; and determining the type information of the target obstacle based on the comprehensive matching degree.

[0011] Secondly, this application provides an obstacle detection device for an autonomous vehicle, comprising: a first processing module for acquiring three-dimensional geometric information of a target obstacle in the vehicle's driving environment; a second processing module for determining multiple image regions corresponding to the target obstacle in an image acquired by the camera based on the three-dimensional geometric information and calibration parameters of at least one camera; the image acquired by the camera includes multi-view images and / or multi-temporal images; a third processing module for inputting the multiple image regions corresponding to the target obstacle and multiple candidate type texts in a dynamic semantic type library into a trained visual language model, and obtaining the semantic similarity between each image region output by the visual language model and each of the candidate type texts; and a fourth processing module for determining the type information of the target obstacle based on the semantic similarity corresponding to each of the image regions.

[0012] The obstacle detection device for autonomous vehicles provided in this application obtains the three-dimensional position information of obstacles through geometric perception, then projects the three-dimensional information onto a camera image to extract multiple corresponding image regions. These image regions, along with candidate text from a dynamic semantic type library, are then input into a pre-trained visual language model for cross-modal semantic similarity calculation. Finally, the similarity results of all image regions are fused to determine the obstacle type. This achieves the ability to dynamically expand the open set detection capability of the recognition type without retraining the model, improving the accuracy and robustness of the semantic recognition of general obstacles in autonomous driving systems. It is applicable to downstream prediction or planning control scenarios and has a wide range of applications.

[0013] Thirdly, this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the obstacle detection method for an autonomous vehicle as described in the first aspect above.

[0014] Fourthly, this application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the obstacle detection method for an autonomous vehicle as described in the first aspect above.

[0015] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the obstacle detection method for an autonomous vehicle as described in the first aspect above.

[0016] The above-mentioned one or more technical solutions in the embodiments of this application have at least one of the following technical effects: the three-dimensional position information of the obstacle is obtained through geometric perception, and then the three-dimensional information is projected into the camera image to extract multiple corresponding image regions. Then, these image regions and candidate texts in the dynamic semantic type library are input into the pre-trained visual language model for cross-modal semantic similarity calculation. Finally, the similarity results of all image regions are fused to determine the obstacle type. This realizes the ability to dynamically expand the open set detection capability of the recognition type without retraining the model, improves the accuracy and robustness of the semantic recognition of general obstacles in the autonomous driving system, and is applicable to downstream prediction or planning control scenarios, etc., with a wide range of applicable scenarios.

[0017] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0018] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments in conjunction with the following drawings, in which: Figure 1 is a flowchart illustrating one of the obstacle detection methods for autonomous vehicles provided in an embodiment of this application; Figure 2 is a system architecture diagram illustrating the obstacle detection method for autonomous vehicles provided in an embodiment of this application; Figure 3 is a flowchart illustrating another of the obstacle detection methods for autonomous vehicles provided in an embodiment of this application; Figure 4 is a flowchart illustrating a third of the obstacle detection methods for autonomous vehicles provided in an embodiment of this application; Figure 5 is a structural diagram illustrating an obstacle detection device for autonomous vehicles provided in an embodiment of this application; and Figure 6 is a structural diagram illustrating an electronic device provided in an embodiment of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0020] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0021] The obstacle detection method, obstacle detection device, electronic device, and readable storage medium for autonomous vehicles provided in this application will be described in detail below with reference to the accompanying drawings and through specific embodiments and application scenarios.

[0022] The obstacle detection method for autonomous vehicles can be applied to the terminal, specifically executed by the hardware or software within the terminal.

[0023] The obstacle detection method for autonomous vehicles provided in this application embodiment can be executed by an electronic device or a functional module or entity in an electronic device that can implement the obstacle detection method for autonomous vehicles. The electronic devices mentioned in this application embodiment include, but are not limited to, mobile phones, tablets, computers, cameras, and wearable devices. The obstacle detection method for autonomous vehicles provided in this application embodiment will be described below using an electronic device as the execution subject.

[0024] As shown in Figure 1, the obstacle detection method for the autonomous vehicle includes steps 110, 120, 130, and 140.

[0025] Step 110: Obtain the three-dimensional geometric information of the target obstacle in the vehicle's driving environment; in this step, the vehicle can be a mobile platform such as a car or truck with autonomous driving or assisted driving capabilities.

[0026] The vehicle's driving environment is the external physical space in which the vehicle is located in real time, including roads, traffic participants, and various static or dynamic objects.

[0027] Target obstacles can be general obstacles in the vehicle's driving environment. In autonomous driving, general obstacles are represented as any obstacle other than dynamic obstacles, such as vehicles, pedestrians, two-wheeled vehicles, and three-wheeled vehicles, or any obstacle that affects passage. These generally include traffic cones, water-filled barriers, bottles, animals, stones, and mounds of earth in the scene. General obstacles have complex shapes, random and dynamic distributions, and it is impossible to predefine all possible semantic types.

[0028] Three-dimensional geometric information includes the shape and position information of the target obstacle in three-dimensional space, which can be represented by, for example, a 3D bounding box, a polygon, or a point cloud cluster (3D point cloud cluster).

[0029] In actual implementation, Figure 2 illustrates the system architecture corresponding to the obstacle detection method for autonomous vehicles provided in this application embodiment. The system architecture includes an obstacle clustering module, which can be used to recall common obstacles in the recall scenario. For example, it can be based on the method of LiDAR clustering, that is, ground removal and clustering of point cloud, or it can be based on voxel division of 3D space, obtaining the occupancy state of voxels through occupancy prediction model, and then clustering the voxels in the occupancy state, or it can be obtained based on other methods, which are not limited in this application.

[0030] Step 120: Based on three-dimensional geometric information and calibration parameters of at least one camera, determine multiple image regions corresponding to the target obstacle in the images acquired by the camera; in this step, at least one camera can be one or more vision sensors installed on the vehicle.

[0031] At least one camera's calibration parameters are the camera's extrinsic parameters, namely the calibration matrix of the camera's mounting position and orientation relative to the vehicle coordinate system, used to accurately project 3D spatial coordinates onto 2D image pixel coordinates.

[0032] The images acquired by the camera include multi-view images and / or multi-temporal images. Multiple image regions are two-dimensional image blocks (pixel regions) extracted from images of the same target obstacle from different camera perspectives and / or different historical moments. This allows the construction of an observation set of multi-view and / or multi-temporal image blocks, which improves the redundancy of the original observations and can provide rich and complementary visual cues for subsequent semantic analysis to address issues such as single-view occlusion and changes in lighting.

[0033] As shown in Figure 2, the system architecture includes a coordinate projection module. The coordinate projection module can project the three-dimensional geometric information of the target obstacle onto the image space using the calibrated camera (sensor) extrinsic parameters, and extract the 2D pixel area surrounding the obstacle in the image.

[0034] Step 130: Input multiple image regions corresponding to the target obstacle and multiple candidate type texts from the dynamic semantic type library into the trained visual language model, and obtain the semantic similarity between each image region and each candidate type text output by the visual language model. In this step, the dynamic semantic type library can store text descriptions of possible obstacle types (such as "cone", "cardboard box" and "plastic bag"). The dynamic semantic type library can dynamically add, delete or modify the text descriptions in the library. Multiple candidate type texts can be determined from the dynamic semantic type library. Multiple candidate type texts can be dynamically added, deleted or modified based on user needs without retraining the entire model.

[0035] Visual language models are neural network models trained on large-scale text-image pairs (such as CLIP (Contrastive Language-Image Pre-Training) models, etc.). They have cross-modal semantic alignment capabilities, enabling them to map images and text to a common semantic space and calculate their similarity.

[0036] Semantic similarity is used to characterize the degree of semantic matching between an image region and a type of text description. Semantic similarity can be a scalar score to quantify the degree of matching between an image region and candidate type text in the shared semantic space understood by the visual language model.

[0037] Visual language models can perform fully connected matching, where each image region is compared with every candidate type of text in the semantic type library to form a similarity matrix.

[0038] As shown in Figure 2, the system architecture may include a VLM (Vision-Language Models) semantic extraction module, which constructs a dynamic semantic space that can dynamically expand the number of predicted semantic types.

[0039] The Visual Language Model (VLM Semantic Extraction Module) can maintain a dynamically sized library of semantic types (text) and output the comparison score (i.e. semantic similarity) between image regions and each candidate type of text.

[0040] Step 140: Determine the type information of the target obstacle based on the semantic similarity of each image region.

[0041] In this step, the semantic similarity of each image region can be processed to reduce the semantic noise of the image region at a single viewpoint and / or a single moment. Based on multiple semantic similarities obtained from multiple viewpoints and / or multiple time series of image regions, a comprehensive decision can be made to determine the final semantic type of the target obstacle, i.e., the type information of the target obstacle.

[0042] In this application, by transforming the recognition problem into a matching problem between image patches and a dynamic text library, the type detection of general obstacles in open sets can be achieved. To identify new types, it is only necessary to add new descriptions to the text library, without spending a lot of manpower and time to add type annotations, thus resolving the contradiction between closed set models and open environments.

[0043] Matching can be performed based on pre-trained visual language models (VLM models), which reduces the difficulty of training and deployment. It can utilize powerful general VLM models or be fine-tuned with text-image pairs in specific domains, making it highly applicable to engineering projects.

[0044] The obstacle detection method for autonomous vehicles provided in this application obtains the three-dimensional position information of obstacles through geometric perception, then projects the three-dimensional information onto a camera image to extract multiple corresponding image regions. These image regions, along with candidate text from a dynamic semantic type library, are then input into a pre-trained visual language model for cross-modal semantic similarity calculation. Finally, the similarity results of all image regions are fused to determine the obstacle type. This method achieves the ability to dynamically expand the open set detection capability of the recognition type without retraining the model, improving the accuracy and robustness of the semantic recognition of general obstacles in autonomous driving systems. It is applicable to downstream prediction or planning control scenarios and has a wide range of applications.

[0045] In some embodiments, step 120 may include: projecting three-dimensional geometric information onto an image acquired by the camera based on the camera's calibration parameters; obtaining the pixel region surrounding the target obstacle in the image acquired by the camera based on the projection result, and determining the pixel region as an image region.

[0046] In this embodiment, the perspective projection geometry principle can be used to calculate the corresponding pixel coordinates on a specified camera plane by using a transformation matrix composed of calibrated extrinsic parameters to represent the three-dimensional spatial point set describing the shape and position of the obstacle.

[0047] Image data cropping can be performed based on pixel coordinates generated by projection. The projection coordinates can be used as boundaries to crop (extract) corresponding rectangular or irregularly shaped pixel blocks from the original image, resulting in local image blocks that only contain (or mainly contain) the target obstacle, while removing irrelevant background information.

[0048] The cropped pixel blocks can be defined as image regions, i.e., the input units of the VLM module.

[0049] By projecting three-dimensional geometric information onto two-dimensional images and obtaining the image region of the target obstacle in the camera image, multiple image regions obtained from different perspectives or at different times point to the same three-dimensional entity in space, providing a reliable data association basis for subsequent multi-source information fusion.

[0050] In some embodiments, at least one camera is a plurality of cameras with different acquisition perspectives, and the plurality of image regions include the image regions corresponding to the target obstacle in the images acquired by the cameras with different acquisition perspectives at the current acquisition time.

[0051] In this embodiment, multiple cameras with different acquisition angles can be multiple cameras distributed around the vehicle body (such as front view, rear view, left view or right view), and each camera has an independent field of view (angle).

[0052] Images from all perspectives were acquired at the same time, providing instantaneous and omnidirectional spatial observation images of the same obstacle.

[0053] At the same moment, the three-dimensional geometric information of the same obstacle can be projected onto the current frame image of each camera with a different viewpoint in parallel, thereby cropping out multiple different image regions. Finally, multiple image blocks of the same obstacle observed at the same moment and from different angles can be obtained. For example, a cone may be shown as a side view in a front-view camera and as a front view in a side-view camera.

[0054] In this application, images from different perspectives may capture different appearance surfaces of obstacles. Fusing this information can provide a more complete appearance feature for the VLM model, which helps to identify view-dependent obstacles (such as boxes where text can only be seen from a specific angle). If an obstacle is partially occluded from a certain perspective (such as being blocked by other vehicles), images from other perspectives may provide unobstructed or less obstructed views, ensuring the system's perceptual robustness in complex scenes.

[0055] In some embodiments, projecting three-dimensional geometric information onto images acquired by the camera based on camera calibration parameters may include: transforming the three-dimensional geometric information to the vehicle coordinate system at at least one historical moment based on the vehicle's position information at at least one historical moment; and projecting the transformed three-dimensional geometric information onto images acquired by the camera at the historical moment based on the camera calibration parameters corresponding to the historical moment; wherein, the multiple image regions include the image region corresponding to the target obstacle in images acquired by the camera at at least one historical moment.

[0056] In this embodiment, the vehicle is in motion, and the three-dimensional coordinates of the obstacle detected at the current time (T0) are relative to the vehicle coordinate system at time T0. To project it onto images from earlier historical times (T-1 and T-2, etc.), spatiotemporal alignment of the coordinate systems can be performed.

[0057] The vehicle's pose (position and orientation) at one or more past moments can be recorded by a positioning system (such as an IMU (Inertial Measurement Unit) or GNSS (Global Navigation Satellite System), that is, the vehicle's position information at at least one historical moment.

[0058] Historical pose information can be used to transform the three-dimensional geometric information of the obstacle at the current time (T0) into the vehicle coordinate system at the target historical time (such as T-1) through three-dimensional rigid body transformation (rotation and translation), so that the three-dimensional point has a coordinate representation in the historical time coordinate system.

[0059] The 3D geometric information that has been transformed to the historical coordinate system can be projected onto the image that was actually acquired and stored at that historical moment using the camera extrinsic parameters at that historical moment, and then cropped.

[0060] Multiple image regions can originate from asynchronous observations in the time dimension. That is, the final set of image patches used for VLM model processing contains historical images of the obstacle extracted from images at one or more past moments.

[0061] This application provides visual observations of obstacles over a time series, which helps to identify objects whose state may change (such as a box being moved or a flashing warning light) or verify the durability of stationary objects.

[0062] Combining multiple historical observations helps to more accurately infer whether an obstacle is static, moving slowly, or has just appeared, providing richer contextual information for downstream prediction modules.

[0063] In actual execution, the coordinate projection module can use the 3D geometric information of obstacles obtained by the obstacle clustering module to project some 3D enclosing points onto the image space using calibrated sensor extrinsic parameters, and then extract the 2D pixel region surrounding the obstacle from the image. The image space includes multiple cameras installed on the autonomous vehicle, as well as historical time-series image information. When extracting historical time-series image information, in addition to using extrinsic parameter information, it is also necessary to use the autonomous vehicle's positioning information to transform the geometric 3D points of the current frame to the 3D coordinate system of the historical image (such as the vehicle system of the autonomous vehicle), and then use the calibrated extrinsic parameter information to project onto the image space of that historical moment to extract the time-series image. This module includes a memory bank for storing historical time-series image information, as well as historical positioning information, used to align to the 3D coordinate space where the historical image is located, and then extract the corresponding time-series image block.

[0064] In some embodiments, step 130 may include: an image encoder based on a visual language model obtaining visual feature vectors for each image region; a text encoder based on a visual language model obtaining text feature vectors for each candidate type of text; and calculating the similarity score between each visual feature vector and each text feature vector to obtain semantic similarity.

[0065] In this embodiment, the image encoder is a neural network (such as VisionTransformer or CNN (convolutional neural network)) responsible for processing visual input in the VLM model. It is a feature extraction function that can map an image into a fixed-length vector, that is, it can transform the original pixel information into a high-level semantic representation that the machine can understand.

[0066] Visual feature vectors encode the deep visual semantic content of image patches, such as shape, texture, color, and local structure, and compress them into a shared semantic feature space.

[0067] A text encoder is a neural network (such as a Transformer) in a VLM model that is responsible for processing text input and can map a text sequence into a fixed-length vector.

[0068] Text feature vectors encode the semantic connotation of a type of text description (such as "red cone"), and text feature vectors are close to image feature vectors describing objects in semantic space.

[0069] The cosine similarity or dot product between visual feature vectors and text feature vectors can be calculated to obtain a scalar score. The higher the score, the better the image content matches the text description in the semantic space learned by VLM.

[0070] In actual execution, Figure 3 illustrates the framework of the VLM semantic extraction module. The VLM semantic extraction module constructs a dynamic semantic space, which can dynamically expand the number of predicted semantic types. This module maintains a dynamically sized semantic type space for possible categories. It extracts features of the text type through the text representation of the category using a text feature extractor. The module takes as input image patches extracted from images at different viewpoints and time sequences, and uses an image feature extractor to extract features from all image patches. The text feature extractor outputs a feature representation of the text type, typically a vector, and a trained document feature extraction model, usually a neural network. The image feature extractor outputs a feature representation of an image patch, typically a vector, and a trained image feature extraction model, usually a neural network. When the image patch enters the image feature extraction module, it undergoes scaling or pooling to transform it to the same dimension. The text-image similarity calculator takes as input the feature representation of the image patch after feature extraction, typically a vector, a model, or an inner product calculation, and finally outputs a comparison score. The entire module, consisting of a text feature extractor, an image feature extractor, and a text-image feature similarity calculator, forms a large neural network model that can be trained simultaneously using text-image pairs.

[0071] In this application, by concretizing semantic similarity into geometric distance calculation in vector space (such as cosine similarity), open set identification is transformed from a concept into a deterministic numerical calculation process.

[0072] Text feature vectors can be generated in real time independently of images (only the newly added text descriptions need to be encoded). Therefore, to expand the recognition types, only new vectors need to be added to the set of text feature vectors, without modifying the image encoder or retraining the entire model, thus realizing the dynamic expansion of the type space.

[0073] In some embodiments, step 140 may include: fusing the semantic similarity of all image regions corresponding to the same target obstacle to obtain the comprehensive matching degree of the target obstacle with respect to each candidate type of text; and determining the type information of the target obstacle based on the comprehensive matching degree.

[0074] In this embodiment, the semantic similarity calculated for each independent image region (from a single viewpoint or a single moment) can be regarded as a preliminary observation hypothesis with noise. The sources of noise include local occlusion, illumination changes, imaging blur, visual bias, or instantaneous misjudgment by the model.

[0075] By fusing the semantic similarity of all image regions corresponding to the same target obstacle, we can integrate multiple independent and potentially erroneous observations and utilize their complementarity and redundancy to suppress random errors, thereby obtaining a more stable estimation result than any single observation.

[0076] In some embodiments, a voting method or a weighted fusion method may be used to fuse the semantic similarity of all image regions corresponding to the same target obstacle, or other methods may be used, which are not limited in this application.

[0077] In the weighted fusion mechanism, the weight of semantic similarity of each image region can be determined based on at least one of the following factors: the source view of the image region, the time difference between the source time and the current time of the image region, and the imaging quality assessment score of the image region.

[0078] The similarity scores of multiple image patches to various types of text can be merged into a more representative output, resulting in a comprehensive matching score.

[0079] For each candidate type text in the semantic type library, a comprehensive matching score is generated, which reflects the overall confidence level that the obstacle belongs to that category after considering all observation perspectives and times.

[0080] The candidate type text with the highest comprehensive matching degree can be used as the final type information, or the type information of the target obstacle can be determined based on other methods; this application does not impose any limitations.

[0081] In actual execution, as shown in Figure 4, the semantic type smoothing module can be used to reduce the semantic noise of single-view image patches. The semantic type smoothing module can perform semantic type filtering or smoothing based on the semantic type information obtained from image patches in multi-view images and / or time-series multi-frame images. For example, a voting method can be used, or more complex logic or algorithms can be used to reduce the semantic noise of certain image patches obtained from multi-time-series and / or multi-view image patches, and finally obtain a more accurate semantic type.

[0082] In this application, the robustness of the system is ensured by fusing the semantic similarity of all image regions.

[0083] The obstacle detection method for autonomous vehicles provided in this application can be executed by an obstacle detection device for autonomous vehicles. This application uses an obstacle detection device for autonomous vehicles executing the obstacle detection method as an example to illustrate the obstacle detection device for autonomous vehicles provided in this application.

[0084] This application also provides an obstacle detection device for autonomous vehicles.

[0085] As shown in Figure 5, the obstacle detection device for the autonomous vehicle includes: a first processing module 510, a second processing module 520, a third processing module 530, and a fourth processing module 540.

[0086] The first processing module 510 is used to acquire the three-dimensional geometric information of the target obstacle in the vehicle driving environment; the second processing module 520 is used to determine multiple image regions corresponding to the target obstacle in the images acquired by the camera based on the three-dimensional geometric information and the calibration parameters of at least one camera; the images acquired by the camera include multi-view images and / or multi-temporal images; the third processing module 530 is used to input the multiple image regions corresponding to the target obstacle and multiple candidate type texts in the dynamic semantic type library into the trained visual language model, and obtain the semantic similarity between each image region output by the visual language model and each candidate type text; the fourth processing module 540 is used to determine the type information of the target obstacle based on the semantic similarity corresponding to each image region.

[0087] The obstacle detection device for autonomous vehicles provided in this application obtains the three-dimensional position information of obstacles through geometric perception, then projects the three-dimensional information onto a camera image to extract multiple corresponding image regions. These image regions, along with candidate text from a dynamic semantic type library, are then input into a pre-trained visual language model for cross-modal semantic similarity calculation. Finally, the similarity results of all image regions are fused to determine the obstacle type. This achieves the ability to dynamically expand the open set detection capability of the recognition type without retraining the model, improving the accuracy and robustness of the semantic recognition of general obstacles in autonomous driving systems. It is applicable to downstream prediction or planning control scenarios and has a wide range of applications.

[0088] In some embodiments, the second processing module 520 may also be used to: project three-dimensional geometric information onto the image acquired by the camera based on the camera's calibration parameters; and in the image acquired by the camera, obtain the pixel region surrounding the target obstacle according to the projection result, and determine the pixel region as the image region.

[0089] In some embodiments, the second processing module 520 can also be used to make at least one camera a camera with multiple different acquisition perspectives, and the multiple image regions include the image regions corresponding to the target obstacle in the images acquired by the multiple cameras with different acquisition perspectives at the current acquisition time.

[0090] In some embodiments, the second processing module 520 may further be used to: transform the three-dimensional geometric information to the vehicle coordinate system at the historical time based on the vehicle's position information at at least one historical time; project the transformed three-dimensional geometric information onto the image captured by the camera at the historical time based on the calibration parameters of the camera at the historical time; wherein, the multiple image regions include the image regions corresponding to the target obstacle in the images captured by the camera at at least one historical time.

[0091] In some embodiments, the third processing module 530 can also be used to: obtain visual feature vectors of each image region by an image encoder based on a visual language model; obtain text feature vectors of each candidate type of text by a text encoder based on a visual language model; calculate the similarity score between each visual feature vector and each text feature vector to obtain semantic similarity.

[0092] In some embodiments, the fourth processing module 540 may also be used to: perform semantic similarity fusion processing on all image regions corresponding to the same target obstacle to obtain the comprehensive matching degree of the target obstacle with respect to each candidate type of text; and determine the type information of the target obstacle based on the comprehensive matching degree.

[0093] The obstacle detection device for autonomous vehicles in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the specific implementation.

[0094] The obstacle detection device for autonomous vehicles in this application embodiment can be a device with an operating system. This operating system can be a Microsoft (Windows) operating system, an Android operating system, an iOS operating system, or other possible operating systems; this application embodiment does not specifically limit the specific operating system.

[0095] The obstacle detection device for autonomous vehicles provided in this application embodiment can realize the various processes implemented in the method embodiments of Figures 1 to 4. To avoid repetition, it will not be described again here.

[0096] In some embodiments, as shown in FIG6, this application embodiment also provides an electronic device 600, including a processor 601, a memory 602, and a computer program stored in the memory 602 and executable on the processor 601. When the program is executed by the processor 601, it implements the various processes of the obstacle detection method embodiment of the above-described autonomous vehicle and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0097] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0098] This application also provides a non-transitory computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the obstacle detection method embodiment for the above-described autonomous vehicle and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0099] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0100] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the obstacle detection method for the above-described autonomous vehicle.

[0101] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0102] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the obstacle detection method embodiment of the above-described autonomous vehicle, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0103] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0104] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0105] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the related technology, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0106] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

[0107] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0108] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

Claims

1. An obstacle detection method for an autonomous vehicle, characterized in that, include: Acquire three-dimensional geometric information of target obstacles in the vehicle's driving environment; Based on the three-dimensional geometric information and the calibration parameters of at least one camera, multiple image regions corresponding to the target obstacle in the images acquired by the camera are determined; the images acquired by the camera include multi-view images and / or multi-temporal images; the multiple image regions corresponding to the target obstacle and multiple candidate type texts in a dynamic semantic type library are input into a trained visual language model to obtain the semantic similarity between each image region output by the visual language model and each of the candidate type texts; based on the semantic similarity corresponding to each image region, the type information of the target obstacle is determined.

2. The obstacle detection method for autonomous vehicles according to claim 1, characterized in that, The step of determining multiple image regions corresponding to the target obstacle in the image acquired by the camera based on the three-dimensional geometric information and the calibration parameters of at least one camera includes: projecting the three-dimensional geometric information into the image acquired by the camera based on the calibration parameters of the camera; obtaining the pixel region surrounding the target obstacle in the image acquired by the camera according to the projection result, and determining the pixel region as the image region.

3. The obstacle detection method for autonomous vehicles according to claim 2, characterized in that, The at least one camera is a plurality of cameras with different acquisition angles, and the plurality of image regions include the image regions corresponding to the target obstacle in the images acquired by the plurality of cameras with different acquisition angles at the current acquisition time.

4. The obstacle detection method for autonomous vehicles according to claim 2, characterized in that, The step of projecting the three-dimensional geometric information onto the image acquired by the camera based on the camera's calibration parameters includes: transforming the three-dimensional geometric information to the vehicle coordinate system at at least one historical moment based on the vehicle's position information at that historical moment; and projecting the transformed three-dimensional geometric information onto the image acquired by the camera at that historical moment based on the camera's calibration parameters at that historical moment; wherein, the plurality of image regions include the image region corresponding to the target obstacle in the image acquired by the camera at at least one historical moment.

5. The obstacle detection method for an autonomous vehicle according to any one of claims 1-4, characterized in that, The step of inputting multiple image regions corresponding to the target obstacle and multiple candidate type texts from a dynamic semantic type library into a trained visual language model, and obtaining the semantic similarity between each image region output by the visual language model and each candidate type text, includes: obtaining visual feature vectors of each image region based on the visual language model; obtaining text feature vectors of each candidate type text based on the text encoder of the visual language model; calculating the similarity score between each visual feature vector and each text feature vector to obtain the semantic similarity.

6. The obstacle detection method for an autonomous vehicle according to any one of claims 1-4, characterized in that, The step of determining the type information of the target obstacle based on the semantic similarity corresponding to each of the image regions includes: performing a fusion process on the semantic similarity of all image regions corresponding to the same target obstacle to obtain the comprehensive matching degree of the target obstacle with respect to each of the candidate type texts; and determining the type information of the target obstacle based on the comprehensive matching degree.

7. An obstacle detection device for an autonomous vehicle, characterized in that, include: The first processing module is used to acquire the three-dimensional geometric information of the target obstacle in the vehicle's driving environment; The second processing module is used to determine multiple image regions corresponding to the target obstacle in the images acquired by the camera based on the three-dimensional geometric information and the calibration parameters of at least one camera; the images acquired by the camera include multi-view images and / or multi-temporal images; The third processing module is used to input multiple image regions corresponding to the target obstacle and multiple candidate type texts in the dynamic semantic type library into the trained visual language model, and obtain the semantic similarity between each image region output by the visual language model and each candidate type text; the fourth processing module is used to determine the type information of the target obstacle based on the semantic similarity corresponding to each image region.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the obstacle detection method for an autonomous vehicle as described in any one of claims 1-6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the obstacle detection method for an autonomous vehicle as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the obstacle detection method for an autonomous vehicle as described in any one of claims 1-6.

Images