Object detection method, apparatus, electronic device, product, and medium

Abstract

The present disclosure relates to an object detection method, device, electronic device, computer program product and machine readable storage medium. The method comprises determining radar point cloud data of a target object based on radar point cloud input data. The method further comprises extracting radar features of the target object based on the radar point cloud data of the target object. In addition, the method further comprises generating radar point cloud output data identifying the target object with text based on the radar features of the target object. In this way, the radar point cloud data corresponding to the target object in the radar point cloud input data can be identified, and the corresponding text can be identified at the relevant target object, which not only saves the cost required for data collection and labeling, but also expands the object categories based on radar detection, and can accurately identify multiple types of target objects.

Description

Technical Field

[0001] This disclosure relates to the field of computers, and more specifically to object detection methods, apparatus, electronic devices, computer program products, and machine-readable storage media. Background Technology

[0002] In open-world object detection, object detection algorithms are needed to identify both unknown and known objects. These algorithms typically focus on recognizing categories found in the datasets they were trained on. The more data collected and the more refined the classifications, the more diverse the types of objects that can be identified.

[0003] In some related technologies, radar object detection relies on radar datasets with limited object categories. To identify more categories of objects, a significant amount of data collection and annotation costs are required. Summary of the Invention

[0004] Embodiments of this disclosure provide an object detection method, apparatus, electronic device, computer program product, and machine-readable storage medium.

[0005] In a first aspect of this disclosure, an object detection method is provided. The method includes determining radar point cloud data of a target object based on radar point cloud input data. The method also includes extracting radar features of the target object based on the radar point cloud data of the target object. Furthermore, the method includes generating radar point cloud output data with textual identifiers for the target object based on the radar features of the target object.

[0006] In a second aspect of this disclosure, an object detection apparatus is provided. The apparatus includes a determination module configured to determine radar point cloud data of a target object based on radar point cloud input data. The apparatus also includes an extraction module configured to extract radar features of the target object based on the radar point cloud data of the target object. Furthermore, the apparatus includes a generation module configured to generate radar point cloud output data with textual identifiers of the target object based on the radar features of the target object.

[0007] In a third aspect of this disclosure, an electronic device is provided. The electronic device includes at least one processor and a memory. The memory is coupled to the at least one processor and has instructions stored thereon that, when executed by the at least one processor, cause the electronic device to perform the methods provided according to the first aspect of this disclosure.

[0008] In a fourth aspect of this disclosure, a computer program product is provided. This computer program product includes a computer program that is executed by a processor to implement the method provided according to a first aspect of this disclosure.

[0009] In a fifth aspect of this disclosure, a machine-readable storage medium is provided. The machine-readable storage medium stores machine-executable instructions, which are executed by a processor to implement the method provided according to a first aspect of this disclosure.

[0010] It should be understood that the description in the Summary of the Invention section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0012] Figure 1 A schematic diagram of an example environment in which some embodiments of this disclosure may be implemented is shown;

[0013] Figure 2 A flowchart of an object detection method according to some embodiments of this disclosure is shown;

[0014] Figure 3 A system architecture diagram for object detection according to some embodiments of this disclosure is shown;

[0015] Figure 4 A schematic diagram illustrating the principle process of radar feature extraction models in the training phase of some embodiments of this disclosure is shown;

[0016] Figure 5 A schematic diagram illustrating the radar feature determination process of a target object according to some embodiments of the present disclosure is shown;

[0017] Figure 6 A schematic diagram illustrating the radar-text pairing process of some embodiments of this disclosure is shown;

[0018] Figure 7 The illustration shows an example process for object detection using a trained radar feature extraction model, according to some embodiments of this disclosure;

[0019] Figure 8 Example block diagrams of object detection apparatuses according to some embodiments of the present disclosure are shown;

[0020] Figure 9 A schematic diagram of the structure of an electronic device according to some embodiments of the present disclosure is shown. Detailed Implementation

[0021] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0022] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0023] As mentioned above, the accuracy of object detection technology depends on the dataset. The more data collected and the more detailed the classification, the more diverse the object detection technology can identify. Both image-based and radar-based object detection technologies rely on labeled datasets. Collecting and labeling these datasets requires significant investment of resources.

[0024] In related technologies, radar detection in intelligent driving scenarios can generally identify cars and people, but other unknown objects require annotation and training before they can be identified. However, various unknown objects can appear on the road, including animals, trees, various types of motor vehicles, and traffic facilities. Identifying these unknown objects requires collecting and annotating a large amount of data. Furthermore, other radar detection applications present different types of unknown objects depending on the scenario, necessitating the collection and annotation of even larger amounts of data based on the specific characteristics of each scenario.

[0025] To address this, embodiments of this disclosure propose an object detection scheme that aligns radar and text features at the object granularity. In embodiments of this disclosure, text features aligned with the radar features of the target object can be determined based on radar point cloud data of the target object. These text features correspond to text, allowing the classification of the target object to be determined based on the text. Then, the target object in the radar point cloud data is labeled based on this text. The features generated by determining the corresponding text aligned with the radar features are implemented by a trained radar feature extraction model. For example, this radar feature extraction model learns the image-text alignment results from an image and text matching model to align the features of radar, image, and text at the target object granularity.

[0026] In this way, without relying on labeled datasets, the category of target objects can be identified based on a trained radar feature extraction model, enabling the identification of multiple object types. Furthermore, this method can be generalized and applied to various application scenarios. It eliminates the need for re-collecting and labeling data specific to each scenario; instead, it uses, for example, image and text matching models as teacher models for learning. By learning the alignment relationships between images and text already learned by the teacher model, multiple object types in various scenarios can be identified.

[0027] Figure 1 A schematic diagram of an example environment 100 in which some embodiments of this disclosure may be implemented is shown. For example... Figure 1 As shown, environment 100 includes vehicle 102 and road 104, with vehicle 102 traveling on road 104. Vehicle 102 is equipped with a camera device 102-1 and a radar detection device 102-2. The camera device 102-1 is used to acquire environmental images within a certain range 102-11 in front of vehicle 102, and the radar detection device 102-2 is used to detect whether there are objects in front of the vehicle within a detection range 102-22. The radar detection device 102-2 can be a lidar detection device (e.g., the device outputs the x, y, z coordinates and reflection intensity of the detection point in three-dimensional space, these parameters constituting radar point cloud data), a millimeter-wave radar detection device (e.g., the device outputs the x, y coordinates, power, and speed of the detection point in a two-dimensional plane), etc. In some examples, vehicle 102 is equipped with one camera device 102-1 and one radar detection device 102-2. In some examples, vehicle 102 may be equipped with one camera device and multiple radar detection devices. This radar detection device can be used to detect any multiple directions in front of, to the front and sides of, behind, and to the rear of the vehicle. In some examples, the vehicle 102 is equipped with multiple camera devices and multiple radar detection devices. These camera devices and radar detection devices can be distributed in any multiple directions in front of, to the front and sides of, behind, and to the rear of the vehicle to acquire environmental images from each direction and detect objects in each direction.

[0028] like Figure 1As shown, the environment 100 also includes a control unit 120, which can be integrated into the vehicle 102 or deployed on a server. The control unit 120 is communicatively connected to both the camera device 102-1 and the radar detection device 102-2, communicating via wired or wireless transmission. The control unit 120 receives environmental images 106-1 acquired by the camera device 102-1 and radar point cloud data 106-2 acquired by the radar detection device 102-2. The control unit 120 can identify radar point cloud data 110 of a target object from the radar point cloud data 106-2. In some examples, the control unit 120 can extract the radar point cloud data 110 of the target object from the radar point cloud data 106-2. In other examples, the control unit can determine the radar point cloud data 110 of the target object based on the radar point cloud data 106-2 and the environmental image 106-1. Subsequently, the control unit 120 determines the text aligned with the radar point cloud data 110 of the target object and outputs radar point cloud data 118 that identifies the target object with text.

[0029] The control unit 120 is equipped with an object detection model for outputting object detection results. This object detection model includes an object extraction module 108, a radar feature extraction model 112, and a generation module 116. The object extraction module 108 extracts radar point cloud data 110 of the target object based on radar point cloud data 106-2. The radar feature extraction model 112 determines the radar feature L of the target object based on the radar point cloud data 110 of the target object. i The radar signature L i The features T generated by the corresponding text mapped to the feature space i Alignment is performed, where i is a natural number from 1 to N. Based on the radar features of the determined target object, the generation module 116 determines the corresponding text-generated features aligned with the radar features 114 of the target object, and outputs radar point cloud data 118 with text identifying the target object. The radar feature extraction model is pre-trained and achieves alignment of radar features, image features, and text features by mapping radar features to a feature space where text features and image features are aligned, and then aligning the radar features with the image features.

[0030] In this way, intelligent driving scenarios can output radar point cloud data with text-based identification of target objects based on collected radar point cloud data, without relying on labeled datasets. This allows for accurate and efficient identification of various object types in the environment. Furthermore, this solution can be applied to other object detection scenarios, such as access control and home applications. In access control scenarios, by using radar detection devices deployed outside the door to collect radar point cloud data of the external scene, various people, objects, and animals present inside the door can be identified. In home applications, by using radar detection devices deployed inside the house to collect radar point cloud data of the internal scene, various people, objects, and animals present inside the house can be identified.

[0031] Figure 2 A flowchart of an object detection method 200 according to some embodiments of this disclosure is shown. Method 200 may, for example, be derived from... Figure 1 The control unit 120 in the environment 100 shown executes the commands. For example... Figure 2 As shown in box 202, method 200 can determine the radar point cloud data of the target object based on the radar point cloud input data. For example, in... Figure 1 In the environment 100 shown, the control unit 120 can determine the radar point cloud data 110 of the target object based on the radar point cloud data 106-2 collected by the radar detection device 102-2. In block 202, determining the radar point cloud data of the target object based on the radar point cloud input data includes determining the target region of the target object based on the radar point cloud input data, and determining the radar point cloud data of the target object based on the target region of the target object. In some implementations, the radar point cloud data of the target object can be determined by clustering using a classic radar detection device. For example, based on the radar point cloud input data, a radar RPN (Region Proposal Network) determines the target region of the target object in the radar point cloud data and uses this target region as the radar point cloud data of the target object. In other implementations, an image RPN determines the target region of the target object in the environmental image based on the environmental image paired with the radar point cloud data. Subsequently, based on the geometric relationship between the radar detection device and the camera device (including their relative positions and orientations in physical space), extrinsic parameter calibration is performed, and the target area defined in the image is mapped to the radar point cloud data. The radar point cloud data of the target object is obtained by cropping.

[0032] In box 204, method 200 extracts radar features of the target object based on radar point cloud data of the target object. In some embodiments, a pre-trained radar feature extraction model is used to extract radar features of the target object from the radar point cloud data of the target object. For example, during the training phase, the radar feature extraction model is trained with the goal of aligning radar features with image features. Furthermore, during training, the alignment relationship between image features and text features obtained from the image and text matching model is learned, thereby enabling the learning of the alignment relationship between radar features and text features through image features. After training, the radar feature extraction model can output radar features that can be aligned with features generated from images and text.

[0033] In box 206, method 200 generates radar point cloud output data with textual identifiers for the target object based on the radar features of the target object. Box 206 can be derived from... Figure 1 The generation module 116 in the environment is implemented. In some embodiments, text features aligned with the radar features are determined from the feature space based on the radar features of the target object. In some embodiments, the radar features of the target object are mapped to the same feature space aligned with the image features and text features. In this way, text features aligned with the radar features can be obtained within the same feature space. In other embodiments, the radar features and image features of the target object are mapped to one feature space, and the image features and text features are mapped to another feature space. In this way, text features aligned with the radar features can be obtained using the image features as an intermediary. Then, the text corresponding to the text features is used to label the target object in the radar point cloud output data. Figure 1 The radar point cloud data 118 has been classified and labeled with text for each target object.

[0034] By utilizing radar point cloud data to detect objects using the above method, the problem of inaccurate object recognition in poor lighting conditions (such as at night) can be solved, and radar detection exhibits higher robustness and accuracy. After extracting the radar features of the target object, the method can determine the text features aligned with it in the same feature space or feature space, or determine the aligned text features through image features. Then, based on the text corresponding to the text features, radar point cloud data with labeled text is output. In this way, the radar feature extraction model can recognize various types of objects without relying on manually labeled data.

[0035] Figure 3A schematic diagram of a system architecture 300 for object detection according to some embodiments of the present disclosure is shown. Block 300A represents the architecture for object detection during the training phase, which includes an image feature extraction model 304, a cropping module 308, and a radar feature extraction model 312. The image feature extraction model 304 extracts feature maps of sample images and sample regions 306 of sample objects based on sample images 302-2 in the training samples, and extracts image features 320 of the sample regions of the sample objects based on the sample regions 306. In some non-limiting examples, feature maps of sample images can be extracted using a feature extraction network, and the feature maps of sample images can be processed using an image RPN to obtain the sample regions of the sample objects. Then, a backbone network, such as a ResNet network, is used to extract the image features 320 of the sample objects from the sample regions of the sample objects. Based on the image features 320 of the sample objects, the cropping model 308 can extract the radar point cloud data 310 of the sample objects from the radar point cloud data 302-1 that is paired with sample images 302-2 in the training samples in a cropping manner. Next, the radar feature extraction model 312 extracts radar features 334 from the sample object. In some examples, the radar features of the sample object can be extracted by the radar feature extraction model based on the radar point cloud data 310 of the sample object. In other examples, the sample radar features can be extracted by the radar feature extraction model based on the sample radar point cloud data. Then, the sample region of the sample object is projected onto the sample radar features to determine the radar features of the sample object. Next, both the radar features 334 and the image features 320 of the sample object are input into the loss calculation module 318, which calculates the loss value. The radar feature extraction model 312 is trained with the goal of minimizing the distance between the image features and radar features of paired sample objects in the feature space. During training, when the distance between the image features and radar features of paired sample objects in the feature space is large, the parameters of the radar feature extraction model 312 are adjusted using the loss value.

[0036] like Figure 3As shown, during the training phase, the image features 320 of the sample object are aligned with the text-generated features in the image-text matching model (not shown in the figure). This image-text matching model can employ, for example, a Contrastive Language-Image Pretraining (CLIP) model, which is trained on massive amounts of data. In the feature space of the CLIP model, image features and corresponding text features are already aligned. Therefore, the text encoder 316 in the image-text matching model generates text features based on text 314 describing an object, which are then aligned with the image features of that object. Thus, based on the image features, the text 314 aligned with the image features can be determined. Once the alignment of radar features 334 and image features 320 in the feature space is achieved, radar features 334 can be aligned with text features generated based on text 314, using image features 320 as an intermediary. In some examples, radar features, image features, and text features are mapped to the same feature space. In some examples, radar features and image features are mapped to one feature space, and image features and text features are mapped to another feature space, thus allowing the text features corresponding to the text to be determined based on the radar features using the image features.

[0037] like Figure 3As shown, box 300B represents the architecture for object detection during the inference phase, including an extraction module 324 and a determination module 328. The extraction module 324 determines the radar point cloud data 326 of the target object based on the radar point cloud input data 322. In some examples, the extraction module 324 may be configured with a radar RPN model, which extracts the radar point cloud data 326 of the target object from the radar point cloud input data 322. In other examples, the extraction module may be configured with an image RPN model, which extracts the target region of the target object from the input image paired with the radar point cloud input data 322, and then obtains the radar point cloud data 326 of the target object through cropping based on the target region of the target object and the radar point cloud input data 322. The determination module 328 is configured with a radar feature extraction model (e.g., radar feature extraction model 312) trained according to the above training architecture. Based on the radar point cloud data 326 of the target object, the radar feature extraction model outputs the radar features 330 of the target object. When there are multiple target objects, the radar features of multiple target objects are output. During the training phase, the radar feature extraction model has been trained to extract radar features aligned with the features generated from the corresponding text. Therefore, during inference, based on the extracted radar features 330, the text corresponding to the aligned text features can be determined. Subsequently, radar point cloud output data 332, identifying all target objects with textual identifiers, is output. As shown in the example, based on the radar point cloud data collected by the radar detection device, target objects within the radar detection range can be identified as people (P), babies (B), trucks (T), cars (C), sports cars (S), and vans (V).

[0038] In this way, by training radar features to align with image features, the radar features can be aligned with the corresponding text features generated from the text within the same feature space based on the alignment relationship between the image and the text. Alternatively, radar features can be aligned with image features in one feature space, and image features with text features in another feature space. Image features can be used as an intermediary to determine the aligned text features based on the radar features. Thus, using the CLIP model with pre-aligned text and images, only the radar feature extraction model needs to be trained. Without requiring additional labeled datasets, radar features can be aligned with text features through image features as an intermediary, expanding the object categories in radar detection technology and enabling accurate identification of multiple object types.

[0039] Figure 4A schematic diagram of the principle process 400 of a radar feature extraction model in the training phase of some embodiments of the present disclosure is shown. Process 400 may include training a radar feature extraction model 404 based on training samples. The training samples include multiple pairs of sample images and sample radar point cloud data 402-1. In some implementations, an image 402-2 of a sample object is extracted based on each sample image. In some examples, a sample region of the sample object in the sample image is extracted via an image RPN, and then the image of the sample object in the sample image is obtained by cropping. In some implementations, the radar feature extraction model 404 determines the radar features of the sample object based on the sample radar point cloud data 402-1. In some examples, a sample region of the sample object in the sample image is extracted via an image RPN, and the radar point cloud data of the sample object is determined based on the sample region of the sample object and the sample radar point cloud data paired with the sample image. Then, the radar feature extraction model determines the radar features of the sample object based on the radar point cloud data of the sample object. In other examples, sample regions of sample objects in sample images are extracted using image RPN. Based on sample radar point cloud data, sample features are determined by a radar feature extraction model. Then, the sample regions of the sample objects are projected onto the sample radar features to determine the radar features of the sample objects.

[0040] like Figure 4 The radar point cloud data 402-1 is input into the radar feature extraction model 404, and the output is the radar feature 406 of the sample object. When there is radar point cloud data of multiple sample objects, multiple radar features 406 are extracted accordingly, namely L1 to L2. N The input image 402-2 and the image encoder of the text matching model 408 output image features 410. When there are multiple images of sample objects, multiple image features 410 are output, namely I1 to I2. N The image feature 410 has been aligned with the corresponding text in the feature space by the image and text matching model 408. The image and text matching model 408 has been pre-trained. The pre-training process includes training based on paired image and text datasets. After training, the image features and text features are aligned in the feature space (see...). Figure 4(The feature space shown next to the middle frame 410). In some examples, the image and text matching model 408 can use the existing CLIP model. In some examples, it can be fine-tuned based on the existing CLIP model so that the alignment of the image and text is adjusted based on the updated text. When training the radar feature extraction model 404, the image features 410 of the sample object are used as a supervision signal to train the radar feature extraction model so that the radar features of the sample object are aligned with the text features generated by the corresponding text. In some implementations, the loss calculation module 414 calculates a loss value based on the input radar features 406 and image features 410 of the sample object. When the loss value is large, the parameters of the radar feature extraction model 404 are adjusted in reverse until the radar features and image features are aligned in the feature space. Through the image features of the sample object, the text features and radar features that are respectively aligned with it can be mapped to the same feature space so that the text features and radar features of the sample object are aligned in the feature space 412. For example, a CLIP-like model or a MOCO-like model can be used to align the radar features and text in the feature space.

[0041] In this way, after training the radar to align with the image, the alignment relationship between the radar and the text can be learned by learning the alignment relationship between the image and the text. This process only requires training the radar feature extraction model, which has a simple training architecture and does not rely on labeled datasets. It can quickly converge by calculating the similarity between radar features and image features using similarity loss, and can quickly and accurately identify various unknown objects in the open world.

[0042] Figure 5 A schematic diagram of a radar feature determination process 500 during the training phase, according to some embodiments of this disclosure, is shown. This process includes determining multiple voxels 506 based on sample radar point cloud data 502. The radar point cloud data 502 is rasterized in 3D space, with each raster corresponding to a certain number of points. Then, a radar feature extraction model 508 extracts features from the multiple voxels 506 to obtain voxelized features 510, thereby determining the sample radar features. In some examples, the radar feature extraction model 508 may be a multilayer perceptron, extracting features from the multiple voxels to obtain voxelized features 510. Then, the sample region 512 of the sample object is projected onto the voxelized features 510 to obtain the radar features 514 of the sample object.

[0043] Furthermore, in some implementations, the process of determining the radar features of a sample object may include determining the radar point cloud data of the sample object. Then, based on the radar point cloud data of the sample object, the radar features of the sample object are determined. Correspondingly, the extraction of radar features of the target object during the inference process can also be obtained by referring to one of the above implementations.

[0044] In this way, radar features projected onto the sample area can be extracted, and the type of sample object in the radar point cloud data can be effectively identified based on the radar features of the sample area.

[0045] Figure 6 The diagram illustrates a radar-text pairing process 600 during the radar extraction model training phase, as shown in some embodiments of this disclosure. Process 600 may include extracting radar features 612-1,…,612-N of the sample object from radar point cloud data 608 using a radar feature extraction model 610. Process 600 may also include determining text features aligned with the radar features of the sample object from the feature space, i.e., pairing text and radar data within the feature space. Within this feature space, the image features and text features of the object have been pre-aligned using an image-text matching model, and the radar features and image features of the object are aligned during the radar extraction model training phase. Thus, text features aligned with radar features can be determined from image features, thereby determining the corresponding text. In some examples, the text feature vector and radar feature vector with the minimum cosine distance are matched within the feature space, and the matching result is determined as the text feature aligned with the radar feature. Figure 6 As shown in the example, radar feature 612-1 of sample object 1 is paired with feature 606-2 of the generated text, and radar feature 612-N of sample object N is paired with feature 606-N of the generated text. Accordingly, this process also applies to the pairing of radar features and text features of target objects during the inference process.

[0046] The image and text matching model can be pre-trained on a dataset of hundreds of millions of image-text pairs. Images and text 602 are encoded into feature vectors by an image encoder and a text encoder 604, respectively. The degree of matching between the image and text is then evaluated by calculating the cosine similarity between these feature vectors. After training, the model parameters are fixed, and then the model is used in process 600. During process 600, the image and text matching model parameters are locked. When fine-tuning the image and text matching model is needed, the text encoder can be adjusted considering computational costs. After adjusting the text encoder, it can be expanded to correspond to more text categories. Embodiments of this disclosure can identify new categories for previously unrecognized target objects and can also identify more refined categories for previously recognized target objects.

[0047] In this way, radar matches text with images as an intermediary, identifying the object's type based on radar point cloud data. This allows radar object detection technology to learn from matched text and images, thus learning more object types. During reasoning, it can accurately identify more unknown objects.

[0048] Figure 7 The diagram illustrates an example inference process 700 for object detection using a trained radar feature extraction model, according to some embodiments of the present disclosure. This process 700 may include a determination module 704 determining radar point cloud data 706 of a target object based on radar point cloud input data 702. The process 700 may also include an extraction module 708, configured with a trained radar feature extraction model, outputting radar features 710 based on the radar point cloud data 706 of the target object. Furthermore, the process may also include a generation module determining corresponding text-generated features aligned with the radar features from the feature space based on the radar features 710, thereby generating radar point cloud output data 714 of the target object with text labels.

[0049] During the training phase, radar features are aligned with corresponding image features in the feature space, and image features are aligned with corresponding text features. Based on this, in the inference phase, without using image and text models, the trained radar feature extraction model outputs radar features, which can then be used to find text features aligned with the radar features in the feature space, thereby determining the text corresponding to the text features. This allows for rapid response to radar point cloud input data and outputs detection results that identify various categories of target objects.

[0050] In an example test, the detection results (radar point cloud data of the identified text) obtained by the radar detection scheme of this disclosure embodiment are compared and verified with the labeled dataset (which contains radar point cloud data of manually labeled object categories). Taking a stroller as an example, the scores obtained by multiple radar point cloud data for the category of pedestrian (the average of the model's detection precision and recall) are all below 0.34. It is evident that this disclosure embodiment has high accuracy in identifying the target object as a stroller, while the possibility of misidentifying it as a pedestrian is low. This disclosure embodiment can obtain the radar point cloud data of the target object using radar RPN (this document can also utilize the extraction of the target region of the image paired with radar to obtain the radar point cloud data of the target object). This allows for the location of the target object to be located first, and then the category of the target object at this location can be identified. This enables relatively accurate identification of the object category based on the target object's location, with a low misidentification rate. It is worth noting that existing radar detection devices do not need to identify the object category when performing target detection and identification, and therefore do not need to obtain or utilize the target region of the object. However, this disclosure embodiment can utilize the target region to identify the category of the target object in the corresponding region. When the radar feature extraction model is based on the input radar point cloud data, it can output radar point cloud data of target objects containing identifiable text, that is, expand the target area of ​​the target object containing location information to the open world category.

[0051] Figure 8 Example block diagrams of an object detection apparatus 800 according to some embodiments of the present disclosure are shown. The apparatus 800 includes a determination module 802 configured to determine radar point cloud data of a target object based on radar point cloud input data. The apparatus 800 also includes an extraction module 804 configured to extract radar features of the target object based on the radar point cloud data of the target object. Furthermore, the apparatus 800 includes a generation module 806 configured to generate radar point cloud output data with textual identifiers of the target object based on the radar features of the target object.

[0052] In some embodiments, the determining module 802 is configured to determine the target area of ​​a target object based on radar point cloud input data, and to determine radar point cloud data of the target object based on the target area of ​​the target object. In some embodiments, the generating module 806 is configured to determine text features aligned with the radar features from a feature space based on the radar features of the target object, and to generate radar point cloud output data that identifies the target object with text based on the text features aligned with the radar features, wherein the text features correspond to the text.

[0053] It is understood that the device 800 of this disclosure can achieve at least one of the many advantages that the method or process described above can achieve. For example, the device 800 can identify the radar features of a target object based on radar point cloud data collected by a radar detection device, and can determine the corresponding text-generated features that correspond to the radar features from the feature space, and finally output radar point cloud data labeled with the category of the target object. This device can quickly respond to radar point cloud data without relying on a labeled dataset, and can accurately identify the type of target object in various scenarios.

[0054] Figure 9 A schematic block diagram of an example device 900 that can be used to implement embodiments of the present disclosure is shown. Device 900 may include the control unit from the foregoing method embodiments. Figure 9 As shown, device 900 includes a processor 901, which can perform various appropriate actions and processes based on computer program instructions loaded into random access memory (RAM) 903 according to computer program instructions stored in read-only memory (ROM) 902. RAM 903 may also store various programs and data required for the operation of device 900. The processor 901, ROM 902, and RAM 903 are interconnected via bus 904. Input / output (I / O) interface 905 is also connected to bus 904.

[0055] The various processes and procedures described above, such as method 200, can be executed by processor 901. For example, in some embodiments, method 200 may be implemented as a software program tangibly contained in a machine-readable medium. In some embodiments, part or all of the software program may be loaded and / or installed on device 900 via ROM 902. When the software program is loaded into RAM 903 and executed by processor 901, one or more actions of method 200 described above may be performed.

[0056] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload programmable logic devices (CPLDs), and so on.

[0057] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0058] This disclosure can be a method, apparatus, system, and / or program product. The program product may include a machine-readable storage medium on which machine-readable program instructions for performing various aspects of this disclosure are loaded. The machine-readable program instructions described herein can be downloaded from the machine-readable storage medium to various computing / processing devices, or downloaded via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the machine-readable program instructions from the network and forwards them to the machine-readable storage medium in the respective computing / processing device.

[0059] Machine program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. Machine-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as a standalone software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the machine-readable program instructions to implement various aspects of this disclosure.

[0060] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. Furthermore, although operations are depicted in a specific order, this should be understood as requiring that such operations be performed in the specific order shown or in sequential order, or requiring that all illustrated operations be performed to achieve the desired result. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the foregoing discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented individually or in any suitable sub-combination in multiple implementations.

[0061] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. An object detection method (200), comprising: Based on the radar point cloud input data, determine the radar point cloud data of the (202) target object; Based on the radar point cloud data of the target object, extract (204) the radar features of the target object; as well as Based on the radar characteristics of the target object, generate (206) radar point cloud output data with textual identification of the target object.

2. The method (200) according to claim 1, wherein determining (202) the radar point cloud data of the target object based on the radar point cloud input data includes: Based on the radar point cloud input data, the target area of ​​the target object is determined; as well as Based on the target area of ​​the target object, the radar point cloud data of the target object is determined.

3. The method (200) according to claim 1 or 2, wherein generating (206) radar point cloud output data with textual identification of the target object based on the radar features of the target object comprises: Based on the radar features of the target object, text features aligned with the radar features are determined in the feature space, wherein the radar features of the corresponding object and the text features of the corresponding object are aligned in the feature space; as well as Based on the text features aligned with the radar features, radar point cloud output data is generated that identifies the target object with text, wherein the text corresponds to the text features.

4. The method (200) according to claim 1, wherein extracting (204) the radar features of the target object based on the radar point cloud data of the target object includes: Based on the radar point cloud data of the target object, a trained radar feature extraction model is used to extract radar features of the target object. The training process of the radar feature extraction model includes training with an image and text matching model as the teacher model to obtain radar features aligned with the text features generated by the corresponding text.

5. The method (200) according to claim 4, wherein the training process of the radar feature extraction model includes: Acquire training samples, which include multiple pairs of sample images and sample radar point cloud data; Based on the sample image, extract the image of the sample object; Based on the sample radar point cloud data paired with the sample image, the radar feature extraction model determines the radar features of the sample object; Based on the image of the sample object, the image features of the sample object are determined by the image and text matching model; and Using the image features of the sample object as a reference, the radar feature extraction model is trained so that the radar features of the sample object are aligned with the text features generated from the corresponding text.

6. The method (200) according to claim 5, wherein determining the radar features of the sample object by the radar feature extraction model based on the sample radar point cloud data paired with the sample image includes: Based on the sample image, determine the sample region of the sample object; Based on the sample region of the sample object and the sample radar point cloud data paired with the sample image, the radar point cloud data of the sample object is determined; as well as The radar feature extraction model determines the radar features of the sample object based on the radar point cloud data of the sample object.

7. The method (200) according to claim 5, wherein determining the radar features of the sample object by the radar feature extraction model based on the sample radar point cloud data paired with the sample image comprises: Based on the sample image, determine the sample region of the sample object; Based on the sample radar point cloud data, the radar feature extraction model determines the sample radar features; as well as The radar features of the sample object are determined by projecting the sample region of the sample object onto the sample radar features.

8. The method (200) according to claim 7, wherein determining the sample radar features by the radar feature extraction model based on the sample radar point cloud data includes: The radar feature extraction model determines multiple voxels based on the sample radar point cloud data; as well as The radar feature extraction model determines the sample radar features by extracting voxelized features based on the multiple voxels.

9. The method (200) according to claim 5, wherein determining the image features of the sample object based on the image of the sample object by the image and text matching model comprises: Based on the image of the sample object, the image encoder of the image and text matching model determines the image features of the sample object, wherein the image features are aligned with the text features in the feature space.

10. The method (200) according to claim 5, wherein training the radar feature extraction model with reference to the image features of the sample object, such that the radar features of the sample object are aligned with the text features generated from the corresponding text, comprises: Based on the image features and radar features of the sample object, the loss value is calculated using a loss function; as well as Based on the loss value, the parameters of the radar feature extraction model are adjusted so that the radar features are aligned with the image features, and then with the text features.

11. The method (200) according to claim 10, further comprising: Using the image features of the sample object, the text features and radar features that are respectively aligned with the image features are aligned in the feature space.

12. An object detection device (800), comprising: The determination module (802) is configured to determine the radar point cloud data of the target object based on the radar point cloud input data; The extraction module (804) is configured to extract radar features of the target object based on the radar point cloud data of the target object; as well as The generation module (806) is configured to generate radar point cloud output data that identifies the target object with text based on the radar features of the target object.

13. An electronic device (900), comprising: At least one processor (901); as well as A memory (902) coupled to the at least one processor and having instructions stored thereon, which, when executed by the at least one processor, cause the electronic device to perform the method according to any one of claims 1 to 11.

14. A computer program product comprising a computer program that is executed by a processor to implement the method according to any one of claims 1 to 11.

15. A machine-readable storage medium storing machine-executable instructions, wherein the machine-executable instructions are executed by a processor to implement the method according to any one of claims 1 to 11.

Images