System and apparatus suitable for facilitating object detection, and a processing method in association thereto

The 4D fusion of LiDAR and camera data in autonomous vehicles enhances object detection accuracy and efficiency by synchronizing and aligning 3D point cloud and RGB data, addressing the limitations of conventional data collection techniques.

WO2024240483A1PCT designated stage expired Publication Date: 2024-11-28CONTINENTAL AUTOMOTIVE TECHNOLOGIES GMBH
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
PCT/EP2024/062507
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-22
Filing Date
2024-05-07
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

Conventional data collection techniques for object detection in autonomous vehicles, such as periodic sensor measurements, are not optimal or sufficient for accurately detecting objects in external environments, particularly at distant ranges or with insufficient point cloud data.

Method used

A processing method that combines data from multi-modal sensors like LiDAR and cameras using 4D fusion, where 3D point cloud data from LiDAR is synchronized and fused with RGB data from cameras, enabling geometric alignment and feature extraction through Point Diffusion-Refinement and self-attention mechanisms to enhance object detection accuracy.

Benefits of technology

This approach facilitates more accurate and efficient object detection by integrating temporal information, improving the detection of distant objects and maintaining computational efficiency, thereby addressing the limitations of conventional methods.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure EP2024062507_28112024_PF_FP_ABST
    Figure EP2024062507_28112024_PF_FP_ABST
Patent Text Reader

Abstract

There is provided a processing method (300) suitable for facilitating object detection. The processing method (300) can include a processing step (304) wherein one or more processing tasks can be performed in association with initial processing, preliminary processing and / or fusion-based processing. The initial processing can, include calibrating and / or synchronizing one or more input signals which can be associated with data associated with a first sensor type and data associated with a second sensor type. The preliminary processing can include preliminary processing of data associated with a first sensor type to generate a first set of preliminary signals and preliminary processing of data associated with a second sensor type to generate a second set of preliminary signals. The fusion-based processing can include fusing the first set of preliminary signals and the second set of preliminary signals in time.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] SYSTEM AND APPARATUS SUITABLE FOR FACILITATING OBJECT DETECTION, AND A PROCESSING METHOD IN ASSOCIATION THERETO

[0002] Field Of Invention

[0003] The present disclosure generally relates to one or both of a system and an apparatus suitable for facilitating object detection (e.g., detection of at least one object). The present disclosure further relates a processing method which can be associated with the system and / or the apparatus.

[0004] Background

[0005] For a vehicle such as an autonomous vehicle, data can be collected by one or more sensors (e.g., a camera) to, for example, ascertain environment external to the vehicle.

[0006] Conventional techniques for data collection can, for example, be based on collection of data periodically. In a specific example, data collected can be based on sensor measurement recorded at regular time intervals to attempt to obtain an accurate representation of the external environment.

[0007] The present disclosure contemplates that data collection based on conventional technique(s) may not be optimal and / or sufficient in regard to object detection (i.e. , detection of one or more objects).

[0008] The present disclosure contemplates that it be desirable to address (or, at least partially address) at least one issue associated with the conventional technique(s).

[0009] Summary of the Invention

[0010] In accordance with an aspect of the disclosure, there is provided a processing method which can, for example, be suitable for facilitating object detection. The processing method can include a processing step. In regard to the processing step, one or more processing tasks can, for example, be performed in association with any one of, or any combination of, initial processing, preliminary processing and fusion-based processing (i.e., initial processing, preliminary processing and / or fusion-based processing; at least one of initial processing, preliminary processing and fusion-based processing).

[0011] The initial processing can, for example, include one or both of calibrating and synchronizing (at least one of calibrating and synchronizing; calibrating and / or synchronizing) one or more input signals which can be associated with data associated with a first sensor type and data associated with a second sensor type, in accordance with an embodiment of the disclosure. In one example, one or both of the first sensor type and the second sensor type (i.e., at least one of the first sensor type and the second sensor type) can be carried by a vehicle, and object detection can, for example, correspond to detection of at least one object associated with an environment external to the vehicle.

[0012] The preliminary processing can, for example, include preliminary processing of data associated with a first sensor type (e.g., at least one LiDAR sensor) to generate a first set of preliminary signals and preliminary processing of data associated with a second sensor type (e.g., at least one camera sensor) to generate a second set of preliminary signals, in accordance with an embodiment of the disclosure. In one example, data associated with the first sensor type can correspond to point cloud data and data associated with the second sensor type can correspond to RGB (Red Green Blue) data. In one specific example, point cloud data can be preliminarily processed to generate one or more point cloud feature maps and RGB data can be preliminarily processed to generate one or more RGB feature maps. In one specific example, preliminary processing point cloud data to generate at least one point cloud feature map may comprise at least one of and / or any combination of: processing point cloud data, preferably by manner of Point Diffusion-Refinement (PDR) paradigm; including at least one time-based parameter; encoding point cloud data to create at least one pseudo image; generating at least one point cloud feature map. In one specific example, preliminary processing RGB data to generate at least one RGB feature map may comprise at least one of and / or any combination of: generating at least one reference source; extracting at least one RGB feature map. In yet a more specific example, the point cloud feature map(s) and the RGB feature map(s) can be capable of being fused in time (e.g., time-based fusing of the point cloud feature map(s) and the RGB feature map(s)). In one specific example, the point cloud feature map and the RGB feature map may be fused in time by at least one of and / or any combination of: using projections to enable geometric alignment; performing connection architecture search in 4D, preferably by differentiable architecture search with selfattention mechanisms. In yet a more specific example, using projections to enable geometric alignment comprises at least one of and / or any combination of: for each point pillar location, obtaining at least one 2D point; enhancing pillar features by manner of learning fine-grained correlation among points; concatenating the enhanced pillar features and the obtained at least one 2D point.

[0013] The fusion-based processing can, for example, include fusing the first set of preliminary signals and the second set of preliminary signals in time, in accordance with an embodiment of the disclosure. For example, in one embodiment, data associated with the first sensor type can be independently preliminarily processed from data associated with the second sensor type.

[0014] In one embodiment, the processing method can, for example, further include an input step wherein one or more one input signals can be received.

[0015] In one embodiment, the processing method can, for example, further include an output step wherein one or more output signals (e.g., which can be generated based on fusion-based processing) can be communicated. For example, the output signal(s) can be associated with at least one 3D (three-dimension) indication to indicate at least one detected object.

[0016] The present disclosure further contemplates a computer program (not shown) which can include instructions which, when the program is executed by a computer (not shown), cause the computer to carry out the input step, the processing step and / or the output step of the processing method.

[0017] The present disclosure yet further contemplates a computer readable storage medium (not shown) having data stored therein representing software executable by a computer (not shown), the software including instructions, when executed by the computer, to carry out the input step, the processing step and / or the output step of the processing method.

[0018] In the above manner, it is contemplated that data collection can be provided / facilitated in a comparatively more optimal, comparatively more sufficient manner and / or comparatively more efficient manner for, for example, facilitation of object detection, in accordance with an embodiment of the disclosure. Specifically, it is contemplated that fusing / combining data from multi-modal sensors in 4D can possibly facilitate object detection (e.g., detection of one or more objects in an environment which can, for example, be external to a vehicle and / or detection of one or more objects at distant ranges) in a (comparatively more) sufficient, optimal and / or efficient manner, in accordance with an embodiment of the disclosure. It is also contemplated that the efficient processing of sensor data may retain more information which may better facilitate object detection, in accordance with an embodiment of the disclosure.

[0019] The above-described advantageous aspect(s) of the processing method of the present disclosure can also apply analogously (all) the aspect(s) of a below described apparatus of the present disclosure. Likewise, all below described advantageous aspect(s) of the apparatus of the disclosure can also apply analogously (all) the aspect(s) of above-described processing method of the disclosure.

[0020] In accordance with an aspect of the disclosure, there is provided an apparatus which can, for example, be suitable for facilitating object detection (e.g., detection of one of more objects), in accordance with an embodiment of the disclosure. Moreover, the apparatus can, for example, correspond to an electronic module (e.g., which can be carried by a vehicle such as an autonomous vehicle), in accordance with an embodiment of the disclosure.

[0021] The apparatus can, for example, include a first module, a second module and a third module, in accordance with an embodiment of the disclosure.

[0022] In one embodiment, the first module (e.g., corresponding to a receiver) can, for example, be configured to receive at least one input signal (e.g., communicable from at least one device such as at least one LiDAR sensor and / or at least one camera sensor). The second module (e.g., corresponding to a processor) can, for example, be configured to one or both of process and facilitate processing of (i.e., process and / or facilitate processing of; at least one of process and facilitate processing of) the input signal(s) according to the processing method, in accordance with an aspect of the disclosure, to generate at least one output signal. The third module (e.g., corresponding to a transmitter) can, for example, be configured to communicate the output signal(s).

[0023] In one embodiment, the input signal(s) can be associated with data associated with a first sensor type (e.g., at least one LiDAR sensor) and data associated with a second sensor type (e.g., at least one camera sensor). Moreover, the first sensor type and the second sensor type can, for example, be carried by a vehicle, and object detection can, for example, correspond to detection of at least one object associated with an environment external to the vehicle.

[0024] In the above manner, it is contemplated that data collection can be provided / facilitated in a comparatively more optimal, comparatively more sufficient manner and / or comparatively more efficient manner for, for example, facilitation of object detection, in accordance with an embodiment of the disclosure. Specifically, it is contemplated that fusing / combining data from multi-modal sensors in 4D can possibly facilitate object detection (e.g., detection of one or more objects in an environment which can, for example, be external to a vehicle) in a (comparatively more) sufficient, optimal and / or efficient manner, in accordance with an embodiment of the disclosure.

[0025] Brief Description of the Drawings

[0026] Embodiments of the disclosure are described hereinafter with reference to the following drawings, in which:

[0027] Fig. 1 shows a system which can include at least one apparatus, according to an embodiment of the disclosure; Fig, 2 shows the apparatus of Fig, 1 in further detail, according to an embodiment of the disclosure; and

[0028] Fig. 3 shows a processing method in association with the system of Fig. 1 , according to an embodiment of the disclosure.

[0029] Detailed Description

[0030] The present disclosure contemplates a vehicle such as an autonomous vehicle that can, for example, carry one or more sensors. In one example, a vehicle can carry a plurality of sensors for sensing external environment (e.g., environment external to the vehicle). The plurality of sensors can, for example, include one or more sensor types. For example, the sensor types can include a first sensor type and a second sensor type. The first sensor type can, for example, be different from the second sensor type, in accordance with an embodiment of the disclosure. The first sensor type can, for example, correspond to / include / be associated with LiDAR (light detection and ranging) based sensor(s) (e.g., simply referable to as “LiDAR sensor(s)”), in accordance with an embodiment of the disclosure. The second sensor type can, for example, correspond to / include / be associated with camera based sensor(s) (e.g., simply referable to as “camera sensor(s)”) such as RGB (red green blue) wavelengths based sensors (e.g., simply referable to as “RGB camera(s)”), in accordance with an embodiment of the disclosure.

[0031] Generally, data can be obtained from the sensor(s) carried. For example, point cloud data can be obtained from the LiDAR sensor(s) and RGB data can be obtained from the RGB camera(s).

[0032] The present disclosure contemplates that point cloud data may possibly be insufficient (e.g., a faraway object located at a distance range which is so far that insufficient points are reflected by the faraway object and / or for a small object which is so small in size that insufficient points are reflected by the small object). Point cloud data can, for example, be referable to 3D (three-dimensional) point cloud input(s) where the context is appropriate, in accordance with an embodiment of the disclosure. The present disclosure contemplates the possibility that RGB data (e.g., which may possibly contain more information as compared with point cloud data) can be combined / fused with the 3D Point Cloud input(s).

[0033] Specifically, the present disclosure contemplates the possibility of improved (e.g., more accurate) detection of one or more objects (e.g., 3D objects) via fusion of data from multi-modal sensors (e.g., a combination of a first sensor type such as LiDAR sensor(s) and a second sensor type such as RGB camera(s)).

[0034] The present disclosure further contemplates the possibility that fusion of data communicated from / generated by multi-modal sensors (e.g., using both the first and second sensor types simultaneously), may increase complexity (e.g., pairing the information associated with detected data from the first sensor type to information associated with detected data from the second sensor type may add further complexity) since a direct correspondence (e.g., detected data from the first sensor type and detected data from the second sensor type) as between data communicated from / generated by the multi-modal sensors may not be available. Moreover, computational efficiency may possibly be adversely impacted. Furthermore, it is contemplated that fusing already mature features which may have lost spatial information may possibly result in 3D object detection which may not be optimal / ideal (e.g., less accurate). Appreciably, increase in complexity, adverse impact to computational efficiency and / or loss (e.g., lost spatial information) may, for example, adversely impact efficiency and / or optimality.

[0035] The present disclosure generally contemplates the possibility of fusing information in 4D (i.e., 4 dimensions where, for example, 3 dimensions can be associated with 3D representation of one or more detected objects in the external environment and the 4thdimension can be time-based) which may address / mitigate the abovementioned increase in complexity, adverse impact to computational efficiency and / or loss generally associated with fusion of multi-modal sensors. For example, information from multi-modal sensors (e.g., from both the first and second sensor types) can be fused / combined in 4D (i.e., the 4th dimension being a time-based parameter “t”). In a more specific example, 3D point cloud inputs from at least one LiDAR sensor and RGB data from at least one RGB camera can be fused / combined in association with time in a manner so as to facilitate building and / or learning of one or more connections between feature representations from different modalities and levels of abstraction. It is completed that in doing so, direct correspondence by possibly be capable of being established, in accordance with an embodiment of the disclosure.

[0036] In accordance with an embodiment of the disclosure, it is contemplated that sensor data from each sensor type (e.g., the first and second sensor types) can be processed in a manner such that one or more rich features can be obtained, at least one additional input stream can be generated (e.g., from RGB frames) and can be combined with point cloud in time to possibly facilitate effective object detection (e.g., detect faraway objects effectively) and / or point cloud data can be fused / combined with RGB data, in time. The point cloud data and the RGB data can, for example, be fused / combined (in time) in an efficient and learnable manner, in accordance with an embodiment of the disclosure.

[0037] In the above manner, it is contemplated that data collection can be provided / facilitated in a comparatively more optimal, comparatively more sufficient manner and / or comparatively more efficient manner for, for example, facilitation of object detection, in accordance with an embodiment of the disclosure. Specifically, it is contemplated that fusing / combining data from multi-modal sensors in 4D can possibly facilitate object detection (e.g., detection of one or more objects in an environment which can, for example, be external to a vehicle) in a (comparatively more) sufficient, optimal and / or efficient manner, in accordance with an embodiment of the disclosure.

[0038] The foregoing will be discussed in further detail with reference to Fig. 1 to Fig. 3 hereinafter.

[0039] Referring to Fig. 1 , a system 100 is shown, according to an embodiment of the disclosure. The system 100 can, for example, be suitable for facilitating object detection (e.g., detection of one of more objects), in accordance with an embodiment of the disclosure. For example, the system 100 can, for example, be capable (e.g., configured to) of data collection and / or processing collected data in a manner so as to facilitate object detection, in accordance with an embodiment of the disclosure. The object(s) can, for example, correspond to / include / be associated with 3D object(s), in accordance with an embodiment of the disclosure. Moreover, the object(s) can, for example, be associated with / be located in an environment. The environment can, for example, be external to a vehicle such as an autonomous vehicle, in accordance with an embodiment of the disclosure.

[0040] Moreover, the system 100 can, for example, be carried by a vehicle (not shown), in accordance with an embodiment of the disclosure. In accordance with another embodiment, the system 100 can, for example, be partially carried by a vehicle (e.g., a portion of the system 100 can be carried by the vehicle and another portion of the system 100 can be carried external to the vehicle). In yet another embodiment, at least one portion of the system 100 can, for example, be carried by a vehicle.

[0041] As shown, the system 100 can include one or more apparatuses 102, one or more device 104 and, optionally, at least one communication network 106, in accordance with an embodiment of the disclosure.

[0042] The apparatus(es) 102 can be coupled to the device(s) 104. Specifically, the apparatus(es) 102 can, for example, be coupled to the device(s) 104 via the communication network 106, in accordance with an embodiment of the disclosure.

[0043] In one embodiment, the apparatus(es) 102 can be coupled to the communication network 106 and the device(s) 104 can be coupled to the communication network 106. Coupling can be by manner of one or both of wired coupling and wireless coupling. The apparatus(es) 102 can, in general, be configured to communicate with the device(s) 104 via the communication network 106, according to an embodiment of the disclosure.

[0044] Generally, in accordance with an embodiment of the disclosure, the apparatus(es) 102 can be configured to receive one or more input signals and process (and / or facilitate processing of) the input signal(s) to gen erate / d erive one or more output signals. Moreover, in accordance with an embodiment of the disclosure, the device(s) 104 can, for example, be configured to one or both of generate the input signal(s) and communicate the input signal(s) to the apparatus(es) 102.

[0045] The apparatus(es) 102 can, for example, be configured to process and / or facilitate processing of the input signal(s) to generate / derive / produce the output signal(s), in accordance with an embodiment of the disclosure. For example, the apparatus(es) 102 can be configured to process (and / or facilitate processing of) the input signal(s) by manner of any one of initial processing, preliminary processing and fusion-based processing, or any combination thereof (i.e. , initial processing, preliminary processing and / or fusion-based processing) to generate / derive / produce the output signal(s). In one embodiment, fusion-based processing can, for example, be in 4D (e.g., fusionbased processing in 4D) so as to possibly facilitate object detection (e.g., detection of one or more objects in an environment which can, for example, be external to a vehicle) in a (comparatively more) sufficient, optimal and / or efficient manner, in accordance with an embodiment of the disclosure. The apparatus(es) 102 will be discussed later in further detail (e.g., in the context of an example scenario and / or with reference to Fig. 2), in accordance with an embodiment of the disclosure

[0046] The device(s) 104 can, for example, be configured to generate the input signal(s) and / or communicate the input signal(s), in accordance with an embodiment of the disclosure. For example, the input signal(s) can be communicated from the device(s) 104 to the apparatus(es) 102. The device(s) can, for example, be configured to detect one or more objects located in / associated with an environment (e.g., external environment to a vehicle) and generate one or more input signals based on detection of the object(s), in accordance with an embodiment of the disclosure. In one example, the system 100 can include a plurality of devices 104 which can, for example, include / be associated with / correspond to at least one device associated with / corresponding to a first device type and at least one device associated with / corresponding to a second device type, in accordance with an embodiment of the disclosure. The first device type can, for example, correspond to a first sensor type and the second device type can, for example, correspond to a second sensor type, in accordance with an embodiment of the disclosure. In one embodiment, the first sensor type can be different / distinct from the second sensor type. The device(s) 104 will be discussed later in further detail with reference to an example scenario, in accordance with an embodiment of the disclosure.

[0047] The communication network 106 can, for example, correspond to an Internet communication network, a wired-based communication network, a wireless-based communication network, or any combination thereof. Communication (i.e., between the apparatus(es) 102 and the device(s) 104) via the communication network 106 can be by manner of one or both of wired communication and wireless communication.

[0048] The foregoing will now be discussed in the context of an example scenario, in accordance with an embodiment of the disclosure.

[0049] In the example scenario, the vehicle can correspond to an autonomous vehicle carrying at least one apparatus 102 and a plurality of devices 104. The apparatus 102 and the devices 104 can be coupled in one or both of a wired manner and wirelessly (i.e., in a wired manner and / or wirelessly via the communication network 106 (e.g., a control area network). The devices 104 can, for example, include one or more devices associated with a first device type (e.g., a first sensor type) and a second device type (e.g., a second sensor type), in accordance with an embodiment of the disclosure. The first sensor type can, for example, correspond to LiDAR sensor(s) and the second sensor type can, for example, correspond to RGB camera(s), in accordance with an embodiment of the disclosure. In this regard, it is appreciable that the plurality of devices 104 can include / be associated with / correspond to at least one LiDAR sensor and at least one RGB camera. Moreover, by manner of sensing / detecting one or more objects in an environment external to the autonomous vehicle, point cloud data can be obtained from the LiDAR sensor(s) and RGB data can be obtained from the RGB camera(s).

[0050] In this regard, in the example scenario, the input signal(s) communicated from the devices 104 can include / correspond to / be associated with point cloud data and RGB data. The apparatus(es) 102 can, in accordance with an embodiment of the disclosure, be configured to receive, and process and / or facilitate processing of the input signal(s) in a manner as will be discussed in the context of a specific example scenario hereinafter.

[0051] In the specific example scenario, point cloud data and RGB data can, for example, be calibrated and / or synchronized. For example, the apparatus(es) 102 can, as an option, be configured to perform one or more initial processing tasks associated with one or both of calibrating and synchronizing (e.g., the point cloud data and the RGB data), in accordance with an embodiment of the disclosure. The point cloud data and the RGB data can, for example, be associated with LiDAR video frame(s) and RGB video frame(s) respectively, in accordance with an embodiment of the disclosure. The apparatus(es) 102 can, for example, be configured to process the LiDAR and RGB video frames separately (e.g., the LiDAR video frame(s) can be processed independently from the RGB video frame(s)), in accordance with an embodiment of the disclosure.

[0052] In regard to the LiDAR video frame(s), in the context of the specific example scenario, the apparatus(es) 102 can be configured to process and / or facilitate processing of point cloud data based on / by manner of Point Diffusion-Refinement (PDR) paradigm (e.g., an example of network architecture and / or training procedure can be based on Section B EXPERIMENT of “A Conditional Point Diffusion-Refinement Paradigm for 3D Point Cloud Completion” by Zhaoyang Lyu, Zhifeng Kong, Xudong Xu, Liang Pan and Dahua Lin). It is contemplated that processing by manner of PDR paradigm can, for example, facilitate generating uniform and / or high quality complete point clouds, in accordance with an embodiment of the disclosure. It is contemplated that point cloud data can be indicative of a point cloud which can include / be associated with a plurality of points and each point in the point cloud can be associated with a feature, in accordance with an embodiment of the disclosure. Moreover, the apparatus(es) 102 can, for example, be configured to include (e.g., by manner of adding) at least one time-based parameter such as a time indicator “t” to the feature of each point of the point cloud, in accordance with an embodiment of the disclosure. Additionally, the apparatus(es) 102 can, for example, be configured to encode the point cloud data to create / generate a pseudo-image feature map (i.e. , pseudo image), in accordance with an embodiment of the disclosure. For example, the apparatus(es) 102 can be configured to encode the point cloud data using / via Pointpillar network to create / generate at least one pseudo-image feature map (i.e., pseudo image). The apparatus(es) 102 can, for example, be further configured to facilitate generating at least one feature map based on the pseudo image, in accordance with an embodiment of the disclosure. For example, the apparatus(es) 102 can be configured to communicate the pseudo-image to at least one ResNet-like structure to produce the feature map(s) (e.g., referable to as “point cloud feature maps(s)”), in accordance with an embodiment of the disclosure. The feature map(s) (e.g., point cloud feature map(s)) can, for example, contain / include each pillar 3D coordinates.

[0053] In the above manner, it is appreciable that the apparatus(es) 102 can, for example, be configured to perform and / or facilitate one or more processing tasks in association with preliminary processing of the LiDAR video frame(s), in accordance with an embodiment of the disclosure. In accordance with an embodiment of the disclosure, it is contemplated that preliminary processing of the LiDAR video frame(s) can, for example, include any one of, or any combination of, the following:

[0054] • PDR based processing

[0055] • Including at least one time-based parameter (e.g., a time indicator “t”)

[0056] • Encoding to create / generate at least one pseudo image

[0057] • Generating at least one feature map (e.g., referable to as “point cloud feature map”)

[0058] In regard to the RGB video frame(s), in the context of the specific example scenario, the apparatus(es) 102 can, for example, be configured to process / facilitate processing of RGB data to generate / produce at least one feature map (e.g., referable to as “RGB feature map(s)”). For example, the apparatus(es) 102 can be configured to communicate the RGB data for processing by a convolutional neural network (CNN) such as a video-based CNN to generate / produce at least one RGB feature map, in accordance with an embodiment of the disclosure. It is contemplated that a RGB video frame can, for example, be associated with at least one still image. The apparatus(es) 102 can, for example, be configured to generate / create at least one reference source (e.g., at least one additional input source) based on / in association with each still image to facilitate recognition of one or more distant (e.g., faraway) objects, in accordance with an embodiment of the disclosure. For example, the at least one reference source may be super-resolution images which may increase the diversity of features and may assist in efficient recognition of distant objects. For example, the apparatus(es) 102 can be configured to generate / create the reference source(s) based on processing in association with a plurality of resolution diffusion models, in accordance with an embodiment of the disclosure. In a specific example, the plurality of resolution diffusion models can correspond to a plurality of super-resolution diffusion models (e.g., an example of network architecture can be based on section 2.5 of “Photorealistic Text-to-lmage Diffusion Models with Deep Language Understanding” by Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet and Mohammad Norouzi; Moreover, it is notable that in this reference, only 2 steps of resolution are performed whereas the present disclosure contemplates additional one(s) in association with, for example, 64x64 -> 256x256, 256x256->512x512, 512x512->1024x1024. Section F.2 and F.3 of the reference provide examples of architecture details.) which can, for example, include a small-to-medium (STM) model, a medium-to-large (MTL) model and / or a large-to-extra large (LXL) model.

[0059] In one embodiment, in regard to STM, a medium resolution image (i.e., larger resolution as compared to the still image) can, for example, be generated based on a still image.

[0060] In one embodiment, in regard to MTL, a large resolution image (i.e., larger resolution as compared to the medium resolution image) can, for example, be generated based on the medium resolution image.

[0061] In one embodiment, in regard to LXL, an extra-large resolution image (i.e., larger as compared to the large resolution image) can, for example, be generated based on the large resolution image. In this regard, it is appreciable that the reference source can, for example, correspond to the extra-large resolution image, in accordance with an embodiment of the disclosure. In one embodiment, it is contemplated that datasets can, for example, include high- resolution RGB images with a large diversity of driving scenarios. In training, each image can possibly be cropped into patches with a size of 256x256, 512x512, 1024x1024 as High Resolution (HR) ground truth. It is further contemplated that to obtain corresponding Low Resolution (LR) images, HR images can possible be down- sampled with bicubic kernel. For example, for evaluation, HR images can be down- sampled with bicubic kernel to obtain the LR images and SISR (Single Image SuperResolution) method(s) can be applied on the LR images to obtain the SR predictions without cropping. For example, resolution of HR and their corresponding LR can, for example, vary such as [64x64, 256x256], [256x256, 512x512], and [512x512, 1024x1024] according to STM, MTL, and LXL, in accordance with an embodiment of the disclosure.

[0062] It is yet further contemplated that in accordance with an embodiment of the disclosure, diffusion model(s) may possibly be capable of being replaced by a / another type of neural network, such as Generative Adversarial Network and / or Flow-based model(s).

[0063] Moreover, a separate processing tower / chain can, for example, be defined (e.g., by the apparatus(es) 102) in association with the STM, the MTL and / or the LXL, in accordance with an embodiment of the disclosure. It is contemplated that each processing tower / chain can, for example, be associated with / correspond to a ResNet (Residual Network)-like structure to extract the at least one feature map (e.g., at least one feature map can be extracted from each processing tower / chain), in accordance with an embodiment of the disclosure.

[0064] In the above manner, it is appreciable that the apparatus(es) 102 can, for example, be configured to perform and / or facilitate one or more processing tasks in association with preliminary processing of the RGB video frame(s), in accordance with an embodiment of the disclosure. In accordance with an embodiment of the disclosure, it is contemplated that preliminary processing of the RGB video frame(s) can, for example, include any one of, or any combination of, the following: Generating / producing at least one feature map (e.g., referable to as “RGB feature map”)

[0065] Generating / creating at least one reference source

[0066] Moreover, in the specific example scenario, the apparatus(es) 102 can, for example, be configured to perform (and / or facilitate performing of) one or more further processing tasks subsequent to preliminary processing of the LiDAR video frame(s) and / or preliminary processing of the RGB video frame(s), in accordance with an embodiment of the disclosure. The further processing task(s) can, for example, include / correspond to / be associated with fusion-based processing in 4D (NB: as mentioned earlier, the 4thdimension can, for example, be time-based), in accordance with an embodiment of the disclosure.

[0067] For example, the point cloud feature map(s) and the RGB feature map(s) can be fused / combined in time (e.g., time-based fusion / combination of the point cloud feature map(s) and the RGB feature map(s)).

[0068] In a more specific example, the RGB feature map(s) and the point cloud feature map(s) can be combined / fused by manner of using projections (e.g., camera intrinsic and extrinsic parameters) so as to enable geometric alignment, for each point pillar location, at least one 2D point can be obtained, and pillar feature can be enhanced by manner of learning fine-grained correlation among points (i.e., Point-wise attention), in accordance with an embodiment of the disclosure. Moreover, the enhanced pillar feature and the 2D point(s) can, for example, be concatenated, in accordance with an embodiment of the disclosure. Furthermore, processing task(s) in association with a connection architecture search in 4D can, for example, be performed (e.g., Dynamic connection by differentiable architecture search with self-attention mechanisms and for each pillar, the network can possibly dynamically select which RGB based data to fuse information from - in a more specific example, taking a higher resolution feature from an early layer or a low resolution feature from a later layer.).

[0069] In one embodiment, for example, size of anchor box(es) can be [4.7, 2.1 , 1 .7], and 2 rotations (0 and 45 degrees) can be used at each feature map location. For PointPillar pseudo-image creation, the following parameters can, for example, be suitable / possible:

[0070] • grid size (224, 224, 1)

[0071] • x-range and y-range of (-74.88, 74.88)

[0072] • z-range of (-5, 5)

[0073] • maximum number of points per cell, N = 128

[0074] It is contemplated that 10,000 pillars can, for example, be set. Moreover, data augmentation can, for example, be applied to point cloud(s) (e.g., random 3D rotations and flips). Furthermore, camera matrices can, for example, be updated based on augmentation(s) such that the projection(s) can possibly still apply. Additionally, it is contemplated that augmentation may, for example, not be necessary in association with RGB stream(s). A feature dim of 64 for the input point(s) can possible be used in association with the PointPillars network and the pseudo-image created can include / correspond to a 224 x 224 shape. This can, for example, be followed by 3 convolutional blocks with 4, 6, and 6 repeats. Each block can, for example, include a convolution, batch norm and ReLU activation. This can, for example, be followed by 3 deconvolutional layers which can generate the prediction(s). A RGB single frame network can, for example, be based on / correspond to a standard ResNet 18. The video network can, for example, be based on TinyVideoNetworks. Specifically, for example, a network which can include 6 Residual blocks can possibly be used (e.g., the first two blocks can apply both spatial and temporal convolutions (in that order) to data). Moreover, the Diving Dataset (which can include 1 950 segments that are each 20 seconds long which can be about 200 frames with a total of 390 000 frames) can be trained / usable as basis for training. The LiDAR data can, for example, be processed into point clouds which can, for example, be grouped by timestamp and aligned with the RGB frames.

[0075] Subsequently, as a possible option, the apparatus(es) 102 can, for example, be configured to generate one of more 3D indications (e.g., 3D box(es)) to indicate one or more detected objects, in accordance with an embodiment of the disclosure. For example sequence(s) of 16 frames can be taken as input and 3D boxes for the last frame can possibly be predicted, and filtering (e.g., filtering out) can be based on box(es) associated with probability of less than 0.4, box(es) larger than 30m in length and 5m in width and boxes smaller than 0.5m in length and width, in accordance with an embodiment of the disclosure.

[0076] The present disclosure contemplates that, in one example, Point Diffusion-Refinement (PDR) can possibly effectively and / or efficiently extract multi-level features from partially observed point clouds to guide completion. Moreover, spatial locations of 3D points can, for example, possibly be capable of being manipulated to obtain smooth surfaces and sharp details. The training dataset may include a plurality of pairs of 3D colored point clouds. Each pair of 3D point clouds may include a low-resolution point cloud and a high-resolution point cloud, where both point clouds capture a common environment. For example, the low-resolution point clouds may be obtained using 32- or 64-channel resolution lidar sensors, while the high-resolution point clouds may be obtained using 128-channel high resolution lidar sensors. The low-resolution point clouds are used as input to the diffusion network, while the high-resolution point clouds are used as ground truth to the diffusion network. This trained point diffusion refinement model is used to generate complete point cloud for the given point cloud sequence from LIDAR data. For example, the KITTI dataset, as contemplated per “Vision meets Robotics: The KITTI Dataset” by Andreas Geiger, Philip Lenz, Christoph Stiller and Raquel Urtasun can be processed to generate the training dataset. The KITTI dataset can, for example, include 64-channel high resolution point clouds. The 64-channel point clouds can, for example, be duplicated and can, for example, have several rows removed from them, to result in corresponding low-resolution point clouds, in accordance with an embodiment of the disclosure.

[0077] The present disclosure further contemplates that, in one example, an additional input source (e.g., additional RGB stream) can possibly be beneficial for recognition of faraway object(s). For example, two or more different resolutions can, for example, possibly increase the diversity of features available for connection learning.

[0078] The present disclosure yet further contemplates that, in one example, Point-wise attention can possibly facilitate dynamic information aggregation across two modalities at fine-grained level with negligible overhead(s) (e.g., negligible extra cost) in computation. For example, dynamic connection learning between features of different modalities can possibly be applicable to many input feature sources and can possibly be agnostic to aligning different sensors for autonomous driving, in accordance with an embodiment of the disclosure. Dynamic connection learning may also leverage motion cues and dense image information to detect distant objects while ensuring computational efficiency.

[0079] The present disclosure yet additionally further contemplates that, in one example, each modality can possibly be processed with a suitable architecture producing one or more rich features, which can be aligned and / or fused at one or more different levels by dynamic connection learning so as to possibly facilitate improved detection rate in 3D objects and / or facilitate competitive runtime, in accordance with an embodiment of the disclosure.

[0080] It is contemplated that the system 100 can, for example, possibly be useful for facilitating object detection (e.g., detection of one of more 3D objects) which can serve as basis for perception stack for the purpose of path planning, motion prediction and / or collision avoidance etc., in accordance with one or more embodiments of the disclosure.

[0081] As mentioned earlier, the apparatus(es) 102 can be configured to process (and / or facilitate processing of) the input signal(s) by manner of any one of initial processing, preliminary processing and fusion-based processing, or any combination thereof (i.e., initial processing, preliminary processing and / or fusion-based processing) to generate / derive the output signal(s). Moreover, fusion-based processing can, for example, be in 4D (e.g., fusion-based processing in 4D) so as to possibly facilitate object detection.

[0082] In the context of the above discussed example scenario (in which a specific example scenario was further discussed), the input signal(s) can, for example, be associated with / correspond to / include one or both of data (e.g., point cloud data) communicated from / captured by / generated by a first sensor type (e.g., LiDAR sensor) and data (e.g., RGB data) communicated from / captured by / generated by a second sensor type (e.g., RGB camera). In one embodiment, the input signal(s) can, for example, be associated with / correspond to / include data (e.g., point cloud data and RGB data) communicated from / captured by / generated by both a first sensor type and a second sensor type. Appreciably, the input signal(s) can, for example, be associated with / correspond to / include data from multi-modal sensors (e.g., a first sensor type and a second sensor type). The output signal(s) can, for example, be associated with / correspond to / include one of more 3D indications (e.g., 3D box(es) which can, for example, be visually perceivable by manner of, for example, being displayed via a display screen) to indicate one or more detected objects.

[0083] Moreover, in the context of the above discussed example scenario (in which a specific example scenario was further discussed), initial processing can, for example, be associated with / correspond to / include initial processing tasks associated with one or both of calibrating and synchronizing (i.e., calibrating and / or synchronizing). Furthermore, preliminary processing can, for example, be associated with / correspond to / include preliminary processing of one or both of data associated with a first sensor type (preliminary processing of the LiDAR video frame(s)) and data associated with a second sensor type (e.g., preliminary processing of the RGB video frame(s)) to produce / generate at least one set of preliminary signals (e.g., a first set of preliminary signals and / or a second set of preliminary signals), in accordance with an embodiment of the disclosure. In one embodiment, preliminary processing can, for example, be associated with / correspond to / include preliminary processing of both data associated with a first sensor type and data associated with a second sensor type (e.g., preliminary processing of the LiDAR video frame(s) and preliminary processing of the RGB video frame(s)) to produce / generate a first set of preliminary signals (e.g., based on data associated with a first sensor type) and a second set of preliminary signals (e.g., based on data associated with a second sensor type) respectively. For example, data associated with the first sensor type can be preliminarily processed independently of / separately from data associated with the second sensor type. Additionally, fusionbased processing can, for example, be associated with / correspond to / include processing of the preliminary processed data associated with the first sensor type and the preliminarily processed data associated with the second sensor type by manner of time-based (e.g., the 4thdimension) fusing / combining (e.g., the point cloud feature map(s) and the RGB feature map(s) can be fused / combined in time). In the above manner, it is contemplated that data collection can be provided / facilitated in a comparatively more optimal, comparatively more sufficient manner and / or comparatively more efficient manner for, for example, facilitation of object detection, in accordance with an embodiment of the disclosure. Specifically, it is contemplated that fusing / combining data from multi-modal sensors in 4D can possibly facilitate object detection (e.g., detection of one or more objects in an environment which can, for example, be external to a vehicle) in a (comparatively more) sufficient, optimal and / or efficient manner, in accordance with an embodiment of the disclosure.

[0084] The above-described advantageous aspect(s) of the system 100 of the present disclosure can also apply analogously (all) the aspect(s) of a below described apparatus(es) 102 and / or processing method of the present disclosure. Likewise, all below described advantageous aspect(s) of the apparatus(es) 102 and / or the processing method of the disclosure can also apply analogously (all) the aspect(s) of above described system 100 of the disclosure.

[0085] The aforementioned apparatus(es) 102 will be discussed in further detail with reference to Fig. 2 hereinafter.

[0086] Referring to Fig. 2, an apparatus 102 is shown in further detail in the context of an example implementation 200, according to an embodiment of the disclosure.

[0087] In the example implementation 200, the apparatus 102 can correspond to an electronic module 200a which can, for example, be capable of performing one or more processing tasks, in accordance with an embodiment of the disclosure.

[0088] The electronic module 200a can, for example, include a casing 200b. Moreover, the electronic module 200a can, for example, carry any one of a first module 202, a second module 204, a third module 206, or any combination thereof.

[0089] In one embodiment, the electronic module 200a can carry a first module 202, a second module 204 and / or a third module 206. In a specific example, the electronic module 200a can carry a first module 202, a second module 204 and a third module 206, in accordance with an embodiment of the disclosure.

[0090] In this regard, it is appreciable that, in one embodiment, the casing 200b can be shaped and dimensioned to carry any one of the first module 202, the second module 204 and the third module 206, or any combination thereof.

[0091] The first module 202 can be coupled to one or both of the second module 204 and the third module 206. The second module 204 can be coupled to one or both of the first module 202 and the third module 206. The third module 206 can be coupled to one or both of the first module 202 and the second module 204. In one example, the first module 202 can be coupled to the second module 204 and the second module 204 can be coupled to the third module 206, in accordance with an embodiment of the disclosure. Coupling between the first module 202, the second module 204 and / or the third module 206 can, for example, be by manner of one or both of wired coupling and wireless coupling. Each of the first module 202, the second module 204 and the third module 206 can correspond to one or both of a hardware-based module and a software-based module, according to an embodiment of the disclosure.

[0092] In one example, the first module 202 can correspond to a hardware-based receiver which can be configured to receive one or more input signals.

[0093] The second module 204 can, for example, correspond to a network-based / software- based and / or hardware-based (e.g., a microprocessor) processing module which can be configured to perform (and / or facilitate performance of) one or more processing tasks in association with any one of, or any combination of, the following:

[0094] • initial processing,

[0095] • preliminary processing

[0096] • fusion-based processing

[0097] Specifically, the second module 204 can, for example, be configured to process and / or facilitate processing of the received input signal(s) by manner of initial processing, preliminary processing and / or fusion-based processing in a manner so as to generate / derive one or more output signal(s), in accordance with an embodiment of the disclosure.

[0098] The third module 206 can, in one example, correspond to a hardware-based transmitter which can be configured to communicate the output signal(s) from the electronic module 200a, in accordance with an embodiment of the disclosure. The output signal(s) can, for example, be communicated from the electronic module 200a to one or more devices 104 and / or one or more other apparatuses 102, in accordance with an embodiment of the disclosure.

[0099] In another example, the third module 206 can correspond to a display portion (e.g., display screen) capable of facilitating visual perception of the output signal(s) by at least one user, in accordance with an embodiment of the disclosure.

[0100] The present disclosure contemplates the possibility that the first and second modules 202 / 204 can be an integrated software- ha rd ware based module (e.g., an electronic part which can carry a software program / algorithm in association with receiving and processing functions / an electronic module programmed to perform the functions of receiving and processing). The present disclosure further contemplates the possibility that the first and third modules 202 / 206 can be an integrated software- ha rd ware based module (e.g., an electronic part which can carry a software program / algorithm in association with receiving and transmitting functions / an electronic module programmed to perform the functions of receiving and transmitting). The present disclosure yet further contemplates the possibility that the first and third modules 202 / 206 can be an integrated hardware module (e.g., a hardware-based transceiver) capable of performing the functions of receiving and transmitting.

[0101] In the above manner, it is contemplated that data collection can be provided / facilitated in a comparatively more optimal, comparatively more sufficient manner and / or comparatively more efficient manner for, for example, facilitation of object detection, in accordance with an embodiment of the disclosure. Specifically, it is contemplated that fusing / combining data from multi-modal sensors in 4D can possibly facilitate object detection (e.g., detection of one or more objects in an environment which can, for example, be external to a vehicle) in a (comparatively more) sufficient, optimal and / or efficient manner, in accordance with an embodiment of the disclosure.

[0102] The above-described advantageous aspect(s) of the apparatus 102 of the present disclosure can also apply analogously (all) the aspect(s) of a below described processing method of the present disclosure. Likewise, all below described advantageous aspect(s) of the processing method of the disclosure can also apply analogously (all) the aspect(s) of above described apparatus 102 of the disclosure. It is to be appreciated that these remarks apply analogously to the earlier discussed system 100 of the present disclosure.

[0103] Referring to Fig. 3, a processing method 300 in association with the system 100 is shown, according to an embodiment of the disclosure. The processing method 300 can, for example, be suitable for facilitating object detection (e.g., detection of one of more objects), in accordance with an embodiment of the disclosure.

[0104] For example, the processing method 300 can, for example, be associated with data collection and / or processing collected data in a manner so as to facilitate object detection, in accordance with an embodiment of the disclosure. The object(s) can, for example, correspond to / include / be associated with 3D object(s), in accordance with an embodiment of the disclosure. Moreover, the object(s) can, for example, be associated with / be located in an environment. The environment can, for example, be external to a vehicle such as an autonomous vehicle, in accordance with an embodiment of the disclosure.

[0105] The processing method 300 can, for example, include any one of an input step 302, a processing step 304 and an output step 306, or any combination thereof, in accordance with an embodiment of the disclosure. For example, the processing method 300 can include an input step 302, a processing step 304 and / or an output step 306, in accordance with an embodiment of the disclosure.

[0106] In one embodiment, the processing method 300 can include an input step 302, a processing step 304 and an output step 306. In another embodiment, the processing method 300 can include an input step 302 and a processing step 304. In yet another embodiment, the processing method 300 can include a processing step 304 and an output step 306. In yet a further embodiment, the processing method 300 can include a processing step 304. In yet a further additional embodiment, the processing method 300 can include one of an input step 302, a processing step 304 and an output step 306 (i.e., an input step 302, a processing step 304 or an output step 306). In yet another further additional embodiment, the processing method 300 can, for example, include an input step 302 and an output step 306.

[0107] With regard to the input step 302, one or more processing tasks in association with receiving one or input signals can be performed, in accordance with an embodiment of the disclosure. For example, the apparatus(es) 102 can be configured to perform the processing task(s) in association with receiving at least one input signal generated by and / or communicated from the device(s) 104 (e.g., a first sensor type and / or a second sensor type), as discussed earlier with reference to Fig. 1 , in accordance with an embodiment of the disclosure.

[0108] With regard to the processing step 304, one or more processing tasks can be performed, in accordance with an embodiment of the disclosure, in association with any one of, or any combination of, the following:

[0109] • initial processing,

[0110] • preliminary processing

[0111] • fusion-based processing

[0112] Specifically, the received input signal(s) can, for example, be processed by manner of initial processing, preliminary processing and / or fusion-based processing so as to generate one or more output signals, in accordance with an embodiment of the disclosure. For example, the apparatus(es) 102, as discussed earlier with reference to Fig. 1 , can be configured to perform and / or facilitate the processing task(s) in association with initial processing, preliminary processing and / or fusion-based processing in association / in connection with the input signal(s) in a manner so as to generate one or more output signals, as discussed earlier with reference to Fig. 1 , in accordance with an embodiment of the disclosure. With regard to the output step 306, one or more processing tasks in association with communicating one or more output signals can be performed in accordance with an embodiment of the disclosure. The output signal(s) can, for example, be communicated (e.g., transmitted) from the apparatus(es) 102 and / or communicated (e.g., displayed) by the apparatus(es) 102 in a manner so as to be visually perceivable. For example, the apparatus(es) 102 can be configured to perform the processing task(s) in association with communicating the output signal(s), as discussed earlier with reference to Fig. 1 , in accordance with an embodiment of the disclosure.

[0113] The present disclosure further contemplates a computer program (not shown) which can include instructions which, when the program is executed by a computer (not shown), cause the computer to carry out the input step 302, the processing step 304 and / or the output step 306, as discussed with reference to the processing method 300.

[0114] The present disclosure yet further contemplates a computer readable storage medium (not shown) having data stored therein representing software executable by a computer (not shown), the software including instructions, when executed by the computer, to carry out the input step 302, the processing step 304 and / or the output step 306, as discussed with reference to the processing method 300.

[0115] In view of the foregoing, it is appreciable that the present disclosure generally contemplates a processing method 300 which can, for example, be suitable for facilitating object detection (e.g., detection of one of more objects), in accordance with an embodiment of the disclosure.

[0116] The processing method 300 can include a processing step 304. In regard to the processing step, one or more processing tasks can, for example, be performed in association with any one of, or any combination of, initial processing, preliminary processing and fusion-based processing (i.e., at least one of initial processing, preliminary processing and fusion-based processing).

[0117] The initial processing can, for example, include one or both of calibrating and synchronizing (i.e., at least one of calibrating and synchronizing) one or more input signals which can be associated with data associated with a first sensor type and data associated with a second sensor type, in accordance with an embodiment of the disclosure. In one example, one or both of the first sensor type and the second sensor type (i.e., at least one of the first sensor type and the second sensor type) can be carried by a vehicle, and object detection can, for example, correspond to detection of at least one object associated with an environment external to the vehicle.

[0118] The preliminary processing can, for example, include preliminary processing of data associated with a first sensor type (e.g., at least one LiDAR sensor) to generate a first set of preliminary signals and preliminary processing of data associated with a second sensor type (e.g., at least one camera sensor) to generate a second set of preliminary signals, in accordance with an embodiment of the disclosure. In one example, data associated with the first sensor type can correspond to point cloud data and data associated with the second sensor type can correspond to RGB (Red Green Blue) data. In one specific example, point cloud data can be preliminarily processed to generate one or more point cloud feature maps (e.g., which can correspond to / be associated with a first set of preliminary signals, in accordance with an embodiment of the disclosure) and RGB data can one preliminarily processed to generate one or more RGB feature maps (e.g., which can correspond to / be associated with a second set of preliminary signals, in accordance with an embodiment of the disclosure). In one embodiment, data associated with the first sensor type can, for example, be independently preliminarily processed from data associated with the second sensor type.

[0119] The fusion-based processing can, for example, include fusing the first set of preliminary signals (e.g., which can include / be associated with / correspond to the point cloud feature map(s)) and the second set of preliminary signals (e.g., which can include / be associated with / correspond to the RGB feature map(s)) in time, in accordance with an embodiment of the disclosure. In one example, the point cloud feature map(s) (e.g., which can correspond to / be associated with the first set of preliminary signals, in accordance with an embodiment of the disclosure) and the RGB feature map(s) (e.g., which can correspond to / be associated with the second set of preliminary signals, in accordance with an embodiment of the disclosure) can be capable of being fused in time (e.g., time-based fusing of the point cloud feature map(s) and the RGB feature map(s)).

[0120] In one embodiment, the processing method 300 can, for example, further include an input step 302 wherein one or more one input signals can be received.

[0121] In one embodiment, the processing method 300 can, for example, further include an output step 306 wherein one or more output signals (e.g., which can be generated based on fusion-based processing) can be communicated. For example, the output signal(s) can be associated with at least one 3D (three-dimension) indication to indicate at least one detected object.

[0122] Further, in view of the foregoing, it is appreciable that the present disclosure further generally contemplates an apparatus 102 in association with the processing method 300, in accordance with an embodiment of the disclosure. The apparatus 102 can, for example, be suitable for facilitating object detection (e.g., detection of one of more objects), in accordance with an embodiment of the disclosure. Moreover, the apparatus 102 can, for example, correspond to an electronic module 200a as discussed earlier with reference to Fig. 2, in accordance with an embodiment of the disclosure.

[0123] The apparatus 102 can, for example, include a first module 202, a second module 204 and a third module 206, in accordance with an embodiment of the disclosure.

[0124] In one embodiment, the first module 202 (e.g., corresponding to a receiver) can, for example, be configured to receive at least one input signal (e.g., communicable from at least one device 104 such as at least one LiDAR sensor and / or at least one camera sensor). The second module 204 (e.g., corresponding to a processor) can, for example, be configured to one or both of process and facilitate processing of (i.e. , at least one of process and facilitate processing of) the input signal(s) according to the processing method 300 as discussed above to generate at least one output signal. The third module 206 (e.g., corresponding to a transmitter) can, for example, be configured to communicate the output signal(s). In one embodiment, the input signal(s) can be associated with data associated with a first sensor type (e.g., at least one LiDAR sensor) and data associated with a second sensor type (e.g., at least one camera sensor). Moreover, the first sensor type and the second sensor type can, for example, be carried by a vehicle, and object detection can, for example, correspond to detection of at least one object associated with an environment external to the vehicle.

[0125] In the above manner, it is contemplated that data collection can be provided / facilitated in a comparatively more optimal, comparatively more sufficient manner and / or comparatively more efficient manner for, for example, facilitation of object detection, in accordance with an embodiment of the disclosure. Specifically, it is contemplated that fusing / combining data from multi-modal sensors in 4D can possibly facilitate object detection (e.g., detection of one or more objects in an environment which can, for example, be external to a vehicle) in a (comparatively more) sufficient, optimal and / or efficient manner, in accordance with an embodiment of the disclosure. It is also contemplated that temporal integration of RGB and point cloud features through a 4D- Net for object detection may integrate both projection fusion mechanisms and connectivity search, dynamically learning optimal methods and locations for feature fusion automatically, such that the approach is better suited to leveraging motion cues and dense image information for detecting distant objects while maintaining computational efficiency.

[0126] It should be appreciated that the embodiments described above can be combined in any manner as appropriate (e.g., one or more embodiments as discussed in the “Detailed Description” section can be combined with one or more embodiments as described in the “Summary of the Invention” section).

[0127] It should be further appreciated by the person skilled in the art that variations and combinations of embodiments described above, not being alternatives or substitutes, may be combined to form yet further embodiments. In one example, the communication network 106 can be omitted. Communication (i.e., between the apparatus(es) 102 and the device(s) 104) can be by manner of direct coupling. Such direct coupling can be by manner of one or both of wired coupling and wireless coupling.

[0128] In another example, the system 100 can be associated with a vehicle and an environment external to the vehicle. Moreover, the system 100 can, for example, be a standalone-based system which may possibly be independent of the vehicle (e.g., the system 100 may, for example, possibly not be carried by the vehicle), in accordance with an embodiment of the disclosure.

[0129] In yet another example, the apparatus(es) 102 can be coupled to one or more other networks which can be configured for processing (e.g., processing via neural network(s) such as ResNet and / or CNN etc., as discussed earlier and / or processing networks such as PointPillar network etc., as discussed earlier). By manner of communicating one or more signals (e.g., input signal(s), input signal(s) processed based on initial processing and / or preliminarily processed input signal(s)) to such other network(s) for processing, the apparatus(es) 102 can, for example, be considered to be facilitating processing, in accordance with an embodiment of the disclosure. Moreover, signal(s) processed by such other network(s) can, for example, be communicated to the apparatus(es) 102 and / or the device(s) 104 for, for example, further processing, in accordance with an embodiment of the disclosure.

[0130] In yet a further example, one of more device(s) 104 can, for example, include / be associated with / correspond to / host / carry / be a part of one or more other networks which can be configured for processing (e.g., processing via neural network(s) such as ResNet and / or CNN etc., as discussed earlier and / or processing networks such as PointPillar network etc., as discussed earlier), in accordance with an embodiment of the disclosure. Moreover, signal(s) processed by such other network(s) can, for example, be communicated to the apparatus(es) 102 for, for example, further processing, in accordance with an embodiment of the disclosure. In the foregoing manner, various embodiments of the disclosure are described for addressing at least one of the foregoing disadvantages. Such embodiments are intended to be encompassed by the following claims, and are not to be limited to specific forms or arrangements of parts so described and it will be apparent to one skilled in the art in view of this disclosure that numerous changes and / or modification can be made, which are also intended to be encompassed by the following claims.

Claims

Claim(s)1. A processing method (300) suitable for facilitating object detection, the processing method (300) comprising: a processing step (304) wherein at least one processing task is capable of being performed in association with at least one of: initial processing, preliminary processing fusion-based processing wherein initial processing comprises at least one of calibrating and synchronizing at least one input signal associable with: data associated with a first sensor type; and data associated with a second sensor type, wherein preliminary processing comprises preliminary processing of data associated with a first sensor type to generate a first set of preliminary signals and preliminary processing of data associated with a second sensor type to generate a second set of preliminary signals, and wherein fusion-based processing comprises fusing the first set of preliminary signals and the second set of preliminary signals in time.

2. The processing method (300) of claim 1 , wherein data associated with the first sensor type is independently preliminarily processed from data associated with the second sensor type.

3. The processing method (300) of any of the preceding claims, wherein the first sensor type corresponds to a LiDAR (light detection and ranging) sensor, and wherein the second sensor type corresponds to a camera sensor.

4. The processing method (300) of any of the preceding claims, wherein data associated with the first sensor type corresponds to point cloud data, and wherein data associated with the second sensor type corresponds to RGB (Red Green Blue) data.

5. The processing method (300) of any of the preceding claims, at least one of the first sensor type and the second sensor type being carried by a vehicle and object detection being detection of at least one object associated with an environment external to the vehicle.

6. The processing method (300) of any of the preceding claims, point cloud data being preliminarily processed to generate at least one point cloud feature map.

7. The processing method (300) of claim 6, wherein preliminarily processing point cloud data to generate at least one point cloud feature map comprises at least one of: processing point cloud data, preferably by manner of Point Diffusion- Refinement (PDR) paradigm; including at least one time-based parameter; encoding point cloud data to create at least one pseudo image; generating at least one point cloud feature map.

8. The processing method (300) of any of the preceding claims, RGB data being preliminarily processed to generate at least one RGB feature map.

9. The processing method (300) of claim 8, wherein preliminarily processing the RGB data to generate at least one RGB feature map comprises at least one of: generating at least one reference source; extracting at least one RGB feature map.

10. The processing method (300) of any of the preceding claims, the point cloud feature map and the RGB feature map capable of being fused in time.11 . The processing method (300) of claim 10, wherein the point cloud feature map and the RGB feature map are fused in time by at least one of: using projections to enable geometric alignment; performing connection architecture search in 4D, preferably by differentiable architecture search with self-attention mechanisms.

12. The processing method (300) of claim 11 , wherein using projections to enable geometric alignment comprises at least one of: for each point pillar location, obtaining at least one 2D point; enhancing pillar features by manner of learning fine-grained correlation among points; concatenating the enhanced pillar features and the obtained at least one 2D point.

13. The processing method (300) of any of the preceding claims, further comprising an input step (302) wherein at least one input signal is received.

14. The processing method (300) of any of the preceding claims, further comprising an output step (306) wherein at least one output signal generated based on fusionbased processing is communicable.

15. The processing method (300) of any of the preceding claims, the output signal being associated with at least one 3D (three-dimension) indication to indicate at least one detected object.

16. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out at least one of the input step (302), the processing step (304) and the output step (306) according to the processing method (300) of any of the preceding claims.

17. A computer readable storage medium having data stored therein representing software executable by a computer, the software including instructions, when executed by the computer, to carry out at least one of the input step (302), the processing step (304) and the output step (306) according to the processing method (300) of any of claims 1 to 15.

18. An apparatus (102) comprising: a first module (202) configurable to receive at least one input signal; a second module (204) configurable to at least one of process and facilitate processing of the input signal according to the processing method (300) of any of claims 1 to 15 to generate at least one output signal; and a third module (206) configurable to communicate at least one output signal.

19. The apparatus (102) of claim 18, the input signal being associated with data associated with a first sensor type and data associated with a second sensor type, wherein the first sensor type and the second sensor type are carried by a vehicle, and wherein object detection corresponds to detection of at least one object associated with an environment external to the vehicle.

Citation Information

Patent Citations

  • Multi-Task Multi-Sensor Fusion for Three-Dimensional Object Detection

    US20230043931A1

  • Detection method, system, and device based on fusion of image and point cloud information, and storage medium

    WO2022156175A1