Object detection circuit and object detection method
The object detection circuit and method address high processing complexity and quality loss in sensor fusion by using unique feature models for each sensor, enhancing detection speed and accuracy through AI-assisted data fusion.
Patent Information
- Application Number
- JP2022517741
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2019-09-25
- Filing Date
- 2020-09-10
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2040-09-10
AI Technical Summary
Existing object detection methods using multiple sensors require high processing complexity, leading to increased power consumption and cost, and often result in slow or degraded final detection due to the need for manual processing and quality loss during sensor data fusion.
An object detection circuit and method that utilizes unique feature models for each sensor to generate and combine object probability data, incorporating artificial intelligence for sensor fusion, thereby reducing complexity and improving detection speed and accuracy.
The solution provides a reliable and fast method for object detection by minimizing quality loss and system complexity, while ensuring accurate and efficient fusion of sensor data, enabling applications in autonomous driving and surveillance systems.
Smart Images

Figure 0007586173000003 
Figure 0007586173000004 
Figure 0007586173000005
Abstract
Description
[Technical field]
[0001] The present disclosure generally relates to object detection circuits and methods. [Background technology]
[0002] Generally, methods and devices for detecting objects are known. For example, in the automotive field (e.g., automotive security, autonomous driving, etc.), it is desirable to have a fast and reliable method for detecting objects on a roadway.
[0003] Known methods for detecting objects (eg, vehicles, pedestrians, etc.) rely on the output of one or more sensors.
[0004] For each sensor, object detection may be performed and then the sensor data or data indicative of objects detected for each sensor may be fused to receive a final object detection.
[0005] Thus, such known methods for object detection with multiple sensors may require object detection for each of the sensors used, which may come with high processing complexity and / or may result in slow or degraded final detection of the object after the sensor data is fused.
[0006] While techniques exist for detecting objects, it is generally desirable to provide object detection circuits and methods. Summary of the Invention
[0007] According to a first aspect of the present invention, the present disclosure provides a method for producing a semiconductor device comprising: acquiring first feature data based on first sensing data of a first sensor; comparing the first feature data to a first predetermined feature model representative of a predetermined object, the first predetermined feature model being specific to the first sensor, thereby generating first object probability data; acquiring second feature data based on second sensing data of a second sensor; comparing the second feature data to a second predefined feature model representative of the predefined object, the second predefined feature model being specific to the second sensor, thereby generating second object probability data; Combining the first and second object probability data, thereby generating combined probability data for detecting the predetermined object. The present invention provides an object detection circuit configured to:
[0008] According to a second aspect of the present invention, the present disclosure provides a method for producing a semiconductor device comprising: acquiring first feature data based on first sensing data of a first sensor; comparing the first feature data to a first predefined feature model representative of a predefined object, the first predefined feature model being specific to the first sensor, thereby generating first object probability data; acquiring second feature data based on second sensing data of a second sensor; comparing the second feature data to a second predefined feature model representative of the predefined object, the second predefined feature model being specific to the second sensor, thereby generating second object probability data; combining the first object probability data and the second object probability data, thereby generating combined probability data for detecting the predetermined object; The present invention provides an object detection method, including:
[0009] Further aspects are set out in the dependent claims, the following description and the drawings. [Brief description of the drawings]
[0010] Embodiments of the present disclosure will now be described, by way of example only, with reference to the accompanying drawings, in which: [Figure 1] FIG. 2 illustrates a coordinate system including the signal distribution acquired by the stereo camera. [Diagram 2]FIG. 2 shows a coordinate system including the signal distribution acquired by the radar sensor. [Diagram 3] FIG. 1 is a diagram showing a template (feature model) of a stereo camera. [Figure 4] FIG. 1 is a diagram showing a template (feature model) for a radar sensor. [Diagram 5] FIG. 13 illustrates a fused object pose probability distribution. [Figure 6] FIG. 2 illustrates a coordinate system including objects detected based on feature maps of a stereo camera and a radar sensor. [Figure 7] FIG. 1 shows a block diagram of a method according to the present disclosure. [Figure 8] FIG. 2 is a schematic diagram illustrating a method for generating a feature map according to the present disclosure. [Figure 9] FIG. 2 illustrates a schematic diagram of one method for determining object probability data according to the present disclosure. [Figure 10] FIG. 1 illustrates a block diagram of a vehicle including an object detection circuit according to the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0011] Before describing the embodiment in detail with reference to FIG. 1, a general description will be given.
[0012] As mentioned initially, known methods of detecting objects can result in high processing complexity, which in turn can result in high power consumption and increased costs.
[0013] It is generally recognized that reducing power consumption and cost is desirable.
[0014] Furthermore, it is recognized that in order to provide (semi-)automated driving, a complete and reliable understanding of the environment is required, e.g. for route planning, hazard estimation, etc.
[0015] It is therefore desirable to provide a method that uses the output of several sensors and the fusion of their measurements, while at the same time providing a reliable and fast method of detecting objects from a predefined list of objects.
[0016] It is further recognized that several demands may be placed on a system or circuit to provide such a reliable and fast method of detecting objects.
[0017] One requirement is abstraction: for example, low-level sensor measurements (e.g. distance) need to be filtered for information that does not belong to object detection and may not be suitable for planning purposes (e.g. path) such that relevant information can be bundled into a representation of the environment, which may meet requirements of compactness, completeness and accuracy, e.g. it could be a list of objects with parameters indicating their location. Additionally, the sensor-specific appearance of an object must be taken into account (e.g., an object may appear differently via radar than via a camera).
[0018] A further requirement is sensor fusion: different sensors may rely on different measurement principles and the output data of each sensor may also be different, which may increase the effort to operate the sensors without increasing the complexity of the system as well.
[0019] A further requirement is the fusion of all available measurements. It is recognized that known devices may convert low-level measurements into high-level abstract representations for each sensor and then fuse the resulting information into a combined representation. However, information discarded in such abstraction may no longer be available during or after the fusion, such that its quality may be degraded. It is therefore desirable to at least reduce the amount of quality loss.
[0020] A further requirement is to overcome view-dependent object appearance. It is recognized that a predefined object may vary in appearance to the sensor, such that the orientation of the object may depend on the viewing direction (i.e., viewing angle) of the sensor. For example, there may be self-occlusion effects of the object, such that the sensor can only observe a portion of the object. The observed portion may appear different depending on the viewing angle of the sensor, and therefore it is generally desirable to overcome such effects in order to reliably detect objects.
[0021] A further requirement is intra-class object invariance. Different parts of the same object may be considered as belonging to different objects. For example, an object may have different shapes and / or colors in different spots, so that different shapes may be recognized as different objects. However, in general, it is desirable to provide a method for detecting objects as one object, even though they have different shapes and / or colors.
[0022] A further requirement is multi-object detection: it is necessary to detect multiple objects (e.g. the nearest object), e.g. to detect all objects within the detection range.
[0023] A further requirement is sensor noise: in order to avoid degradation of the measurement quality, it is desirable to provide a way to compensate (e.g., filter) the sensor noise and / or measurement imperfections without degradation of the quality.
[0024] Accordingly, some embodiments relate to an object detection circuit configured to obtain first feature data based on first sensing data of a first sensor, compare the first feature data to a first predetermined feature model representative of a predetermined object, the first predetermined feature model being specific to the first sensor, thereby generating first object probability data, obtain second feature data based on second sensing data of a second sensor, compare the second feature data to a second predetermined feature model representative of the predetermined object, the second predetermined feature model being specific to the second sensor, thereby generating second object probability data, and combine the first and second object probability data, thereby generating combined probability data for detecting the predetermined object.
[0025] The object detection circuitry may include one or more processors, such as a CPU, a GPU, one or more FPGAs (field programmable gate arrays), a computer, a server, and the like, as well as multiple combinations of the specified elements.
[0026] Since the object detection circuit may be applied in any field where the object detection circuit is utilized for object detection, such as (surveillance) camera systems, the object detection circuit may further be included in or associated with an automotive environment, for example such that it may be executed according to the operating state of the automobile, without limiting the present disclosure.
[0027] The object detection circuitry can be configured to obtain feature data, i.e., feature data can be provided to the object detection circuitry by at least one sensor that can be coupled, connected, associated, etc. to the object detection circuitry. In some embodiments, the object detection circuitry can include at least one sensor, such that the feature data can be obtained within the object detection circuitry.
[0028] In some embodiments, feature data acquired by multiple sensors may be distinguished (structurally or logically), i.e., feature data from a first sensor may be referred to as first feature data, feature data from a second sensor may be referred to as second feature data, and so on.
[0029] In some embodiments, the first feature data may be based on the first sensing data, for example, the first sensor may perform measurements according to an internal configuration, for example, may sense reflections of radio waves (e.g., in the case of a radar sensor), such that an analog or digital sensor output, referred to as the first sensing data, is generated.
[0030] The first feature data may comprise an evaluation result of the first sensing data, e.g. signal intensity according to distance at a predefined spatial resolution, which may generate a feature map as it is commonly known.
[0031] In some embodiments, the first sensor may be configured to generate the first feature data, while in some embodiments the first feature data is generated within the object detection circuitry.
[0032] Obtaining feature data may satisfy the abstraction requirements as described above, thereby providing a detection list of objects including a given object, including position, orientation, and correction estimates, from which further pose refinement and / or tracking of the objects may be performed.
[0033] Furthermore, by acquiring feature data multiple times, the requirement of fusion of all available measurements can be met, as described above.
[0034] The object detection circuitry may be further configured to compare the first feature data to a first predefined (first predetermined) model.
[0035] This comparison may be performed by performing a mapping between the first feature data and the model feature data (e.g., comparing the generated feature map with a predefined feature map) and detecting common and / or comparable signals.
[0036] Thus, this feature model may include data indicative of a predefined feature map that may be specific to the first sensor in the sense that the first feature data may be equivalent to this feature model, i.e., the radar feature model may be compared to feature data obtained based on the first sensing data of the radar sensor, etc., without limiting the present disclosure to the radar sensor. For example, a camera (RGB, infrared, time-of-flight), a lidar sensor, a time-of-flight sensor may constitute the first sensor, or multiple sensing data, such as a fusion signal of an RGB and time-of-flight camera, may be fused into the first sensing data.
[0037] The feature models may be representative of predefined objects, for example, in an automotive environment, the predefined objects may be automobiles, pedestrians, roadway markers, etc., which may be indicated by specific signal signatures, i.e., object-specific sensing data may be generated in response to detection of the predefined objects.
[0038] The object-specific sensing data may be predefined and acquired through a calibration process, a training process, etc., and an assignment between the predefined object and the first feature data may be performed based on a comparison of the first feature data with a first predetermined feature model.
[0039] A comparison between the first feature data and the first predetermined feature model may be performed to generate first object probability data indicating a correspondence between the first feature data and the first predetermined feature model. In other words, it may be determined to what extent the first feature data may correspond to the predetermined object, or to what extent it is likely that the first feature data indicates the predetermined object.
[0040] A similar process can be performed on the second feature data based on the second sensing data of the second sensor, in which case the duplicated description will be omitted.
[0041] However, it should be noted that in general, the first and second sensors may be the same or different sensors, for example, the first sensor may be a radar and the second sensor may be an RGB camera, or the like.
[0042] Furthermore, the present disclosure does not limit the second sensor to be configured to operate in the same manner as the first sensor. For example, in the first sensor, the first feature data may be generated based on the first sensing data, while in the second sensor, only the second sensing data may be acquired, and the second feature data may be generated within the object detection circuit, or vice versa.
[0043] The object detection circuitry may be further configured to combine the first object probability data and the second object probability data.
[0044] This allows the sensor fusion requirements to be met, as discussed above.
[0045] The combination may be based on multiplexing, multiplication, addition, etc. of the first and second subject probability data to result in a probability value, but in an embodiment the combination may be based on abstract data structures generated by artificial intelligence, neural networks, may be generated based on supervised or unsupervised training, etc.
[0046] The combination procedure may be followed by normalization of the combined probability data, as is commonly known.
[0047] The predefined object may be detected based on the combined probability data, i.e., the location of the object in space may be determined. Furthermore, since the first and second feature data may be acquired for multiple locations with a (sensor-specific) spatial resolution, the pose of the predefined object may be determined.
[0048] In some embodiments, the object detection circuitry is further configured to detect multiple maxima of the combined probability data indicative of at least one location parameter of the predetermined object, and determine the at least one location parameter of the predetermined object.
[0049] The combined probability data may represent a distribution of probabilities of characteristics (or occurrences) of the predefined object in space, such that high probabilities (i.e., probabilities above a predetermined threshold) may be maximized and probabilities below a predetermined threshold may be minimized, such that probabilities above a predetermined threshold in the combined probability data may be taken into account for determining at least one location parameter of the predefined object.
[0050] In one embodiment, the maximum may indicate the likelihood (or hypothesis) of an object (e.g., single or multiple) at a given location, such that the (symbolic) location of the maximum in probability space indicates the (actual) spatial location and orientation of the object.
[0051] In some embodiments, the detection of multiple (i.e., at least two) maxima may be based on known maximum detection processes; for example, if the maxima are defined to have a Gaussian distribution (or any other distribution, such as Lorentzian, Dirac, Fermi, Poisson, etc.), then a Gaussian distribution (or any other distribution) may be detected, as is commonly known.
[0052] Multiple maxima may be detected such that they are arranged in a spatial order (or pattern) such that a position parameter is determined.
[0053] For example, if the predefined object is a car, two maxima may be detected, which are aligned on a horizontal line of a (virtual) coordinate system, and the two maxima are two meters apart from each other. The first and second sensors may be aligned on the horizontal axis and parallel to the horizontal line.
[0054] In such a case, it may be detected that the detected vehicle pose is parallel to the horizontal axis of the sensor.
[0055] Furthermore, if a sensor is implemented in a vehicle including an object detection circuit according to the present disclosure, it may infer that a detected vehicle may be perpendicular to the vehicle including the object detection circuit, such as by issuing a warning to a driver of the vehicle including the object detection circuit or by initiating emergency braking, without limiting the present disclosure in that respect.
[0056] On the other hand, if two maxima are detected on a line tilted from the horizontal sensor axis, the detected vehicle attitude and / or position angle may be determined.
[0057] However, the present disclosure is not limited to position parameters being pose or angle, as in some embodiments the at least one position parameter includes at least one of position, distance, angle, and pose.
[0058] Thus, the position can be determined and expressed in a (virtual) coordinate system, which can also be shown to the user of the object detection circuitry on a display, etc.
[0059] Distance may be determined directly from the combined probability data and therefore may be concluded from location without limiting this disclosure.
[0060] The pose and / or angle may further include an orientation of a predefined object, for example, if a car is the predefined object, it may be determined where the front and / or rear of the car is.
[0061] In some embodiments, the object detection circuitry is further configured to determine at least one of an accuracy and precision of the detection of the predetermined object.
[0062] For example, measurements may be degraded by environmental influences (e.g., temperature, light conditions, etc.), degraded calibration of one (or both) of the sensors, sensor movement (e.g., when implemented in a car) or predefined object movement (e.g., due to the Doppler effect).
[0063] Thus, the accuracy of detection can be determined based on an assumed error (function), which can take into account any of the above-mentioned conditions without limiting the disclosure in that respect.
[0064] Additionally, the accuracy of the detection can be determined (eg, based on precision), which can indicate whether or not a predefined object corresponds to the detected object.
[0065] In some embodiments, the object detection circuitry is further configured to track the detected predetermined object.
[0066] For example, known object tracking techniques can be applied.
[0067] However, in some embodiments, the object detection circuitry, when applied for detection of a predefined object, may perform the same processing for tracking again, such that tracking may accommodate repeated detection of the predefined object.
[0068] In some embodiments, the object detection circuitry is further configured to generate a first feature map based on the first sensing data, generate a second feature map based on the second sensing data, and transfer the first and second feature maps to a predefined coordinate system.
[0069] The feature map may be, for example, a map (e.g., based on the first and / or second sensing data) depicting the acquired first and second feature data, either in one map or two maps. The feature map may, in some embodiments, include the detected maxima and, without limiting the disclosure in that respect, be represented by a coordinate system from a bird's eye perspective.
[0070] Additionally, feature maps may be generated for different layers of space so that data indicative of the vertical direction can be obtained.
[0071] In general, a feature represented in a feature map may be a detected signal (one or more maxima) that indicates a particular characteristic of a predefined object. For example, a feature may be indicated by a detected signal emanating from a particular part of a car, e.g., a window, a back, etc., having a particular feature signature.
[0072] The predefined coordinate system may be a common coordinate system of the first and second feature maps, or may be different coordinate systems, which may be convertible between each other and may be transferred to the common coordinate system.
[0073] The predefined coordinate system is not limited to a particular type of coordinate system, as it can represent Euclidean coordinates, spherical coordinates, cylindrical coordinates, polar coordinates, etc., and the origin of the predefined coordinate system may be, for example, a specific position of one of the two sensors, a detected object, or any other position in space.
[0074] In some embodiments, the first and second sensors include at least one of a radar sensor, a lidar sensor, a camera, or a time-of-flight sensor, as discussed herein.
[0075] In some embodiments, at least one of the first predefined feature model and the second predefined feature model is based on supervised training of artificial intelligence.
[0076] Artificial intelligence (AI) can use machine learning based methods or explicit feature-based methods, such as edge detection, shape matching via histogram-based methods, template match-based methods, color match-based methods, etc. In some embodiments, machine learning algorithms can be used to perform object recognition, e.g., to compare detected predefined objects with recognized objects to improve detection accuracy, which can be based on at least one of Scale Invariant Feature Transfer (SIFT), Gray Level Co-occurrence Matrix (GLCM), Gabor features, Tubeness, etc. Furthermore, the machine learning algorithm may be based on a classifier technique, where such machine learning algorithm may be based on at least one of random forest, support vector machine, neural net, Bayes net, etc. Furthermore, the machine learning algorithm may apply a deep learning technique, where such deep learning technique may be based on at least one of autoencoder, general malicious network, weakly supervised learning, bootstrap, etc.
[0077] Supervised learning may further be based on regression algorithms, perception algorithms, Bayesian classification, Neiver-Bayer classification, next-nearest neighbor classification, artificial neural networks, and the like.
[0078] In such an embodiment, the artificial intelligence can be fed ground truth data that corresponds to or can be based on a predefined object such that it can learn to assign the first and / or second feature data to the ground truth data, thereby developing or generating the first and / or second feature models.
[0079] In one embodiment, the predefined object is based on a class of predefined objects.
[0080] As mentioned above, the present disclosure may relate to object detection in the automotive field, and in such embodiments, the predefined object classes may include cars, passengers, road signs, streets, houses, trees, living things, traffic lights, etc.
[0081] However, in other embodiments, the object detection circuitry may be applied in surveillance systems, such as warehouses, where the predefined object classes may include customers, employees, shelves, products, etc., without limiting the present disclosure to automotive or surveillance applications. In general, the present disclosure may be applied in any field where object detection may be performed.
[0082] In some embodiments, the object detection circuitry is further configured to iteratively convolve the first feature data with a first predetermined feature model, thereby generating first object probability data, iteratively convolve the second feature data with a second predetermined feature model, thereby generating second object probability data, and iteratively convolve the first and second object probability data, thereby generating combined object probability data.
[0083] For example, multiple first feature data can be obtained (e.g., for different heights or layers, for different angles, etc.), which can each be convolved with the first predetermined feature model in sequential order (i.e., iteratively), which in the present disclosure can be referred to as vertical iterations. Further, multiple second feature data may be obtained, which may be respectively convolved with the second predetermined feature model in successive order in the vertical iteration as well.
[0084] Additionally, the first generated first object probability data (e.g., from the first measurement of the first sensor) may be convolved with the first generated second object probability data (e.g., by summing the respective probabilities), which may be referred to as horizontal iteration, thereby generating combined probability data.
[0085] Some embodiments relate to an object detection method, as discussed herein, that includes the steps of obtaining first feature data based on first sensing data of a first sensor, comparing the first feature data to a first predefined feature model representative of a predetermined object, the first predefined feature model being specific to the first sensor, thereby generating first object probability data, obtaining second feature data based on second sensing data of a second sensor, comparing the second feature data to a second predefined feature model representative of the predetermined object, the second predefined feature model being specific to the second sensor, thereby generating second object probability data, and combining the first object probability data and the second object probability data, thereby generating combined probability data for detecting the predetermined object.
[0086] The object detection method may be performed using object detection circuitry according to the present disclosure, such as a processor (or multiple processors) configured to perform the object detection method.
[0087] In some embodiments, as discussed herein, the object detection method includes detecting multiple maxima in the combined probability data indicative of at least one location parameter of the predefined object, and determining the at least one location parameter of the predefined object. In some embodiments, as described herein, the at least one location parameter includes at least one of a position, a distance, an angle, and an attitude. In some embodiments, the method further includes determining at least one of the accuracy and precision of detecting the predefined object as discussed herein. In an embodiment, the method further includes tracking the detected pre-defined object as discussed herein. In some embodiments, the method further includes generating a first feature map based on the first sensing data, generating a second feature map based on the second sensing data, and transferring the first and second feature maps to a predefined coordinate system, as discussed herein. In some embodiments, the first and second sensors include at least one of a radar sensor, a lidar sensor, a camera, or a time-of-flight sensor, as discussed herein. In some embodiments, as discussed herein, at least one of the first predefined feature model and the second predefined feature model is based on supervised training of artificial intelligence. In some embodiments, the predefined object is based on a class of predefined objects, as described herein. In some embodiments, as discussed herein, the method further includes the steps of iteratively convolving the first feature data with a first predefined feature model, thereby generating first target probability data, iteratively convolving the second feature data with a second predefined feature model, thereby generating second target probability data, and iteratively convolving the first convolved feature data and the second convolved feature data, thereby generating joint probability data.
[0088] The methods described herein are also implemented in some embodiments as a computer program that, when executed on a computer and / or processor, causes the computer and / or processor to perform the methods. In some embodiments, a machine-readable non-transitory recording medium is also provided for storing in a computer program product that, when executed by a processor, such as the processor described above, performs the methods described herein.
[0089] In general, each sensor (i.e., the first and second sensors, but not limited to this) may provide a (predetermined) number of measurements M, where in response to each measurement f, a feature may be assigned to two coordinates lx and lz (or, for short, l = (lx, lz)) in a two-dimensional coordinate system such that a feature map (or multiple feature maps, each measurement indicative of a feature map) may be generated, as discussed herein.
[0090] Each feature map can have a number of i grid cells, where the grid cells can have the same or different sizes, and each grid cell can contain two values representing the number of measurements that detect a particular feature n, expressed as a function such as n(f=1 | l,i), and the number of measurements that do not detect the particular feature, i.e., n(f=0 | l,i).
[0091] Based on all or a subset of the obtained feature maps, we estimate the probability of object presence in the object parameter space p(O(c,θ) = 1 | f1,...,f M ), where O is the object and c is the coordinates of the object (c x , c z ), where θ denotes the pose (i.e., angle) of the object, and (c,θ) is the (3-dimensional) parameter space.
[0092] In particular, c x contains the center of the object in the x direction, and c z contains the center of the object in the z direction, and θ contains the orientation of the object in the xz plane, where θ may be restricted to an interval of values such as (0,2π).
[0093] Thereby, the requirement of overcoming view-dependent object appearance, as mentioned above, can be met.
[0094] The probability p may be determined according to the following formula:
number
[0095] In some embodiments, a discretized representation of the parameter space is generated, which may be represented by a three-dimensional array including a number of discretized points along the x-axis, a number of discretized points along the z-axis, and a number of discretized points along an orientation axis (i.e., angle).
[0096] Furthermore, in some embodiments, p(O(c,θ) = 1 | f1,...,f M ) may be calculated to reduce the number of inference steps to the number of convolutions.
[0097] In some embodiments, the condition p(O(c,θ) = 1 | f i (l) = 1) and p(O(c,θ) = 1 | f i (l)=0) is p(O(c,θ) = 1 | f i (l)=1) = p(O(cl,θ) = 1 | f i (l) = 1) and p(O(c,θ) = 1 | f i (l)=0) = p(O(cl,θ) = 1 | f i Shift invariance is implemented for l, such as (l)=0).
[0098] This allows us to meet the requirement of within-class object invariance as mentioned above.
[0099] Thus, in such an embodiment, if a feature is detected at l (i.e., f = 1) or if it is not detected at l (i.e., f = 0), these conditions can indicate that the object is located at a position having an orientation (cl; θ).
[0100] In some embodiments, the condition p(O(c-1, θ)=1| f i (0) = 1) and p(O(cl,θ) = 1 | f i(0) = 0) is determined based on supervised training from actual or simulated sensor measurements, which in some embodiments may be noisy for a given number of given objects with a given pose.
[0101] Thereby, the sensor noise may be recognized and / or filtered to meet the sensor noise requirements as described above.
[0102] In some embodiments, the local maximum, i.e., p(O(cl,θ)=1| f i (0) = 1) is placed in the parameter space while excluding other parameters based on non-maximum suppression.
[0103] In some embodiments, the location of the local maximum (c, θ) can indicate the pose (e.g., position and orientation) of a given object (or objects), where the accuracy of the determined pose depends on the accuracy of p(O(c, θ)=1 | f1,...,f M ), where p(O(c,θ)=1 | f1,...,f M ) exceeds a certain threshold, accuracy may be awarded.
[0104] Thereby, as mentioned above, the requirement of multiple object detections may be met by providing a list of detections, and the pose accuracy may depend on the resolution of the pose space (c, θ).
[0105] In one embodiment, the accuracy of the determined pose is determined by the function p(O(c,θ)=1 | f i ,...,f M ), where greater precision may correspond to a higher value of the curvature (in other embodiments, greater precision may correspond to a lower value of the curvature).
[0106] The pose (or orientation) detection may be refined with successive measurements based on the accuracy and / or precision of the measurements.
[0107] Returning to FIG. 1, a first coordinate system 1 is depicted which includes two axes 2a and 2b, each of which indicates a position coordinate.
[0108] Further, three signal distributions 3a, 3b and 3c (first feature data) are shown, which are obtained based on first sensing data by a stereo camera as a first sensor. In order to omit repetitive discussion, only signal distribution 3a will be discussed here.
[0109] The signal distribution 3a comprises a first sub-signal 4 and a second sub-signal 5, where the first sub-signal 4 has a stronger signal strength than the second sub-signal 5.
[0110] FIG. 2 shows a second coordinate system 10 including two axes 12a and 12b which correspond to the axes 2a and 2b of FIG.
[0111] Furthermore, three signal distributions 13a, 13b and 13c (second feature data) are shown, which are obtained based on second sensing data by a radar sensor as the second sensor.
[0112] Signal distribution 13a (and similarly 13b and 13c, although not explicitly described) includes a first sub-signal 14, a second sub-signal 15, and a third sub-signal 16, which are grouped for illustrative reasons by different patterns, with the pattern strengths corresponding to the respective signal strengths, i.e., first sub-signal 14 has the strongest signal strength of the first to third sub-signals 14-16, second sub-signal 15 has the second strongest signal strength of the first to third sub-signals 14-16, and third sub-signal 16 has the third strongest signal strength of the first to third sub-signals 14-16.
[0113] In the presently described embodiment, the signal distributions 3a, 3b, 3c, 13a, 13b, and 13c described with reference to Figures 1 and 2 are obtained by acquiring the first and second sensing data of three cars as predetermined objects, that is, the signal distributions 3a to 3c and 13a to 13c indicate cars.
[0114] FIG. 3 illustrates a template (i.e., feature model) 20 for a stereo camera as a sensor, which is based on supervised training of artificial intelligence.
[0115] The templates 20 represent different poses of the vehicles 24-39, each of which is assigned a probability as discussed herein.
[0116] Furthermore, different intensities of the signal are symbolically depicted by different lines 21-23, with line 21 representing the strongest intensity, line 22 representing the second strongest intensity, and line 23 representing the weakest intensity detected, without limiting the present disclosure in that respect, as a continuous distribution of intensities may be detected as well, as is commonly known.
[0117] The assignments are displayed in the following table: Each reference symbol (corresponding to the pose shown in the figure) is assigned a probability, and the pose is represented by an angle θ. [Table 1]
[0118] FIG. 4 shows a template (ie, feature model) 40 for radar as a sensor, based on supervised training of artificial intelligence.
[0119] Template 40 represents different poses of the vehicle that correspond to the ordering (probabilities and angles) of template 20 of FIG. 3, so a repeated description (along with the reference numbers) is omitted.
[0120] The different intensities of the signal are symbolically depicted by differently hatched ellipses 41-43, with ellipse 41 representing the strongest intensity, ellipse 42 representing the second strongest intensity and ellipse 43 representing the weakest intensity detected, although as is generally known, a continuous distribution of intensities may be detected as well and this disclosure is not limited in that respect.
[0121] The log-probabilities of the object poses are determined using templates 20 and 40, as discussed herein, and the object pose probability distributions are fused to generate a fused pose probability distribution (or fused feature model).
[0122] FIG. 5 illustrates the fused pose probability distribution 50 with the same ordering of vehicle poses as in FIGS. 3 and 4 so that repetitive description (along with the reference numbers) is omitted.
[0123] The different intensities of the signal are symbolically depicted by different hatched rectangles 51-53, with rectangle 51 representing the strongest intensity, rectangle 52 representing the second strongest and rectangle 53 representing the weakest intensity detected, without limiting the present disclosure, as a continuous distribution of intensities may be detected as well, as is commonly known.
[0124] The log-probabilities of the first and second feature data (of Figures 1 and 2) are fused into a combined pose probability distribution 50 for each of the three objects (cars) in Figures 1 and 2 resulting in the poses of the three cars.
[0125] The results of the comparison are displayed in Figure 6.
[0126] FIG. 6 illustrates a coordinate system 60 that includes two axes 61a and 61b, each of which indicates a position coordinate.
[0127] Additionally, three detected objects 62a, 62b, and 62c are shown overlaid with the sum of features 63 from feature maps 1 and 10 of Figures 1 and 2, respectively. The display of the three detected objects further includes pointers 64a, 64b, and 64c that indicate the pose of each detected object 62a, 62b, and 62c, respectively.
[0128] FIG. 7 shows a block diagram of a method 70 according to the present disclosure.
[0129] Sensor measurements are performed at 71. In this embodiment, a stereo camera image pair is acquired at 711 as the first sensing RAW data, and radar data is acquired at 712 as the second sensing RAW data.
[0130] Low level processing is performed on each sensing raw data.
[0131] For stereo camera image pairs, calibration and disparity estimation is performed at 713. For radar data, calibration and fast Fourier transform is performed at 714.
[0132] At 72, the low level processed first and second sensed raw data are transferred to respective feature maps such that the low level processed first sensed raw data is transferred to a bird's eye view feature map at 721 and the low level processed second sensed raw data is transferred to a bird's eye view (panoramic) feature map at 722.
[0133] At 73, the log-probabilities are determined from the feature maps, as discussed herein.
[0134] In particular, at 731 the pose log probability of the stereo camera feature maps is determined based on the previously trained conditional probabilities that are fed into the pose log probability determination at 732 .
[0135] Further, at 733, a pose log probability volume for the radar sensor is determined based on the previously trained conditional probabilities which are fed into a pose log probability volume determination at 734.
[0136] The determined pose log probability quantities are combined (ie, fused) at 735 as discussed herein.
[0137] At 74, local maxima are determined and processed as discussed herein.
[0138] Based on the combined probabilities, the maximum is found with a non-maximum suppression algorithm in 741, as discussed herein.
[0139] At 742, the attitude, accuracy, and precision are determined based on the detected maximum value as described herein.
[0140] At 75, further processing is performed based on the determined pose (i.e., position and orientation), accuracy and precision, as described herein.
[0141] Further refinement of the detected pose is performed at 751 and the detected object (or detected objects) are tracked at 752, particularly as discussed herein.
[0142] FIG. 8 illustrates generally a method 80 for generating a feature map.
[0143] A coordinate system 81 is depicted that includes a number of cells 82 defined by coordinates lx and lz (ie, lx, lz).
[0144] Additionally, coordinate system 81 includes multiple detections 83 that overlap the boundaries of cell 82 , without limiting the present disclosure, as detections 83 may generally lie entirely within cell 82 .
[0145] A detection 83 is assigned to the one of the plurality of cells 82 whose detection signal is most present, ie a cell-selective accumulation of the detections 83 is performed at 84 .
[0146] A coordinate system 85 is depicted, which basically corresponds to the coordinate system 81, and a duplicate explanation is omitted.
[0147] However, coordinate system 85 differs from coordinate system 81 in that detections 83 are accumulated into features 86 .
[0148] For each cell corresponding to a particular coordinate, the number of detections is determined, where an empty cell, e.g., cell (4,3), is represented by n(f = 1 | l = ((4,3), i) = 0; cells with one detection or feature, e.g., cells (5,16), (23,13), and (17,16), are represented by n(f = 1 | l = ((5,16), i) = 1, n(f = 1 | l = ((23,13), i) = 1, and n(f = 1 | l = ((16,17), i) = 1, respectively; and a cell with two detections or features, e.g., cell (19,5), is represented by n(f = 1 | l = ((19,5), i) = 2, without limiting the present disclosure with respect to the maximum number of detections in a cell.
[0149] FIG. 9 illustrates generally a method 90 for determining object probability data according to the present disclosure.
[0150] In the following description of method 90, only measurements from two sensors 91 and 91′ are taken into account, however, as one skilled in the art can take from FIG. 9, the present disclosure is not limited to two sensors, as it can be envisioned for any number of sensors with any number of measurements.
[0151] First feature data based on sensor measurements of sensor 91 is convolved at 93 with a predefined feature model 92 (ie, predefined conditional probabilities based on supervised training that is specific to sensor 91).
[0152] The second feature data is based on sensor measurements of sensor 91' and is convolved with a predefined feature model 92' in 93' (ie, predefined conditional probabilities based on supervised training that is specific to sensor 91').
[0153] Further, a sum 94 of the convoluted data of 93 and the convoluted data of 93' is determined, which serves as a basis for further convolution of further sensing data to detect an object, or in one embodiment, as a basis for detecting an object.
[0154] FIG. 10 illustrates a block diagram of a vehicle 100 including a CPU 101 configured to function as an object detection circuit according to the present disclosure.
[0155] The vehicle 100 further includes a stereo camera 102 for acquiring first sensing data and a radar 103 for acquiring second sensing data, which are supplied to the CPU 101 so that first and second feature data are generated.
[0156] However, in some embodiments, the first and second feature data are generated within the first and second sensors or within specific circuitry for generating the first and / or second feature data, as discussed herein.
[0157] For example, a feature map may be computed for each sensor and transferred to a fusion and detection unit configured to fuse the feature maps, determine a probability, and detect an object.
[0158] It should be understood that the above-described embodiments describe methods with exemplary ordering of method steps. However, the particular ordering of method steps is provided for illustrative purposes only and should not be construed as binding. For example, the ordering of 91 and 91' in the embodiment of FIG. 9 may be swapped. Also, the order of 711 and 712 may be swapped in the embodiment of Figure 7. Also, the order of 721 and 722 may be changed in the embodiment of Figure 7. Other variations in the order of method steps will be apparent to those skilled in the art.
[0159] In one embodiment, a non-transitory machine-readable storage medium is also provided that stores a computer program product that, when executed by a processor, such as the processor described above, performs the methods described.
[0160] All units and entities described in this specification and claimed in the accompanying claims may be implemented as integrated circuit logic, for example on a chip, unless otherwise stated, and the functions provided by such units and entities may be implemented by software, unless otherwise stated.
[0161] In so far as the embodiments of the disclosure described above are implemented, at least in part, using software-controlled data processing apparatus, it will be understood that computer programs providing such software control, and the transmission, storage, or other media on which such computer programs are provided, are contemplated as aspects of the present disclosure.
[0162] Furthermore, it should be noted that the present disclosure is illustratively described with respect to automobile detection and in-vehicle implementation, but is not limited to the described embodiment. In particular, the present disclosure can be applied in any ground-based or sea-based navigation system, such as driver assistance systems in automobiles, autonomous vehicles, robots, boats, ships, etc., in each case or situation where the surrounding scenario can be represented in a two-dimensional coordinate system.
[0163] The present technology can also be configured as follows. (1) acquiring first feature data based on first sensing data of a first sensor; comparing the first feature data to a first predetermined feature model representative of a predetermined object, the first predetermined feature model being specific to the first sensor, thereby generating first object probability data; acquiring second feature data based on second sensing data of a second sensor; comparing the second feature data to a second predetermined feature model representative of a predetermined object, the second predetermined feature model being specific to the second sensor, thereby generating second object probability data; Combining the first object probability data and the second object probability data, thereby generating combined probability data for detecting a predetermined object. It was configured as Object detection circuit. (2) detecting a plurality of maxima of the joint probability data indicative of at least one position parameter of the given object; determining said at least one position parameter of said given object It is further configured as follows: An object detection circuit as described in (1). (3) The at least one position parameter includes at least one of a position, a distance, an angle, and an attitude. An object detection circuit according to (1) or (2). (4) further configured to determine at least one of accuracy and precision of the detection of the predetermined object. The object detection circuit according to any one of (1) to (3). (5) further configured to track the detected predetermined object. The object detection circuit according to any one of (1) to (4). (6) generating a first feature map based on the first sensing data; generating a second feature map based on the second sensing data; Transforming the first feature map and the second feature map into a predetermined coordinate system. It is further configured as follows: The object detection circuit according to any one of (1) to (5). (7) The first sensor and the second sensor include at least one of a radar sensor, a lidar sensor, a camera, or a time-of-flight sensor. The object detection circuit according to any one of (1) to (6). (8) At least one of the first predetermined feature model and the second predetermined feature model is based on supervised training of artificial intelligence. The object detection circuit according to any one of (1) to (7). (9) The predetermined object is based on a class of the predetermined object. The object detection circuit according to any one of (1) to (8). (10) iteratively convolving the first feature data with the first predetermined feature model, thereby generating the first object probability data; iteratively convolving the second feature data with the second predetermined feature model, thereby generating the second object probability data; Iteratively convolving the first object probability data and the second object probability data, thereby generating the joint probability data for the object. It is further configured as follows: The object detection circuit according to any one of (1) to (9). (11) acquiring first feature data based on first sensing data of a first sensor; comparing the first feature data to a first predetermined feature model representative of a predetermined object, the first predetermined feature model being specific to the first sensor, thereby generating first object probability data; acquiring second feature data based on second sensing data of a second sensor; comparing the second feature data to a second predefined feature model representative of a predefined object, the second predefined feature model being specific to the second sensor, thereby generating second object probability data; combining the first object probability data and the second object probability data, thereby generating combined probability data for detecting a predetermined object; 1. An object detection method comprising: (12) detecting a plurality of maxima of said joint probability data indicative of at least one position parameter of said predetermined object; determining said at least one position parameter of said given object; Also includes The object detection method according to (11). (13) The at least one position parameter includes at least one of a position, a distance, an angle, and an attitude. The object detection method according to (11) or (12). (14) determining at least one of the accuracy and precision of the detection of the predetermined object; The object detection method according to any one of (11) to (13). (15) The method further includes the step of tracking the detected predetermined object. The object detection method according to any one of (11) to (14). (16) generating a first feature map based on the first sensing data; generating a second feature map based on the second sensing data; transforming the first feature map and the second feature map into a predetermined coordinate system; Also includes The object detection method according to any one of (11) to (15). (17) The first sensor and the second sensor include at least one of a radar sensor, a lidar sensor, a camera, or a time-of-flight sensor. The object detection method according to any one of (11) to (16). (18) At least one of the first predetermined feature model and the second predetermined feature model is based on supervised training of artificial intelligence. The object detection method according to any one of (11) to (17). (19) The predetermined object is based on a class of the predetermined object. The object detection method according to any one of (11) to (18). (20) iteratively convolving the first feature data with the first predetermined feature model, thereby generating the first object probability data; iteratively convolving the second feature data with the second predetermined feature model, thereby generating the second object probability data; iteratively convolving the first object probability data and the second object probability data, thereby generating said joint probability data for the object; Also includes The object detection method according to any one of (11) to (19). (21) A computer program comprising a program code that, when executed on a computer, causes the computer to execute the method according to any one of (11) to (20). (22) A machine-readable non-transitory storage medium storing a computer program product that, when executed by a processor, performs any one of the methods described in (11) to (20).
Claims
1. acquiring first feature data based on first sensing data of a first sensor; comparing the first feature data to a first predetermined feature model representative of a predetermined object, the first predetermined feature model being specific to the first sensor, thereby generating first object probability data; acquiring second feature data based on second sensing data of a second sensor; comparing the second feature data to a second predetermined feature model representative of a predetermined object, the second predetermined feature model being specific to the second sensor, thereby generating second object probability data; combining the first object probability data and the second object probability data, thereby detecting a plurality of maxima of combined probability data indicative of at least one location parameter including at least one of a position, a distance, an angle, and an attitude of a predetermined object; and determining at least one position parameter including at least one of a position, a distance, an angle, and an attitude of the predetermined object based on a result of the detection. Object detection circuit.
2. and determining at least one of an accuracy and precision of the detection of the predetermined object based on a result of the detection.
2. The object detection circuit of claim 1.
3. and further configured to track the detected predetermined object.
2. The object detection circuit of claim 1.
4. generating a first feature map based on the first sensing data; generating a second feature map based on the second sensing data; Transforming the first feature map and the second feature map into a predetermined coordinate system. It is further configured as follows:
2. The object detection circuit of claim 1.
5. The first sensor and the second sensor include at least one of a radar sensor, a lidar sensor, a camera, or a time-of-flight sensor.
2. The object detection circuit of claim 1.
6. At least one of the first predefined feature model and the second predefined feature model is based on supervised training of artificial intelligence.
2. The object detection circuit of claim 1.
7. The given object is based on a class of the given object.
2. The object detection circuit of claim 1.
8. iteratively convolving the first feature data with the first predetermined feature model, thereby generating the first object probability data; iteratively convolving the second feature data with the second predetermined feature model, thereby generating the second object probability data; Iteratively convolving the first object probability data and the second object probability data, thereby generating the joint probability data for the object. It is further configured as follows:
2. The object detection circuit of claim 1.
9. acquiring first feature data based on first sensing data of a first sensor; comparing the first feature data to a first predetermined feature model representative of a predetermined object, the first predetermined feature model being specific to the first sensor, thereby generating first object probability data; acquiring second feature data based on second sensing data of a second sensor; comparing the second feature data to a second predefined feature model representative of a predefined object, the second predefined feature model being specific to the second sensor, thereby generating second object probability data; combining the first object probability data and the second object probability data, thereby detecting a plurality of maxima of combined probability data indicative of at least one location parameter including at least one of a position, a distance, an angle, and an attitude of a predetermined object; determining at least one position parameter including at least one of a position, a distance, an angle, and an attitude of the predetermined object based on the result of the detection; The object detection method includes:
10. and determining at least one of accuracy and precision of the detection of the predetermined object based on the result of the detection. The object detection method according to claim 9.
11. and tracking the detected predetermined object. The object detection method according to claim 9.
12. generating a first feature map based on the first sensing data; generating a second feature map based on the second sensing data; and transforming the first feature map and the second feature map into a predetermined coordinate system. The object detection method according to claim 9.
13. The first sensor and the second sensor include at least one of a radar sensor, a lidar sensor, a camera, or a time-of-flight sensor. The object detection method according to claim 9.
14. At least one of the first predefined feature model and the second predefined feature model is based on supervised training of artificial intelligence. The object detection method according to claim 9.
15. The given object is based on a class of the given object. The object detection method according to claim 9.
16. iteratively convolving the first feature data with the first predetermined feature model, thereby generating the first object probability data; iteratively convolving the second feature data with the second predetermined feature model, thereby generating the second object probability data; iteratively convolving the first object probability data and the second object probability data, thereby generating said joint probability data for the object; Also includes The object detection method according to claim 9.
Citation Information
Patent Citations
Augmenting layer-based object detection with deep convolutional neural networks
JP2017146957A
Object recognition and classification using multiple sensor modalities
US20180068206A1
Information processing device, information processing method and program
WO2017057056A1
