Trackable object for distance estimation

Abstract

The invention relates to a trackable object for distance estimation. A method for estimating a distance of an object from a vehicle is disclosed. The method comprises the following steps: acquiring an image frame comprising an object; identifying a first key point set of the object from the image frame; obtaining a second key point set related to the first key point set from the nonvolatile memory; obtaining spatial information about a second keypoint set from the non-volatile memory; and evaluating the distance of the object from the vehicle based on the first set of keypoints and the spatial information about the second set of keypoints.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and more particularly to methods and apparatus for various object tracking purposes (e.g., bounding box generation, distance estimation, automatic calibration) for autonomous driving systems. Background Art

[0002] As computing and vehicle technologies continue to advance, features related to automation have become more powerful and widely available, and are able to control vehicles in a wider variety of environments. For example, for automobiles, the Society of Automotive Engineers (SAE) has established a standard (J3016) that identifies six levels of driving automation from "no automation" to "full automation." The SAE standard defines Level 0 as "no automation," where the human driver performs all aspects of the dynamic driving task full time, even when enhanced by warning or intervention systems. Level 1 is defined as "driver assistance," where the vehicle controls steering or acceleration / deceleration (but not both) in at least some driving modes, allowing the operator to perform all remaining aspects of the dynamic driving task. Level 2 is defined as "partial automation," where the vehicle controls steering and acceleration / deceleration in at least some driving modes, allowing the operator to perform all remaining aspects of the dynamic driving task. Level 3 is defined as "conditional automation," where, for at least some driving modes, the automated driving system performs all aspects of the dynamic driving task, with the expectation that the human driver will respond appropriately to intervention requests. Level 4 is defined as "high automation," where, only for specific conditions, the automated driving system performs all aspects of the dynamic driving task, even if the human driver does not respond appropriately to intervention requests. Specific conditions for Level 4 may be, for example, specific types of roads (e.g., highways) and / or specific geographic areas (e.g., geographically isolated metropolitan areas that have been appropriately mapped). Finally, Level 5 is defined as "full automation," where the vehicle is able to operate without operator input under all conditions.

[0003] Tracking of stationary and moving objects is a key function of autonomous driving technology. The basic challenge of any automatic correlation technology involves collecting and interpreting information about the vehicle's surroundings. For example, distance measurement is widely used in advanced driver assistance systems to provide safety alerts to drivers, so distance measurement is very important for autonomous driving. Calibration of sensor parameters (e.g., cameras, etc.) is also a key part, regardless of whether it is in an application (e.g., optical motion capture, image measurement, or machine vision application). The accuracy of the calibration results and the stability of the algorithm directly affect the accuracy of the results produced by the operation of the camera sensor. In addition, the calibration accuracy of the camera can directly affect the capture accuracy of the entire optical motion capture system, and therefore can have a limited error tolerance. For example, a small calibration error can produce a substantial deviation. Therefore, performing the desired camera calibration process is a prerequisite for subsequent operations.

[0004] There are ongoing efforts to improve the detection and tracking of surrounding objects. With such improvements, autonomous vehicles are increasingly able to operate reliably in increasingly complex environments and adapt to expected and unexpected interactions within the environment. For example, in order to operate safely, autonomous vehicles should consider objects such as vehicles, people, trees, animals, buildings, signs, and poles when planning a path through the environment. Typically, a sequence of image frames acquired from one or more sensors is processed by an autonomous driving system to detect and track objects. Typically, manual identification and tracking of moving targets is slow, intensive, and in many cases impractical. Automated solutions for dealing with problems related to autonomous driving have been proposed in recent years. However, techniques related to the automatic processing of continuous image frames have limitations in identifying targets in the field of view of the image frame. In addition, algorithms for bounding box generation, distance estimation, and automatic calibration in one or more image frames do not provide satisfactory subjective quality. In addition, due to the inherent uncertainty of real-world data, many of these algorithms cannot process the data optimally.

[0005] Video / image detection and tracking (including bounding box generation, distance estimation, and automatic calibration) are integral to autonomous driving systems, as they need to constantly monitor their surroundings. Therefore, it is important to develop algorithms that reduce computational complexity while maintaining stability and accuracy of automated driving operations. In addition, advanced algorithms are needed that can optimally process real-time images to reliably detect both stationary and moving objects. Summary of the invention

[0006] The present disclosure proposes a method and apparatus for object detection and tracking (including bounding box generation, distance estimation, and automatic calibration) for use in an autonomous driving system. In some embodiments, a method for tracking an object in one or more image frames is disclosed. The method may include: acquiring a first image frame containing an object; determining a first key point set of the object in the first image frame and a first descriptor set associated with the first key point set; determining a second key point set according to the first key point set and determining a second descriptor set according to the first descriptor set; generating a bounding box of the object based on the second key point set and the second descriptor set. The so-called bounding box represents an area in which the target object is located in the target scene displayed in the current image frame, and the area includes a position and a range. More specifically, the bounding box can be represented as a virtual detection box of any shape surrounding the detected / tracked object, such as a rectangular box, an elliptical box, or a hexagonal box. These bounding boxes are used to mark the position and range of the target object in the target scene. Using a rectangular bounding box as an example, the position annotated by the rectangular box can be understood as the coordinates at the corner or center of the area, and the range annotated by the rectangular box can be understood as the length and width of the area. The area surrounded by the rectangular box is the area of ​​the target object in the target scene. Therefore, bounding boxes are one of the key elements and main image processing tools for autonomous driving as they are integrated in self-driving or autonomous vehicles to identify objects on the road, such as buildings, traffic signals, any obstacles, etc. Bounding boxes help annotate obstacles and enable the autonomous driving system to operate safely, for example, to prevent accidents in congested situations. Although this article uses bounding boxes as an example of bounding regions to describe examples, it will be understood by those of ordinary skill in the art that any other suitable bounding regions may be used instead of bounding boxes, such as bounding circles, bounding ellipses, or any other suitable shapes representing regions of tracked objects.

[0007] In some embodiments, the method may further include: acquiring a second image frame, the second image frame containing the object. In such an embodiment, the second key point set has a motion pattern identified by comparing the key points in the first image frame with the key points in the second image frame, the motion pattern being different from the motion pattern of another key point set separate from the second key point set. In different embodiments, the step of determining the second key point set based on the first key point set and the step of determining the second descriptor set based on the first descriptor set are performed by comparing the distance between the corresponding first key point sets in the first image frame and the second image frame with a threshold value based on a ratio test, based on a cross check, or based on a grid-based motion statistics method. In some embodiments, the method may further include: acquiring a second image frame containing the object; acquiring a first reference bounding box of the object from the first image frame; acquiring a second reference bounding box of the object from the second image frame; generating a bounding box of the object based on the first reference bounding box and the second reference bounding box. In such an embodiment, generating a bounding box of the object based on the first reference bounding box and the second reference bounding box includes: averaging a plurality of reference bounding boxes.

[0008] In some embodiments, another method for estimating the distance of an object from a vehicle during driving is disclosed. The method may include: acquiring an image frame containing an object; identifying a first key point set of the object from the image frame; acquiring a second key point set related to the first key point set from a non-volatile memory; acquiring spatial information about the second key point set from the non-volatile memory; and evaluating the distance of the object from the vehicle based on the first key point set and the spatial information about the second key point set.

[0009] In some embodiments, the image frame is captured by a device associated with a vehicle, a stationary object, or a drone. In some embodiments, assessing the distance of the object from the vehicle includes matching geometric parameters of one or more key points in the second set of key points with geometric parameters of one or more key points in the first set of key points. In some embodiments, the non-volatile memory includes descriptors associated with the first set of key points or the second set of key points. In such embodiments, the distance of the object from the vehicle is assessed based on the descriptors. In some embodiments, the object is a second vehicle, and the method further includes obtaining a make, model, or model year of the second vehicle from the non-volatile memory. In some embodiments, the non-volatile memory is a cloud database.

[0010] In some further embodiments, another method for automatically calibrating a camera sensor during driving is disclosed. The method may include: acquiring a first image frame including an object in motion by a camera; acquiring a second image frame including an object by a camera; determining a first set of key points of the object in the first image frame; determining a second set of key points of the object in the second image frame; acquiring first spatial information about the first set of key points from a non-volatile memory; acquiring second spatial information about the second set of key points from the same or different non-volatile memory; and automatically calibrating the camera based on the first spatial information and the second spatial information.

[0011] In some embodiments, the first image frame and the second image frame are within a sequence of different image frames. In some embodiments, the first image frame is earlier in time than the second image frame in the sequence. In some embodiments, the camera is associated with a vehicle, a stationary object, or a drone. In some embodiments, the non-volatile memory includes a descriptor associated with the first set of key points or the second set of key points. In such embodiments, the spatial information is retrieved from the non-volatile memory based on the descriptor. In some embodiments, the non-volatile memory is a cloud database.

[0012] In some embodiments, the disclosed method may be operated by an apparatus of an autonomous driving system. The apparatus may include at least one processor and a memory storing instructions. These instructions, when executed by at least one processor, cause the at least one processor to perform operations of the disclosed method for object tracking for an autonomous driving system. For example, in some embodiments, the disclosed method may be programmed as computer executable instructions stored in a non-transitory computer-readable medium. The non-transitory computer-readable medium directs the processor of the computer to perform the disclosed method when loaded into the computer. The non-transitory computer-readable medium may include at least one from the group consisting of a hard disk, a CD-ROM, an optical storage device, a magnetic storage device, a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an EPROM, an electrically erasable programmable read-only memory, and a flash memory.

[0013] It should be understood that all combinations of the above concepts and additional concepts described in more detail herein are considered to be part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are considered to be part of the subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the embodiments of the present disclosure or related technologies, the following drawings to be described in the embodiments are briefly introduced. Obviously, these drawings only show some embodiments of the present disclosure, and ordinary technicians in this field can obtain other drawings based on these drawings without creative work. The arrows in the drawings indicate the relationship of using the component from which the arrow starts to train / apply the component to which the arrow points. The embodiments of the present disclosure will be more fully understood and appreciated based on the detailed description below in conjunction with the drawings, in which:

[0015] Figure 1A is an image frame illustrating an example of a bounding box according to some embodiments of the present disclosure;

[0016] Figure 1B is an image frame illustrating an example of a bounding box according to some embodiments of the present disclosure;

[0017] Figure 1C is an image frame illustrating an example of a bounding box according to some embodiments of the present disclosure;

[0018] Figure 2A According to some embodiments of the present disclosure, Figure 1A A simplified diagram of an image frame of a bounding box corresponding to the bounding box shown;

[0019] Figure 2B According to some embodiments of the present disclosure, Figure 1B A simplified diagram of an image frame of a bounding box corresponding to the bounding box shown;

[0020] Figure 2C According to some embodiments of the present disclosure, Figure 1C A simplified diagram of an image frame of a bounding box corresponding to the bounding box shown;

[0021] Figure 3 is a block diagram illustrating an example of a proposed image processing system according to some embodiments of the present disclosure;

[0022] Figure 4 is a diagram illustrating an example of matching corresponding sets of key points for various purposes (e.g., generating bounding boxes, evaluating distances, automatic calibration of cameras, etc.) according to some embodiments of the present disclosure;

[0023] Figure 5 is a flow chart illustrating an example of operations for generating a bounding box according to some embodiments of the present disclosure;

[0024] Figure 6 is a flow chart illustrating another example of the operation of generating a bounding box according to some embodiments of the present disclosure;

[0025] Figure 7is a flow chart illustrating an example of the operation of distance estimation according to some embodiments of the present disclosure;

[0026] Figure 8 is a flowchart illustrating an example of the operation of camera auto-calibration according to some embodiments of the present disclosure; and

[0027] Fig. 9 An exemplary hardware and software environment for an autonomous vehicle according to some embodiments of the present disclosure is illustrated.

[0028] It should be understood that for simplicity and clarity of illustration, the elements shown in the figures are not necessarily drawn to scale. For example, for clarity, the size of some elements may be exaggerated relative to other elements. In addition, where deemed appropriate, reference numerals may be repeated in the drawings to indicate corresponding or similar elements. DETAILED DESCRIPTION

[0029] With reference to the accompanying drawings, the embodiments of the present disclosure are described in detail with technical problems, structural features, purposes and effects of implementation, as follows. Specifically, the terms in the embodiments of the present disclosure are only used for the purpose of describing specific embodiments, rather than limiting the present disclosure. In the following detailed description, many specific details are set forth to provide a thorough understanding of the present invention. However, those skilled in the art will understand that the present invention can be implemented without these specific details. In other cases, well-known methods, processes and components are not described in detail to avoid confusing the present invention. The subject matter of the present invention is specifically pointed out and clearly claimed in the concluding part of the specification. However, with respect to the organization and operation method of the present invention and its purposes, features and advantages, the present invention can be best understood by referring to the following detailed description when read in conjunction with the accompanying drawings. Because the illustrated embodiments of the present invention can be implemented using electronic components and circuits known to those skilled in the art for the most part, in order to understand and appreciate the underlying concepts of the present invention, and in order not to confuse or disperse the teachings of the present invention, the details will not be explained to a greater extent than what is considered necessary as shown above. For example, the specification and / or the accompanying drawings may relate to a processor or a processing circuit. The processor may be a processing circuit. The processing circuit may be implemented as a central processing unit (CPU) and / or one or more other integrated circuits such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a full custom integrated circuit, etc., or a combination of these integrated circuits.

[0030] The following description and / or drawings may relate to images or image frames. An image is an example of a media unit. Any reference to an image may be necessarily applied to a media unit. A media unit may be an example of a sensing information unit (SIU). Any reference to a media unit may be necessarily applied to any type of natural signal, such as but not limited to a signal generated by nature, a signal representing human behavior, a signal representing an operation related to a vehicle signal, a geodetic signal, a geophysical signal, a text signal, a digital signal, a time series signal, etc. Any reference to a media unit may be necessarily applied to a SIU. The SIU may be of any type and may be sensed by any type of sensor, such as a visual light camera. An audio sensor. Sensors that may sense infrared, radar imaging, ultrasound, electro-optical, radiography, light detection and ranging (LIDAR), thermal sensors, passive sensors, active sensors, etc. Sensing may include generating a sample (e.g., pixel, audio signal, etc.) representing a signal sent or otherwise arriving at a sensor. The SIU may have one or more images, one or more video clips, text information about one or more images, text describing motion information, etc.

[0031] Any combination of any modules or units listed in any of the drawings, any part of the specification and / or any claims may be provided. Any of the units and / or modules illustrated in the present application may be implemented with hardware and / or code, instructions and / or commands stored in a non-transitory computer-readable medium, and may be included in a vehicle, outside a vehicle, in a mobile device, in a server, etc. The vehicle may be any type of vehicle, such as a ground transportation vehicle, an aviation vehicle, or a watercraft. The vehicle is also referred to as an ego vehicle. It should be understood that autonomous driving includes at least partially automatic (semi-automatic) driving of a vehicle, including all L2 level types or higher level types defined in the SAE standard.

[0032] The present disclosure proposes a method and apparatus for object detection and tracking (including bounding box generation, distance estimation, automatic calibration) for use in autonomous driving systems. In doing so, various methods and systems for processing images for autonomous driving vehicles are disclosed. Specifically, multiple key points (i.e., characteristic points, feature points, or points of interest) and associated descriptors can be provided for various autonomous driving purposes, for example, in a continuous process (i.e., across a time range / timeline, generating a bounding box suitable for its correct size / position, estimating distance, automatic calibration, etc.). Key points are a simplified way to represent complex objects (e.g., people, trees, animals, buildings, signs and poles, vehicles, etc.) by using a limited number of points, which generally correspond to the distinguishing / important features of the object. In other words, key points are a compact and structured way to express object information that is otherwise encoded in pixels and sensor scans. Preferably, key points can be selected and image patches (patches) around these key points can be processed so that the key points are invariant to image scaling changes and / or rotations, and provide robust matching across a considerable range of distortions, changes in viewpoint and noise, and / or changes in illumination. After detecting and locating key points in an image, these key points can be identified or described by using various descriptors. A descriptor is a vector describing the characteristics of a neighborhood of a key point so that the key point can be identified based on other key points in a given image, and the corresponding key point can be found in different images of the same scene or the same object taken at different time points or from different conditions (e.g., different times, different sensor angles, or different lighting conditions). For example, a descriptor can represent visual features of the content in an image, such as other image features such as shape, color, texture, and / or rotation. In order to be well suited for tasks such as object detection and tracking, descriptors can preferably be different in the sense that a single feature can be correctly matched with a large database of features from multiple target images with high probability. In different embodiments, the descriptor can be image-based or semantic-based. The present disclosure proposes various embodiments for obtaining key point descriptors through different matching algorithms to reduce computational costs, which allow increasing the computational efficiency of an autonomous driving system when detecting and / or tracking objects in real time and / or training models.

[0033] Referring now to the drawings, in which like numerals represent like parts throughout the several views, Figures 1A-1C is an image frame illustrating an example of generating a bounding box. Figures 1A-1C An image of a road and identification of an object (e.g., a vehicle) according to some embodiments is illustrated. The vehicle may be controlled by an autonomous driving system described below. As the vehicle travels on a road, images may be captured from a camera sensor coupled to the vehicle. For example, the image depicts a road lane, white lines representing road lane boundaries, and a moving vehicle. Figure 1A A first image frame 102 is shown including a first bounding box 104; Figure 1B A second image frame 112 including a second bounding box 114 is shown; Figure 1C The third image frame 122 including the third bounding box 124 is illustrated. The images / image frames 102, 112, and 122 are a collection of consecutive image frames, which may be video frames from a video. For example, the image frames 112 and 122 may be consecutive image frames starting from the first image frame 102. The bounding boxes 104, 114, and 124 are associated with an object 130, which is a vehicle in this example. The bounding boxes 104, 114, and 124 may be generated by an object detector (not shown) that identifies the object 130 in the image frames 102, 112, and 122. In an embodiment, in order to determine that the same object 130 is being identified in each of the frames 102, 112, and 122, insight may be obtained from a tracking module (not shown) that assigns an object ID to the object 130 in a particular single frame 102, 112, or 122. Alternatively, as will be explained in more detail below, a keypoint matching process may be performed to establish that the same object 130 is imaged, identified, and tracked in each of the frames 102, 112, and 122. Figures 1A-1C As can be seen in FIG. 1 , the bounding boxes 104 , 114 , and 124 change size and position from one frame to another, wherein the tracked object 130 is in motion.

[0034] Figures 2A-2C supply Figures 1A-1C , which includes bounding boxes 104, 114, and 124 in image frames 102, 112, and 122, respectively. Figure 2A , the first bounding box 104 has a height h0 and a width w0. The center position of the first bounding box 104 is at the pixel coordinates (x0, y0) in the first image frame 102, wherein x0 represents the pixel coordinates of the center position of the first bounding box 104 along the horizontal direction, and y0 represents the pixel coordinates of the center position of the first bounding box 104 along the vertical direction. Similarly, the second bounding box 114 has a height h1 and a width w1, and the center position of the second bounding box 114 is at the pixel coordinates (x1, y1) in the second image frame 112. In addition, the third bounding box 124 has a height h2 and a width w2, and the center position of the third bounding box 124 is at the pixel coordinates (x2, y2) in the third image frame 122.

[0035] like Figures 2A-2CAs shown, the bounding box experiences changes in height and width across image frames 102, 112, and 122, even though the bounding box is associated with the same object 130. Changes in bounding box size may lead to errors in the tracking of object 130. For example, a size difference between the height and width may falsely indicate that object 130 has decreased or expanded in size when object 130 has not actually experienced any shrinkage or expansion. The size changes also create a visual appearance of shrinkage and / or expansion of the bounding box when displayed, which can hinder visual tracking of object 130. In addition, as shown in FIG. Figures 2A-2C As shown, the bounding box in each of frames 102, 112, and 122 also experiences changes in the center position, which can introduce further errors in tracking object 130. For example, between image frames 102 and 112, the system can determine that the bounding box moved a horizontal distance between x0 and x1 and a vertical distance between y0 and y1 based on the center position. Additionally, between image frames 112 and 122, the system can determine that the output bounding box moved a horizontal distance between x1 and x2 and a vertical distance between y1 and y2. Figures 2A-2C As shown, changes in the size of the bounding boxes also contribute to changes in the corresponding center positions. Therefore, the distance between the center positions of the bounding boxes (in the horizontal direction and / or vertical direction) may not correspond to the actual movement of the object 130. In some embodiments, a system that relies on changes in the center positions of the bounding boxes to track the movement of the object 130 may, for example, overestimate the speed of the movement, determine the wrong direction of the movement, and / or otherwise introduce errors when tracking the object 130. Therefore, it is critical to generate bounding boxes of the correct size / position for them in the continuous process for object tracking.

[0036] As described above, the smoothness of the output bounding box may refer to the rate of change of one or more attributes of the output bounding box over a set of consecutive frames. Here, the smoothness of the bounding box between image frames 102 and 112, between image frames 102 and 122, or between image frames 112 and 122 may be determined based on, for example, a change in the width, height, and / or center position of the output bounding box. For example, the smoothness of a bounding box (e.g., represented by bounding boxes 104 and 114) spanning image frames 102 and 112 may be determined based on, for example, the width difference between width w0 and w1, the height difference between height h0 and h1, the horizontal distance between pixel coordinates x0 and x1, the vertical distance between pixel coordinates y0 and y1, or any combination thereof. In addition, the smoothness of the output bounding box (e.g., represented by output bounding boxes 114 and 124) spanning image frames 112 and 122 can be determined based on, for example, the width difference between width w0 and w1, the height difference between height h0 and h1, the horizontal distance between pixel coordinates x0 and x1, the vertical distance between pixel coordinates y0 and y1, or any combination thereof. In order to improve the smoothness of the output bounding box, in some embodiments, certain image processing of the bounding box can be performed before outputting the bounding box for object tracking. The image processing can include, for example, generating a bounding box based on descriptors matching different sets of key points or a history of bounding boxes in previous frames. More specifically, generating the third bounding box 124 can be based on a set of descriptors taken by matching descriptors in previous image frames 102 and 112 or averaging the first bounding box 104 and the second bounding box 114 after compensating for relative image motion between the image frames.

[0037] Figure 3 is a block diagram illustrating an image processing system 300 according to some embodiments of the present disclosure, which performs the proposed method for various object tracking purposes (e.g., bounding box generation, distance estimation, automatic calibration, including but not limited to autonomous driving systems). The image processing system 300 may include: a computer system for using, training or operating one or more systems. It should be understood that these and other arrangements described herein are set forth only as examples. Other arrangements and elements (e.g., machines, interfaces, functions, orders, and groupings of functions, etc.) may be used in addition to or in place of those shown, and some elements may be omitted entirely for clarity. In addition, many of the elements described herein are functional entities that may be implemented as discrete components or distributed components, or in combination with other components, and in any appropriate combination and location. The various functions described herein as being performed by one or more entities may be performed by hardware, firmware, and / or software. For example, some functions may be performed by a processor executing instructions stored in a memory. The image processing system 300 may be a component of a vehicle, a robotic device, or another device. For example, as Fig. 9As shown, the image processing system can be a component of the vehicle 900. Aspects of the present disclosure are not limited to the image processing system 300 as a component of the vehicle 900, and other types of agents, such as buses, ships, drones, or robots, using the image processing system 300 are also contemplated. In some embodiments, the image processing system 300 can be a descriptor-free image processing system.

[0038] In some embodiments, Figure 3 An example of an image processing system 300 is depicted, which can be used to perform object detection and tracking in an image with high accuracy and in real time. As used herein, the term "real time" refers to detecting and tracking objects in an image sequence while capturing multiple image frames. The image processing system 300 may include: a key point detection module 304, a key point matching module 306, a bounding box generation module 308, a bounding box averaging module 310, a distance measurement module 312, and a sensor calibration module 314. The key point detection module 304 can receive one or more image frames 302 in grayscale or color of an image sequence provided by one or more sensors / image sources 301. For example, the proposed image processing method / system can accept image or video data from various types of sensors 301 (e.g., cameras, LIDAR sensors, etc.). In different embodiments, the sensor / image source 301 may include one or more sensors. In some embodiments, the image processing system 300 and the sensor / image source 301 may be part of the same system. In some other embodiments, the image processing system 300 and the sensor / image source 301 may be part of a separate computing system that includes one or more wireless transceivers for wirelessly communicating with the image processing system 300. The image processing system 300 may include a processing unit that may provide sufficient processing power depending on the configuration and operating requirements of the system 300, the processing unit having one or more of the following: any suitable processor, controller, or digital signal processor. For example, in some embodiments, the processing unit may include one or more high-performance processors or one or more GPUs working with a CPU, as described above. In embodiments where there is more than one processor, each processor may be configured to perform different dedicated tasks. In alternative embodiments, dedicated hardware may be used to provide some of the functions performed by the processing unit.

[0039] In some embodiments, the image frame 302 may also be referred to herein as an image, a video frame, or a picture. The image frame 302 may be part of one or more images or a sequence of images. In various embodiments, the image frame 302 may be collected directly from a sensor, or may be sensory data recorded from any device (e.g., a recorded driving image) or shared in real time by another device. For example, the image frame 302 may be raw data from one or more sensors of the same vehicle or a separate vehicle. In some embodiments, the image frame 302 may be an image captured by a camera sensor including red, green, and blue (RGB) values ​​of pixels. The image frame 302 may be a raw SIU, a processed SIU, text information, information derived from the SIU, etc. In various embodiments, the loading of the image frame 302 may be from a local disk, on an appropriate network location, from a remote storage location, etc. Acquiring the image frame 302 may include generating an image, receiving an image, participating in the processing of an image, processing only a portion of an image, and / or receiving only another portion of an image. The processing of the image frame 302 may include at least one of the following: detection, noise reduction, improvement of the signal-to-noise ratio, defining a bounding box, etc. The image frame 302 may be received from one or more sources such as one or more sensors, one or more communication units, one or more memory units, one or more image processors, etc. For example, in some embodiments, the captured input image frame 302 may be true color, which may be converted into a binary image, and noise may be removed by applying morphological operations and filtering techniques. Except for the region of interest, the remaining elements in the image may be removed by using erosion and dilation techniques.

[0040] In some embodiments, the key point detection module 304 can be configured to perform object detection to detect or determine a set of key points for an object and generate descriptors associated with the set of key points. For example, the key point detection module 304 can be configured to identify highly different key points or geometrically interesting key points for the image frame 302, which can then be used by the key point matching module 306 to obtain multiple descriptors. There are various methods and systems for performing key point detection. For example, "classical" methods such as scale-invariant feature transform (SIFT) or (accelerated robust features (SURF)) (including deep methods such as ASLFeat or D2Net) can be used to detect key points and extract descriptors. Using the position of the unique (i.e., good potential) key point or the most unique feature in the identified image frame, the content around the key point can be normalized to calculate the descriptor of the associated key point. The descriptor can be a digital vector describing the visual appearance of the key point. Each descriptor can represent, mark, identify or otherwise describe a single pixel or multiple pixels within the corresponding image frame.

[0041] Next, the keypoint matching module 306 may be configured to compare and match the identified keypoints and associated descriptors across different image frames. Figure 4 4 is a diagram illustrating an example of determining a second set of key points from a first set of key points and determining a second set of descriptors from a first set of descriptors by key point matching. Here, a second set of key points and corresponding key point descriptors 406B for generating a corresponding bounding box in a target image frame 402 can be determined by matching a first set of key points and corresponding key point descriptors 406A in a reference image frame 400. In such an embodiment, each key point 406A and 406B can correspond to a feature of the reference image frame 400 and the target image frame 402, respectively. Therefore, in some embodiments, key points can be referred to as features, such as vehicle logos and taillights. The key point correspondence between the reference image frame 400 and the target image frame 402 is depicted using a connecting line 404 between the corresponding key points and the descriptors 406A and 406B (e.g., a vehicle license plate, a headlight, a handle on a vehicle cargo door, etc.). The connecting line 404 can connect the center of the receptive field of the field for matching the key points 406A and 406B. Connecting lines 404 are provided for illustrative purposes, and the number of connecting lines 404 may correspond to the number of matching key points and descriptors. In some embodiments, for example, during real-world deployment, connecting lines 404 may not be visually generated between matched images 400 and 402. It should be understood that the number of matching key points may vary in different embodiments, and there may be more or fewer matching key points than shown here. In some embodiments of the present disclosure, the constellation of key points 406A of the reference image frame 400 may be matched with the constellation of key points 406B of the target image frame 402. That is, some aspects of the present disclosure are not to find one or more individually matched key points 406A and 406B, but to match groups of key points 406A and 406B. This group matching can eliminate ambiguity such as occlusion or dynamic changes between the respective corresponding key points 406A and 406B in different image frames 400 and 402.

[0042] In some embodiments, a database of previously stored key points and / or descriptors may be utilized to compare key points and / or other characteristics of key points or image patches surrounding key points. It should be appreciated that a neural network may be trained to match key points and / or descriptors. In some embodiments, a neural network may detect features and match detected features independent of certain geometric transformations (e.g., image translation, scaling, and rotation). In such embodiments, the system extracts multiple key points and may generate a descriptor for each key point. In such embodiments, a descriptor may be a string describing a key point that is independent of geometric transformations.

[0043] After matching the key points and the associated descriptors, there may be multiple potential matches for each key point and the associated descriptor. In order to filter out incorrect matches, a matching criterion may be applied, such as a ratio test or a nearest neighbor distance ratio test. For example, the ratio test checks the ratio between the distances of the two best matches for each key point and the associated descriptor. If the ratio is below a certain threshold, the match is considered valid. The nearest neighbor distance ratio test compares the distance of the best match with the distance of the second best match. If the distance ratio is below the threshold, the match is accepted. It should be understood that in different embodiments, the key point matching module 306 may use different key point matching algorithms. For example, another method for finding the correspondence between two key points representing the same feature in different images is by determining when the Euclidean distance of two corresponding descriptors of two key points from different images is less than a threshold or the minimum Euclidean distance in all key point descriptor spaces. Other similar algorithms (such as ratio analysis / test, cross-check, and grid-based motion statistics methods) may also be used to determine whether two key points correspond to each other, so as to determine the second key point set from the first key point set and determine the second descriptor set from the first descriptor set by key point matching. For example, a ratio check may be performed between the difference of the current descriptor to its closest descriptor in another set and the difference of the current descriptor to its second closest descriptor to determine whether the match is reliable. In some embodiments, thresholds on these algorithms may allow for the removal of many keypoint outliers while keeping most matching keypoints intact. It should be appreciated that a balance of algorithm speed, accuracy, and robustness should be considered, wherein the determined matching keypoints and associated descriptors are the basic input for generating bounding boxes for object tracking.

[0044] In some embodiments, the bounding box associated with the object may be generated by the bounding box generation module 308. In some embodiments, the bounding box may be further processed by the bounding box averaging module 310. Figures 1A-1C, the first bounding box 104 and the second bounding box 114 may be "key", "starting" or "reference" bounding boxes generated by the bounding box generation module 308. Based on these key bounding boxes, starting bounding boxes or reference bounding boxes, the third bounding box 124 may be a "final output" bounding box generated, for example, by the bounding box averaging module 310 by averaging the previous (e.g., first and second) bounding boxes 104, 114, as described above. In such an embodiment, before the output bounding box is provided for object tracking in the current frame, the properties of the output bounding box may be set based on the predicted target position or target size. By processing the output bounding box based on the history of bounding boxes in previous frames, changes in the position or size of the output bounding box may become more aligned with the historical average, which may improve the smoothness (and reduce the degree of jitter) of the output bounding box across a set of image frames. Note that detecting key points and matching them only with associated descriptors is not sufficient to generate a bounding box from a "key", "starting" or "reference" bounding box. Instead, after matching, the geometric transformation applied to the "key", "starting" or "reference" bounding box must be determined. The transformation is obtained from the corresponding image coordinates of the matched keypoints using an appropriate transformation model such as similarity transformation, affine transformation and holography. To overcome potential false matches, robust methods such as RANSAC (Random Sample Consensus) can be used.

[0045] Furthermore, distance estimation is one of the most important challenges in providing safety and security alerts to the driver. In order to provide safety and security alerts to the driver, in some embodiments, the key points and associated descriptors may also be configured for distance measurement and calibration purposes. For example, distance estimation, measurement, and evaluation may be performed for each image frame based on the key points and their associated spatial information. In some embodiments, the spatial information may be the relative positions of pairs of image pixels that are defined in three dimensions describing the position of the second pixel relative to the first pixel in terms of distance and orientation. In some embodiments, the spatial information may identify the spatial positions of corresponding image pixel points and attribute information defining one or more attributes associated with the corresponding points. For example, for autonomous driving, the spatial information may include the size of the vehicle and its components (e.g., the size or shape of signs, taillights, windows, tires, etc.) or the relative distance or position of the vehicle and its components. In the step of distance estimation, a formula describing the relationship between pixel area and distance may be developed. For example, as Figure 4As shown, when an object (e.g., a truck in front of a controlled vehicle) is closer to the vehicle, as shown in image frame 402, the same truck appears larger, and the pixel area of ​​key point 406B may be larger. Conversely, when the vehicle in front of the controlled vehicle moves away from the controlled vehicle, as shown in image frame 400, the same vehicle appears smaller, and the pixel area of ​​key point 406A may be smaller. Therefore, a mathematical model may be developed to establish a relationship between pixel area and distance, thereby estimating / assessing the distance between a vehicle and / or object and the controlled vehicle.

[0046] In some embodiments, a camera sensor can be used to capture images of surrounding vehicles or objects, and key features of the vehicle or object can be extracted and located to calculate the distance. For efficient feature extraction and location of feature positions, algorithms such as the above examples of classic and deep key point detector-descriptor algorithms can be used to find key points. In such an embodiment, the distance measurement module 312 can be configured to determine the first set of key points or descriptors of the object or the association between the corresponding second set of key points or descriptors (e.g., reducing the number of key points in the second set to reduce computational complexity while maintaining stability and accuracy). In some embodiments, the first key point set can correspond to measuring the distance of the target object, and the second key point set can be obtained from a non-volatile memory based on the geometric similarity between the corresponding key points of the first key point set and the second key point set.

[0047] In some embodiments, geometric consistency or verification may be applied to further reduce the number of matching key points after the matching process. The distance measurement module 312 may also be configured to obtain spatial information about the second key point set from a non-volatile memory, and estimate the distance of the object from the vehicle based on the first key point set and the spatial information about the second key point set. In some embodiments, each image frame may have about 5-20 matches, which will produce a relatively accurate distance estimate, including for occluded vehicles. For example, in some embodiments, the distance measurement module 312 may be configured to determine the distance between one or more objects (e.g., pedestrians, obstacles, or other vehicles) and the self-vehicle (which is a vehicle containing sensors that sense the environment around the vehicle) based on the first key point set and the spatial information about the second key point set. In some embodiments, the self-vehicle may also be an arbitrary object. In some embodiments, the corresponding key points or descriptors may be a pre-learned set of repeatable key points or descriptors for a series of objects (e.g., different vehicles) in different scenes and stored in a database accessible to the image processing system 300.

[0048] In some other embodiments, the calibration module 314 may be configured for sensor (e.g., one or more camera sensors) automatic calibration purposes, for example, automatically calibrating a vehicle camera sensor during inference (i.e., while driving the vehicle). Since camera calibration is a process of estimating camera parameters based on captured images, calibration may be performed across a sequence of image frames. For example, the calibration module 314 may include a database of real-world spatial information, such as the size of different objects or the 3D position of certain object key points. In such an embodiment, the calibration module 314 may be configured to compare the size of the same or different objects detected during inference in different image frames with the stored real-world spatial information for calibrating the driving system (e.g., camera sensor) based on the detected key points / descriptors and the associated spatial information. For example, in some embodiments, the spatial information of the highest-ranked key points may be selected from the database for calibration. In some embodiments, the automatic calibration function may be implemented by continuously optimizing the calibration state of the camera, which promotes the system to be in the best calibration state at all times, thereby improving the accuracy and fluency of the system operation.

[0049] For example, in Figure 4 In some of the illustrated embodiments, the calibration module 314 may be configured to obtain a video of a moving object (eg, Figure 4 The calibration module 314 may be configured to automatically calibrate the camera using the first image frame 400 (a first image frame 400 including a truck in the vehicle), and obtain a second image frame 402 including the truck through the camera. Then, the calibration module 314 may be configured to determine a first key point set 406A of the truck in the first image frame 400, and determine a second key point set 406B of the truck in the second image frame 402. Next, the calibration module 314 may be configured to obtain first spatial information about the first key point set 406A from a non-volatile memory, and obtain second spatial information about the second key point set 406B from the same or different non-volatile memory. Therefore, the calibration module 314 may be configured to automatically calibrate the camera using the first spatial information and the second spatial information.

[0050] It should be understood that the calibration parameters of the camera may include internal camera parameters and external camera parameters and distortion coefficients. In some embodiments, these camera parameters can be configured to remove lens distortion effects from images, measure planar objects, reconstruct three-dimensional (3-D) scenes from multiple cameras, and perform other computer vision applications. The internal parameters can be unique and can generally be constructed by a parameter matrix (fx, fy, cx, cy) and distortion coefficients (including three radial coefficients k1, k2 and k3, and two tangential coefficients p1 and p2). The external parameters (usually not unique) can be determined by the positional relationship between the camera and the spatial coordinate system, and can be constructed by a rotation matrix (e.g., a rotation matrix R3×3) and a translation matrix (e.g., Tx, Ty, and Tz). For example, in some embodiments, a calculation step can be performed to calculate the size difference between the first spatial information and the second spatial information to obtain new calibration parameters for the camera.

[0051] Now go to Figures 5 to 8 , these figures illustrate four methods 500, 600, 700, and 800 corresponding to the methods and models discussed above that can be used to track objects. Note that the order of methods 500, 600, 700, and 800 is exemplary and represents the order in which the steps of methods 500, 600, 700, and 800 will be performed. Figure 5 As shown, the method 500 of operation may start by acquiring a first image frame containing an object in block 502 and determining a first key point set of the object in the first image frame and a first descriptor set associated with the first key point set in block 506. Then, in block 508, a second key point set of the object in the second image frame and a second descriptor set associated with the second key point set may be determined based on the first key point set and the first descriptor set. Finally, in block 510, a bounding box may be generated based on the second key point set and the second descriptor set.

[0052] In some embodiments, the method may further include acquiring a second image frame containing the same object in block 504. In such an embodiment, the second key point set may have a motion pattern identified by comparing key points in the first image frame and the second image frame, and the motion pattern may be different from a motion pattern of another key point set separate from the second key point set. In various embodiments, the steps of determining the second key point set based on the first key point set and determining the second descriptor set based on the first descriptor set are based on a ratio check, a cross check, or a grid-based motion statistics method.

[0053] In some embodiments, the method of 600 illustrates an operation of generating a bounding box by averaging reference bounding boxes to increase the smoothness of bounding box tracking. For example, the method of 600 may begin by acquiring a first image frame containing an object in box 602 and acquiring a second image frame containing the same object in box 604. Then, in boxes 606 and 608, a first reference bounding box of the object from the first image frame and a second reference bounding box of the object from the second image frame may be acquired, respectively. In such an embodiment, in box 610, a bounding box of the object may be generated based on the first reference bounding box and the second reference bounding box. It should be understood that generating a bounding box of an object based on the first reference bounding box and the second reference bounding box is not limited here, and the bounding box averaging operation may include multiple reference bounding boxes (e.g., three or more bounding boxes). Again, in order to perform bounding box averaging, object motion must be compensated by matching key points and associated descriptors and then solving geometric transformations.

[0054] In some embodiments, Figure 7 As shown, the method of 700 illustrates the operation of measuring or estimating the distance of an object from a vehicle during driving. The method of 700 may be started by: in box 702, an image frame containing an object is obtained. Then, a first key point set of the object from the image frame may be identified in box 704, and a second key point set related to the first key point set may be obtained from a non-volatile memory in box 706. Next, spatial information about the second key point set may be obtained from the non-volatile memory in box 708, and the distance of the object from the vehicle may be estimated based on the first key point set and the spatial information about the second key point set in box 710. In some embodiments, the image frame is captured by a device associated with a vehicle, a stationary object, or a drone. In some embodiments, assessing the distance of the object from the vehicle includes: matching geometric parameters of some key points in the second key point set with geometric parameters of some key points in the first key point set. In some embodiments, the non-volatile memory includes descriptors associated with the first key point set and / or the second key point set. In such an embodiment, the step of assessing the distance of the object from the vehicle includes: using the descriptor. In some embodiments, the object is a second vehicle, and the method further comprises: obtaining a make, model, and / or model year of the second vehicle from the non-volatile memory. In some embodiments, the non-volatile memory is a cloud database.

[0055] In some embodiments, Figure 8As shown, the method of 800 illustrates the operation of automatic calibration, especially for automatic calibration of camera sensors. The method of 800 may be started as follows: In box 802, a first image frame containing an object in motion is acquired by a camera, and in box 804, a second image frame containing an object is acquired by a camera. Then, in boxes 806 and 808, a first key point set of the object in the first image frame and a second key point set of the object in the second image frame may be determined, respectively. Next, first spatial information about the first key point set may be acquired from a non-volatile memory in boxes 810 and 812, respectively, and second spatial information about the second key point set may be acquired from the same or different non-volatile memory. Then, in box 814, the camera may be automatically calibrated by using the first spatial information and the second spatial information. In some embodiments, the first image frame and the second image frame may be in a sequence of different image frames. In some embodiments, the first image frame may be earlier in time than the second image frame. In some embodiments, the camera may be associated with a vehicle, a stationary object, or a drone. In some embodiments, the non-volatile memory may include a descriptor associated with the first key point set or the second key point set. In such an embodiment, retrieving spatial information from the non-volatile memory may include using a descriptor. In some embodiments, the non-volatile memory may be a cloud database. In order to retrieve spatial information from the non-volatile memory, we must know which vehicle we are observing (this may be identified by the make, model, or year of the vehicle), or in other words, identify the vehicle. This can be done, for example, by calculating a global signature of the observed vehicle and matching it to a database of global signatures.

[0056] In some embodiments, the above functions / features can be implemented with hardware, software, firmware, or any combination thereof. If implemented in software, the functions can be stored as one or more instructions or codes on a non-transitory computer-readable storage medium or a non-transitory processor-readable storage medium. The blocks of the methods or algorithms disclosed herein can be implemented in a processor-executable software module, which can reside on a non-transitory computer-readable or processor-readable storage medium. A non-transitory computer-readable or processor-readable storage medium can be any storage medium that can be accessed by a computer or processor. As an example and not limitation, such a non-transitory computer-readable or processor-readable storage medium can include RAM, ROM, EEPROM, flash memory, CD-ROM or other optical disk storage, disk storage or other magnetic storage device, or any other medium that can be used to store the required program code in the form of instructions or data structures and can be accessed by a computer. As used herein, disks and optical disks include compact disks (CDs), laser disks, optical optical disks, digital versatile disks (DVDs), floppy disks, and Blu-ray disks, wherein disks generally reproduce data magnetically, and optical disks reproduce data optically with lasers. The above combinations are also included within the scope of non-transitory computer-readable and processor-readable media. Additionally, the operations of a method or algorithm may reside as one or any combination or set of codes and / or instructions on a non-transitory processor-readable storage medium and / or computer-readable storage medium, which may be incorporated into a computer program product.

[0057] Fig. 9 An exemplary hardware and software environment of an autonomous vehicle 900 in which the various techniques disclosed herein may be implemented is illustrated. For example, the vehicle 900 is shown traveling on a road 901, and the vehicle 900 may include: a powertrain 902 including a prime mover 906, which is powered by an energy source 904 and is capable of providing power to a powertrain 908; and a vehicle operating system 910, which includes a direction control 912, a powertrain control 914, and a brake control 916. The vehicle 900 may be implemented as any number of different types of vehicles, including vehicles capable of transporting people and / or cargo and capable of traveling by land, by sea, by air, underground, under the sea, and / or in space, and it should be understood that the above-described components 902-916 may vary widely based on the type of vehicle in which they are used.

[0058] For simplicity, the embodiments discussed below will focus on wheeled land vehicles, such as cars, vans, trucks, buses, motorcycles, all-terrain vehicles (ATVs), etc. In such embodiments, the energy source 904 may include, for example, a fuel system (e.g., providing gasoline, diesel, hydrogen, etc.), a battery system, a solar panel or other renewable energy source, and / or a fuel cell system. The prime mover 906 may include one or more electric motors and / or internal combustion engines (etc.). The transmission 908 may include wheels and / or tires together with a transmission and / or any other mechanical driving components suitable for converting the output of the prime mover 906 into vehicle motion and one or more brakes configured to controllably stop or slow the vehicle 900 and a direction or steering component suitable for controlling the trajectory of the vehicle 900 (e.g., a rack and pinion steering linkage that enables one or more wheels of the vehicle 900 to pivot about a substantially vertical axis to change the angle of the rotation plane of the wheel relative to the longitudinal axis of the vehicle). In some embodiments, a combination of a powertrain and energy source may be used (e.g., in the case of an electric / gas hybrid vehicle), and in other embodiments, multiple electric motors (e.g., dedicated to individual wheels or axles) may be used as prime movers 906. In the case of a hydrogen fuel cell implementation, prime movers 906 may include one or more electric motors, and energy source 904 may include a fuel cell system powered by hydrogen fuel.

[0059] Steering control 912 may include one or more actuators or sensors for controlling and receiving feedback from a steering or steering assembly to enable vehicle 900 to follow a desired trajectory. Powertrain control 914 may be configured to control the output of powertrain 902 (e.g., controlling the output power of prime mover 906, controlling the gears of a transmission in drivetrain 908, etc.), thereby controlling the speed and / or direction of vehicle 900. Braking control 916 may be configured to control one or more brakes that slow or stop vehicle 900, such as disc or drum brakes coupled to the wheels of the vehicle.

[0060] Other vehicle types (including but not limited to all-terrain vehicles or tracked vehicles, and construction equipment) may utilize different powertrains, drivetrains, energy sources, directional controls, powertrain controls, and brake controls. In addition, in some embodiments, some components may be combined, for example, where the directional control of the vehicle is primarily handled by changing the output of one or more prime movers. Therefore, the embodiments disclosed herein are not limited to the specific application of the technology described herein in autonomous vehicles, wheeled vehicles, and land vehicles.

[0061] In the illustrated embodiment, full or semi-automatic control of the vehicle 900 is implemented in a main vehicle control system 918, which may include: one or more processors 922 and one or more memories 924, each processor 922 being configured to execute program code instructions 926 stored in the memory 924. The processor 922 may include, for example, (multiple) graphics processing units (GPUs) and / or (multiple) central processing units (CPUs). The processor 922 may also include an application-specific integrated circuit (ASIC), other chipsets, logic circuits and / or data processing devices. The memory 924 may be used to load and store data and / or instructions for controlling the system 918, for example. The memory 924 may include any combination of the following: suitable volatile memory (e.g., read-only memory (ROM), dynamic random access memory (DRAM), random access memory (RAM)), non-volatile memory (e.g., flash memory, memory card, storage medium) and / or other storage devices. When the embodiment is implemented in software, the technology described herein may be implemented with modules, processes, functions, entities, etc. that perform the functions described herein. The module may be stored in the memory and executed by the processor. The memory may be implemented within the processor or external to the processor, in which case the memory may be communicatively coupled to the processor via various means as is known in the art.

[0062] The sensors 930 may include various sensors suitable for collecting information from the surrounding environment of the vehicle for controlling the operation of the vehicle 900. For example, the sensors 930 may include: one or more detection and ranging sensors (e.g., RADAR sensor 934, LIDAR sensor 936, or both), satellite navigation (SATNAV) sensor 932, for example, compatible with any of various satellite navigation systems (e.g., GPS (Global Positioning System), GLONASS (Global Navigation Satellite System), Beidou Navigation Satellite System (BDS), Galileo, compass), etc. The radio detection and ranging (RADAR) 934 and light detection and ranging (LIDAR) sensors 936 and digital cameras 938 (which may include various types of image capture devices capable of capturing still images and / or video images) may be used to sense stationary objects and moving objects in the immediate vicinity of the vehicle. The camera 938 may be a monochrome camera or a stereo camera, and may record still images and / or video images. The SATNAV sensor 932 may be used to determine the position of the vehicle on the earth using satellite signals. The sensor 930 may optionally include an inertial measurement unit (IMU) 940. IMU 940 may include multiple gyroscopes and accelerometers capable of detecting linear and rotational motion of vehicle 900 in three directions. One or more other types of sensors (e.g., wheel rotation sensor / encoder 942) may be used to monitor the rotation of one or more wheels of vehicle 900.

[0063] In various embodiments, the removable hardware pod is vehicle agnostic and can therefore be installed on a variety of non-autonomous vehicles, including: cars, buses, vans, trucks, mopeds, tractor trailers, sport vehicles, etc. Although autonomous vehicles typically contain a full sensor suite, in many embodiments, the removable hardware pod can contain a dedicated sensor suite that typically has fewer sensors than a fully autonomous vehicle sensor suite and can include: an IMU, a 3D positioning sensor, one or more cameras, a LIDAR unit, etc. Additionally or alternatively, the hardware pod can collect data from the non-autonomous vehicle itself, such as by integrating with the vehicle's CAN bus to collect various vehicle data including: vehicle speed data, braking data, steering control data, etc. In some embodiments, the removable hardware pod can include a computing device that can aggregate the data collected by the removable pod sensor suite and the vehicle data collected from the CAN bus, and upload the collected data to a computing system for further processing (e.g., uploading the data to the cloud). In many embodiments, a computing device in a removable pod can apply a timestamp to each instance of data before uploading the data for further processing. Additionally or alternatively, one or more sensors within the removable hardware pod can apply a timestamp to the data as it is collected (e.g., a lidar unit can provide its own timestamp). Similarly, a computing device within an autonomous vehicle can apply a timestamp to data collected by the autonomous vehicle's sensor suite, and can upload the timestamped autonomous vehicle data to a computer system for additional processing.

[0064] The output of the sensor 930 can be provided to a set of main control subsystems 920, including, for example, a positioning subsystem, a perception subsystem, a planning subsystem, and a control subsystem. The positioning subsystem is primarily responsible for accurately determining the position and orientation (sometimes also referred to as "pose" or "pose estimation") of the vehicle 900 in its surroundings, usually in a certain reference frame. In some embodiments, the pose is stored in the memory 924 as positioning data. In some embodiments, the surface model is generated from a high-definition map and stored in the memory 924 as surface model data. In some embodiments, the detection and ranging sensors store their sensor data in the memory 924 (for example, the radar data point cloud is stored as radar data). In some embodiments, calibration data is stored in the memory 924. The perception subsystem is primarily responsible for detecting, tracking and / or identifying objects in the environment around the vehicle 900. Machine learning models, such as those discussed above according to some embodiments, can be used to plan vehicle trajectories. The control subsystem 920 is primarily responsible for generating appropriate control signals for controlling various controls in the vehicle control system 918 so as to achieve the planned trajectory of the vehicle 900. Similarly, the machine learning model can be used to generate one or more signals to control the autonomous vehicle 900 to achieve the planned trajectory.

[0065] You should understand that Fig. 9 The collection of components for vehicle control system 918 shown in is only an example. In some embodiments, individual sensors may be omitted. Additionally or alternatively, in some embodiments, Fig. 9 The same type of multiple sensors shown can be used for redundancy and / or to cover different areas around the vehicle. In addition, in addition to the above types, there can also be other types of additional sensors to provide actual sensor data related to the operation and environment of the wheeled land vehicle. Similarly, different types of control subsystems and / or combinations of control subsystems can be used in other embodiments. In addition, although the main control subsystem 920 is illustrated as being separated from the processor 922 and the memory 924, it will be understood that in some embodiments, some or all of the functions of the main control subsystem 920 can be implemented with program code instructions 926 that reside in one or more memories 924 and are executed by one or more processors 922, and in some cases, the main control subsystem 920 can be implemented using (multiple) the same processors and / or memories. The subsystem can be implemented at least in part using various dedicated circuit logics, various processors, various field programmable gate arrays (FPGAs), various application-specific integrated circuits (ASICs), various real-time controllers, etc., as described above, multiple subsystems can utilize circuits, processors, sensors, and / or other components. In addition, the various components in the vehicle control system 918 can be networked in various ways.

[0066] For example, the vehicle 900 may include one or more network interfaces, such as network interface 954, which is suitable for communicating with one or more networks 950 (e.g., a LAN, WAN, wireless network and / or the Internet, etc.) to allow information to be transmitted with other vehicles, computers and / or electronic devices, including, for example, a central service such as a cloud service, from which the vehicle 900 receives environmental data and other data for its automatic control.

[0067] In addition, for additional storage, the vehicle 900 may also include one or more mass storage devices, such as floppy disks or other removable disk drives, hard disk drives, direct access storage devices (DASD), optical drives (e.g., CD drives, DVD drives, etc.), solid-state storage drives (SSD), network attached storage, storage area networks, and / or tape drives, etc. In addition, the vehicle 900 may include a user interface 952 to enable the vehicle 900 to receive multiple inputs from a user or operator and generate outputs for the user or operator, such as one or more displays, touch screens, voice and / or gesture interfaces, buttons and other tactile controls, etc. Otherwise, the user input may be received via another computer or electronic device, such as via an application on a mobile device or via a web interface, such as from a remote operator.

[0068] Disclosed herein are systems and methods relating to object detection and detection confidence. The disclosed methods may be suitable for autonomous driving, but may also be used for other applications, such as robotics, video analysis, weather forecasting, medical imaging, and the like. The present disclosure may be described with respect to an example autonomous driving vehicle 900. Although the present disclosure primarily provides examples using autonomous driving vehicles, the various methods described herein may be implemented using other types of devices, such as robots, camera systems, weather forecasting devices, medical imaging devices, and the like. In addition, these methods may be used to control autonomous driving vehicles, or for other purposes, such as, but not limited to, video surveillance, video or image editing, video or image search or retrieval, object tracking, weather forecasting (e.g., using radar data), and / or medical imaging (e.g., using ultrasound or magnetic resonance imaging (MRI) data).

[0069] It is understood by a person skilled in the art that each of the units, algorithms, and steps described and disclosed in the embodiments of the present disclosure is implemented using electronic hardware or a combination of software and electronic hardware for a computer. Whether the function runs in hardware or in software depends on the conditions of the application and the design requirements of the technical solution. A person skilled in the art can use different ways to implement the functions for each specific application, and such implementation should not exceed the scope of the present disclosure. It is understood by a person skilled in the art that since the working processes of the above-mentioned systems, devices, and units are basically the same, he / she can refer to the working processes of the systems, devices, and units in the above-mentioned embodiments. For ease of description and simplification, these working processes will not be described in detail.

[0070] If the software functional unit is implemented, used and sold as a product, it can be stored in a readable storage medium in a computer. Based on this understanding, the technical solution proposed in the present disclosure can be basically or partially implemented in the form of a software product. Alternatively, a part of the technical solution that is beneficial to traditional technology can be implemented in the form of a software product. The software product in the computer is stored in a storage medium, which includes multiple commands for a computing device (such as a personal computer, a server or a network device) to run all or some of the steps disclosed in the embodiments of the present disclosure. The storage medium includes a USB disk, a mobile hard disk, a ROM, a RAM, a floppy disk or other types of media capable of storing program code. Although the present disclosure has been described in conjunction with what are considered to be the most practical and preferred embodiments, it should be understood that the present disclosure is not limited to the disclosed embodiments, but is intended to cover various arrangements made without departing from the scope of the broadest interpretation of the attached claims.

[0071] However, other modifications, changes and substitutions are also possible. Therefore, the description and drawings are considered to be illustrative, not restrictive. In the claims, any reference numerals placed in brackets should not be interpreted as limiting the claims. The word "comprising" does not exclude the presence of other elements or steps other than those listed in the claims. In addition, the term "one" or "an" as used herein is defined as one or more than one. In addition, the use of introductory phrases such as "at least one" and "one or more" in the claims should not be interpreted as implying that any particular claim containing such introduced claim elements is limited to an invention containing only one such element by introducing another claim element by the indefinite article "one" or "an", even when the same claim includes the introductory phrases "one or more" or "at least one" and indefinite articles such as "one" or "an". The same applies to the use of definite articles. Unless otherwise specified, terms such as "first" and "second" are used to arbitrarily distinguish the elements described by these terms. Therefore, these terms are not necessarily intended to indicate the time or other priority of such elements. The fact that certain measures are stated in mutually different claims does not mean that the combination of these measures cannot be used advantageously. Although certain features of the present invention have been illustrated and described herein, many modifications, substitutions, changes and equivalents may now occur to those skilled in the art. It is therefore to be understood that the appended claims are intended to cover all such modifications and changes that fall within the true spirit of the invention.

[0072] It should be understood that various features of embodiments of the present disclosure described in the context of separate embodiments for clarity may also be provided in combination in a single embodiment. Conversely, various features of embodiments of the present disclosure described in the context of a single embodiment for brevity may also be provided individually or in any suitable sub-combination. Those skilled in the art will appreciate that embodiments of the present invention are not limited by what has been specifically shown and described above. Instead, the scope of the embodiments of the present disclosure is defined by the appended claims and their equivalents.

[0073] The previous description of the disclosed embodiments is provided to enable others to make or use the disclosed subject matter. Various modifications to these embodiments will be apparent, and the general principles defined herein can be applied to other embodiments without departing from the spirit or scope of the previous description. Therefore, the previous description is not intended to be limited to the embodiments shown herein, but to conform to the broadest scope consistent with the principles and novel features disclosed herein. Therefore, the claims are not intended to be limited to the aspects shown herein, but to conform to the full scope consistent with the language claims, wherein the reference to the elements in the singular form is not intended to mean "one and only one", unless explicitly stated so, but "one or more". Unless otherwise specifically stated, the term "some" refers to one or more. All structural and functional equivalents of the elements of the various aspects described in the previous description (which are known or will be known later) are explicitly introduced herein as a reference and are intended to be covered by the claims. In addition, whether such disclosure is explicitly stated in the claims, the content disclosed herein is not intended to be dedicated to the public. Unless the phrase "device for..." is used to explicitly state the elements, the claim elements should not be interpreted as device plus function. It should be understood that the specific order or hierarchy of blocks in the disclosed process is an example of an illustrative method. Based on design preferences, it is understood that the specific order or hierarchy of blocks in the process can be rearranged while remaining within the scope of the previous description. The attached method claims present elements of the various blocks in a sample order and are not meant to be limited to the specific order or hierarchy presented.

[0074] The various examples shown and described are provided only as examples to illustrate the various features of the claims. However, the features shown and described with respect to any given example are not necessarily limited to the associated examples, and may be used or combined with other examples shown and described. In addition, the claims are not intended to be limited by any one example. The above method descriptions and process flow charts are provided only as illustrative examples, and are not intended to require or imply that the blocks of the various examples must be executed in the order presented. As will be understood, the order of the blocks in the above examples can be executed in any order. Words such as "thereafter", "then", "next", etc. are not intended to limit the order of the blocks; these words are simply used to guide the reader through the description of the method. In addition, any reference to a claim element in the singular form, for example, using the article "one", "an" or "the", should not be interpreted as limiting the element to the singular. The various illustrative logic blocks, modules, circuits, and algorithm blocks described in conjunction with the examples disclosed herein may be implemented as electronic hardware, computer software, or a combination of both. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and blocks have been generally described above according to their functions. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the entire system. Those skilled in the art may implement the described functionality in different ways for each specific application, but such implementation decisions should not be interpreted as causing departure from the scope of the present invention. Hardware for implementing the various illustrative logics, logic blocks, modules, and circuits described in conjunction with the examples disclosed herein may be implemented or executed with a general-purpose processor, DSP, ASIC, FPGA, or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Alternatively, some blocks or methods may be performed by circuits specific to a given function.

[0075] Further examples are listed below.

[0076] Embodiment 1. A method for tracking an object in one or more image frames, comprising: acquiring a first image frame, the first image frame including an object; determining a first key point set of the object in the first image frame and a first descriptor set associated with the first key point set; determining a second key point set based on the first key point set and determining a second descriptor set based on the first descriptor set; generating a bounding box of the object based on the second key point set and the second descriptor set.

[0077] Embodiment 2. The method according to embodiment 1 further includes: acquiring a second image frame, the second image frame including an object, wherein a second key point set has a motion pattern identified by comparing key points in the first image frame with key points in the second image frame, and the motion pattern is different from a motion pattern of another key point set separate from the second key point set.

[0078] Embodiment 3. The method according to any one of Embodiments 1-2 further includes: acquiring a second image frame, the second image frame including an object, wherein the step of determining the second key point set based on the first key point set and the step of determining the second descriptor set based on the first descriptor set are completed by comparing the distance between the corresponding first key point sets in the first image frame and the second image frame with a threshold.

[0079] Embodiment 4. The method according to any one of embodiments 1-3, wherein the step of determining the second key point set based on the first key point set and the step of determining the second descriptor set based on the first descriptor set are based on a ratio test.

[0080] Embodiment 5. The method according to any one of embodiments 1-4, wherein the step of determining the second key point set based on the first key point set and the step of determining the second descriptor set based on the first descriptor set are based on cross-checking.

[0081] Embodiment 6. A method according to any one of embodiments 1-5, wherein the step of determining the second key point set based on the first key point set and the step of determining the second descriptor set based on the first descriptor set are based on a grid-based motion statistics method.

[0082] Embodiment 7. The method according to any one of Embodiments 1-6 further includes: acquiring a second image frame, the second image frame including an object; acquiring a first reference bounding box of the object from the first image frame; acquiring a second reference bounding box of the object from the second image frame; and generating a bounding box of the object based on the first reference bounding box and the second reference bounding box.

[0083] Embodiment 8. The method according to embodiment 6, wherein generating a bounding box of the object based on the first reference bounding box and the second reference bounding box comprises: averaging multiple reference bounding boxes.

[0084] Embodiment 9. A device for tracking an object in one or more image frames, the device comprising: interface hardware, which is configured to acquire one or more image frames; a storage unit for storing one or more image frames; a processing unit coupled to the interface hardware and the storage unit, the processing unit configured to track the object in the one or more image frames by performing the following operations: acquiring a first image frame, the first image frame including an object; determining a first key point set of the object in the first image frame and a first descriptor set associated with the first key point set; determining a second key point set based on the first key point set and determining a second descriptor set based on the first descriptor set; generating a bounding box of the object based on the second key point set and the second descriptor set.

[0085] Embodiment 10. According to the device described in Embodiment 9, the operation also includes: acquiring a second image frame, the second image frame including an object, wherein a second key point set has a motion pattern identified by comparing key points in the first image frame with key points in the second image frame, and the motion pattern is different from a motion pattern of another key point set separate from the second key point set.

[0086] Embodiment 11. According to the device described in any one of Embodiments 9-10, the operation also includes: acquiring a second image frame, which includes an object, wherein the step of determining the second key point set based on the first key point set and the step of determining the second descriptor set based on the first descriptor set are completed by comparing the distance between the corresponding first key point sets in the first image frame and the second image frame with a threshold.

[0087] Embodiment 12. The apparatus according to any one of embodiments 9-11, wherein the step of determining the second key point set based on the first key point set and the step of determining the second descriptor set based on the first descriptor set are based on a ratio test.

[0088] Embodiment 13. The apparatus of any one of embodiments 9-12, wherein the step of determining the second set of key points based on the first set of key points and the step of determining the second set of descriptors based on the first set of descriptors are based on cross-checking.

[0089] Embodiment 14. An apparatus according to any one of embodiments 9-13, wherein the step of determining the second key point set based on the first key point set and the step of determining the second descriptor set based on the first descriptor set are based on a grid-based motion statistics method.

[0090] Embodiment 15. According to the device described in any one of Embodiments 9-14, the operation also includes: acquiring a second image frame, the second image frame including an object; acquiring a first reference bounding box of the object from the first image frame; acquiring a second reference bounding box of the object from the second image frame; and generating a bounding box of the object based on the first reference bounding box and the second reference bounding box.

[0091] Embodiment 16. The apparatus of embodiment 15, wherein generating a bounding box of the object based on the first reference bounding box and the second reference bounding box comprises: averaging a plurality of reference bounding boxes.

[0092] Embodiment 17. A non-transitory computer-readable storage medium storing computer instructions, which are executable by one or more processors to perform a process for tracking an object in one or more image frames, the process comprising: acquiring a first image frame, the first image frame including an object; determining a first key point set of the object in the first image frame and a first descriptor set associated with the first key point set; determining a second key point set based on the first key point set and determining a second descriptor set based on the first descriptor set; generating a bounding box of the object based on the second key point set and the second descriptor set.

[0093] Embodiment 18. According to the non-transitory computer-readable storage medium described in Embodiment 17, it also includes: acquiring a second image frame, the second image frame including an object, wherein a second key point set has a motion pattern identified by comparing key points in the first image frame with key points in the second image frame, and the motion pattern is different from the motion pattern of another key point set separate from the second key point set.

[0094] Embodiment 19. A non-transitory computer-readable storage medium according to any one of Embodiments 17-18, further comprising: acquiring a second image frame, the second image frame comprising an object, wherein the step of determining the second key point set based on the first key point set and the step of determining the second descriptor set based on the first descriptor set are performed by comparing the distance between the corresponding first key point sets in the first image frame and the second image frame with a threshold.

[0095] Embodiment 20. According to the non-temporary computer-readable storage medium described in any one of Embodiments 17-19, it also includes: acquiring a second image frame, which includes an object; acquiring a first reference bounding box of the object from the first image frame; acquiring a second reference bounding box of the object from the second image frame; and generating a bounding box of the object based on the first reference bounding box and the second reference bounding box.

[0096] Embodiment 21. A method for estimating the distance of an object from a vehicle, comprising: acquiring an image frame including the object; identifying a first key point set of the object from the image frame; acquiring a second key point set related to the first key point set from a non-volatile memory; acquiring spatial information about the second key point set from the non-volatile memory; and evaluating the distance of the object from the vehicle based on the first key point set and the spatial information about the second key point set.

[0097] Embodiment 22. A method according to embodiment 21, wherein the image frames are captured by a device associated with a vehicle, a stationary object, or a drone.

[0098] Embodiment 23. A method according to any one of embodiments 21-22, wherein evaluating the distance of the object from the vehicle includes: matching geometric parameters of one or more key points in the second key point set with geometric parameters of one or more key points in the first key point set.

[0099] Embodiment 24. A method according to any one of Embodiments 21-23, wherein the non-volatile memory includes a descriptor associated with the first set of key points or the second set of key points.

[0100] Embodiment 25. A method according to Embodiment 24, wherein the distance of the object from the vehicle is evaluated based on the descriptor.

[0101] Embodiment 26. A method according to any one of embodiments 21-25, wherein the object is a second vehicle, and the method further comprises: obtaining the make, model or model year of the second vehicle from a non-volatile memory.

[0102] Embodiment 27. The method according to embodiments 21-26, wherein the non-volatile memory is a cloud database.

[0103] Embodiment 28. A device for estimating the distance of an object from a vehicle, the device comprising: interface hardware configured to acquire image frames; a storage unit for storing image frames; a processing unit coupled to the interface hardware and the memory unit, the processing unit configured to execute a method for estimating the distance of an object from the vehicle during driving, wherein the method comprises: acquiring an image frame including an object; identifying a first set of key points of the object from the image frame; acquiring a second set of key points related to the first set of key points from a non-volatile memory; acquiring spatial information about the second set of key points from the non-volatile memory; and evaluating the distance of the object from the vehicle based on the first set of key points and the spatial information about the second set of key points.

[0104] Embodiment 29. An apparatus according to embodiment 28, wherein the image frames are captured by a device associated with a vehicle, a stationary object, or a drone.

[0105] Embodiment 30. An apparatus according to any one of embodiments 28-29, wherein evaluating the distance of the object from the vehicle comprises: matching geometric parameters of one or more key points in the second key point set with geometric parameters of one or more key points in the first key point set.

[0106] Embodiment 31. An apparatus according to any one of Embodiments 28-30, wherein the non-volatile memory includes a descriptor associated with the first set of key points or the second set of key points.

[0107] Embodiment 32. An apparatus according to Embodiment 31, wherein the distance of the object from the vehicle is evaluated based on the descriptor.

[0108] Embodiment 33. A device according to any one of Embodiments 28-32, wherein the object is a second vehicle, and the method further comprises: obtaining the make, model or model year of the second vehicle from a non-volatile memory.

[0109] Embodiment 34. An apparatus according to any one of embodiments 28-33, wherein the non-volatile memory is a cloud database.

[0110] Embodiment 35. A non-temporary computer-readable storage medium storing computer instructions, which are executable by one or more processors to perform a method for estimating the distance of an object from a vehicle, the method comprising: acquiring an image frame including an object; identifying a first set of key points of the object from the image frame; acquiring a second set of key points related to the first set of key points from a non-volatile memory; acquiring spatial information about the second set of key points from the non-volatile memory; and evaluating the distance of the object from the vehicle based on the first set of key points and the spatial information about the second set of key points.

[0111] Embodiment 36. The non-transitory computer-readable storage medium of Embodiment 35, wherein the image frames are captured by a device associated with a vehicle, a stationary object, or a drone.

[0112] Embodiment 37. A non-transitory computer-readable storage medium according to any one of embodiments 35-36, wherein evaluating the distance of the object from the vehicle includes: matching geometric parameters of one or more key points in the second key point set with geometric parameters of one or more key points in the first key point set.

[0113] Embodiment 38. A non-transitory computer-readable storage medium according to any one of Embodiments 35-37, wherein the non-volatile memory includes a descriptor associated with the first set of key points or the second set of key points.

[0114] Embodiment 39. The non-transitory computer-readable storage medium of Embodiment 38, wherein the distance of the object from the vehicle is evaluated based on the descriptor.

[0115] Embodiment 40. A non-transitory computer-readable storage medium according to any one of Embodiments 35-39, wherein the object is a second vehicle, and the method further comprises: obtaining a make, model, or model year of the second vehicle from a non-volatile memory.

[0116] Embodiment 41. A method for calibrating a camera of a vehicle, comprising: acquiring, by the camera, a first image frame including an object; acquiring, by the camera, a second image frame including an object; determining a first set of key points of the object in the first image frame; determining a second set of key points of the object in the second image frame; acquiring first spatial information about the first set of key points from a non-volatile memory; acquiring second spatial information about the second set of key points from the same or different non-volatile memory; and automatically calibrating the camera based on the first spatial information and the second spatial information.

[0117] Embodiment 42. A method according to embodiment 41, wherein the first image frame and the second image frame are within a sequence of different image frames.

[0118] Embodiment 43. A method according to any one of Embodiments 41-42, wherein the first image frame is acquired earlier in time than the second image frame in the sequence.

[0119] Embodiment 44. A method according to any one of Embodiments 41-43, wherein the camera is associated with a vehicle, a stationary object, or a drone.

[0120] Embodiment 45. A method according to any one of Embodiments 41-44, wherein the non-volatile memory includes a descriptor associated with the first set of key points or the second set of key points.

[0121] Embodiment 46. A method according to embodiment 45, wherein the spatial information is obtained from a non-volatile memory based on the descriptor.

[0122] Embodiment 47. A method according to any one of Embodiments 41-46, wherein the non-volatile memory is a cloud database.

[0123] Embodiment 48. A device for calibrating a camera of a vehicle, the device comprising: interface hardware configured to acquire multiple image frames; a storage unit for storing the multiple image frames; a processing unit coupled to the interface hardware and the memory unit, the processing unit configured to calibrate the camera of the vehicle during driving by: acquiring a first image frame including an object by the camera; acquiring a second image frame including an object by the camera; determining a first set of key points of the object in the first image frame; determining a second set of key points of the object in the second image frame; acquiring first spatial information about the first set of key points from a non-volatile memory; acquiring second spatial information about the second set of key points from the same or different non-volatile memory; and automatically calibrating the camera based on the first spatial information and the second spatial information.

[0124] Embodiment 49. An apparatus according to embodiment 48, wherein the first image frame and the second image frame are within a sequence of different image frames.

[0125] Embodiment 50. An apparatus according to any one of Embodiments 48-49, wherein the first image frame is acquired earlier in time than the second image frame in the sequence.

[0126] Embodiment 51. An apparatus as described in any of Embodiments 48-50, wherein the camera is associated with a vehicle, a stationary object, or a drone.

[0127] Embodiment 52. An apparatus according to any one of Embodiments 48-51, wherein the non-volatile memory includes a descriptor associated with the first set of key points or the second set of key points.

[0128] Embodiment 53. The apparatus of embodiment 52, wherein the spatial information is obtained from a non-volatile memory based on a descriptor.

[0129] Embodiment 54. An apparatus according to any one of Embodiments 48-53, wherein the non-volatile memory is a cloud database.

[0130] Embodiment 55. A non-temporary computer-readable storage medium storing computer instructions, which are executable by one or more processors to perform a method for calibrating a camera of a vehicle, the method comprising: acquiring a first image frame including an object by the camera; acquiring a second image frame including an object by the camera; determining a first set of key points of the object in the first image frame; determining a second set of key points of the object in the second image frame; acquiring first spatial information about the first set of key points from a non-volatile memory; acquiring second spatial information about the second set of key points from the same or different non-volatile memory; and automatically calibrating the camera based on the first spatial information and the second spatial information.

[0131] Embodiment 56. The non-transitory computer-readable storage medium of Embodiment 55, wherein the first image frame and the second image frame are within a sequence of different image frames.

[0132] Embodiment 57. A non-transitory computer-readable storage medium according to any one of Embodiments 55-56, wherein the first image frame is acquired earlier in time than the second image frame in the sequence.

[0133] Embodiment 58. A non-transitory computer-readable storage medium as described in any one of Embodiments 55-57, wherein the camera is associated with a vehicle, a stationary object, or a drone.

[0134] Embodiment 59. A non-transitory computer-readable storage medium according to any one of Embodiments 55-58, wherein the non-volatile memory includes a descriptor associated with the first set of key points or the second set of key points.

[0135] Embodiment 60. The non-transitory computer-readable storage medium of embodiment 59, wherein the spatial information is obtained from a non-volatile memory based on the descriptor.

Claims

1. A method for estimating a distance of an object from a vehicle, comprising: acquiring an image frame including the object; identifying a first set of key points of the object from the image frame; Acquire a second key point set related to the first key point set from a non-volatile memory; Acquire spatial information about the second key point set from the non-volatile memory; as well as Based on the first set of key points and the spatial information about the second set of key points, a distance of the object from the vehicle is estimated.

2. The method according to claim 1, wherein: The image frames are captured by a device associated with a vehicle, a stationary object, or a drone.

3. The method according to claim 1, wherein: Evaluating the distance of the object from the vehicle includes matching geometric parameters of one or more key points in the second set of key points with geometric parameters of one or more key points in the first set of key points.

4. The method according to claim 1, wherein: The non-volatile memory includes descriptors associated with the first set of keypoints or the second set of keypoints.

5. The method according to claim 4, wherein: The distance of the object from the vehicle is estimated based on the descriptor.

6. The method according to claim 1, wherein: The object is a second vehicle, and the method further includes: obtaining a make, model, or model year of the second vehicle from the non-volatile memory.

7. The method according to claim 1, wherein: The non-volatile memory is a cloud database.

8. A system for estimating a distance of an object from a vehicle, the system comprising: interface hardware configured to acquire image frames; A storage unit for storing the image frame; A processing unit coupled to the interface hardware and the storage unit, the processing unit being configured to perform the method according to any one of claims 1 to 7.

9. A non-transitory computer-readable storage medium storing computer instructions executable by one or more processors to perform the method according to any one of claims 1 to 7.

Images