Methods and apparatus for scale recovery from monocular video

Abstract

Methods, apparatus, systems, and manufacturing objects for scaling from monocular video are disclosed. An exemplary non-transient computer-readable medium includes instructions that, when executed, cause a machine to at least segment an input image from a monocular video to detect objects in the camera's field of view; estimate camera parameters from the segmented input image; iteratively refine the estimated camera parameters using known object heights; calculate a scale for the video; iteratively refine the scale based on user input; and report the scaling result for visualization.

Description

Technical Field

[0001] This invention relates generally to video and / or image processing, and more specifically to a method and apparatus for restoring absolute scale from monocular video. Prior Technology

[0002] The autonomous driving industry is developing rapidly. Autonomous vehicles utilize many types of sensors to detect roads, obstacles, traffic control devices, and so on. One type of sensor is the monocular video camera (e.g., dashcam, roof-mounted camera, etc.), which is a video capture device that uses a single visual path to capture two-dimensional images (unlike binocular or stereoscopic vision devices). These monocular video cameras are becoming increasingly popular as input devices for autonomous driving systems due to their low cost and ease of operation. Summary of the Invention

[0003] and Simple Explanation of the Diagram

[0004] [Figure 1] is a block diagram of an example environment in which input video from at least one monocular camera is received in a data center, wherein scale restoration circuitry is used in conjunction with a video database for scale restoration.

[0005] [Figure 2] is a block diagram of an example implementation of the scale restoration circuit in Figure 1.

[0006] [Figures 3 to 6] are flowcharts representing exemplary machine-readable instructions taught in accordance with the present invention, which can be executed to perform the exemplary scale restoration system of Figures 1 and / or 2.

[0007] [Figure 7] is an example of an exemplary user interface, through which the scaling factor result is visualized, and / or the user can iteratively refine the scaling factor for at least one of the monocular input cameras in Figure 1.

[0008] [Figure 8A] is a two-dimensional description of the example camera position and road plane.

[0009] [Figure 8B] is a one-dimensional example of using image projection to determine the height of an object.

[0010] [Figure 9] is a description of the distortion in the top view of the parallel lane in response to camera parameters of height and pitch.

[0011] [Figure 10] is a block diagram of the example processing platform, which is structured to execute the instructions in Figure 3 to implement the scale restoration circuit 110 in Figure 1.

[0012] [Figure 11] is a block diagram of an example implementation of the processor circuit in Figure 10.

[0013] [Figure 12] is a block diagram of another example implementation of the processor circuit in Figure 10.

[0014] [Figure 13] is a block diagram of an exemplary software distribution platform used to distribute software (e.g., software corresponding to the exemplary machine-readable instructions of Figures 3, 4, 5 and / or 6) to customer devices, such as consumers (e.g., for licensing, sales and / or use), retailers (e.g., for sales, resale, licensing and / or re-licensing), and / or original equipment manufacturers (OEMs) (e.g., including products to be distributed to, for example, retailers and / or consumers who purchase directly).

[0015] The drawings are not to scale. Generally, the same reference figures will be used in all drawings and accompanying written descriptions to refer to the same or similar parts. Connection references (e.g., attachment, coupling, connection, and link) are to be interpreted broadly and may include intermediate components between many elements and relative movement between elements, unless otherwise indicated. Thus, a connection reference does not necessarily imply that two elements are directly connected to each other and have a fixed relationship.

[0016] The descriptors "first," "second," "third," etc., are used herein to identify multiple elements or components that can be mentioned separately. Unless otherwise specified or understood based on their usage context, these descriptors are not intended to be attributed to any priority, actual order, or arrangement meaning in the list, or temporal sequence, but are merely used to identify multiple elements or components separately for the purpose of understanding the disclosed example. In some examples, the descriptor "first" may be used to refer to one element in the embodiment, while the same element may be referred to by different descriptors such as "second" or "third" within the scope of the claims. In such examples, it should be understood that these descriptors are used merely for the convenience of referring to multiple elements or components. Implementation

[0017] Artificial intelligence (AI) encompasses machine learning (ML), deep learning (DL), and / or other artificial machine-driven logic that enables machines (e.g., computers, logic circuits, etc.) to use models to process input data and produce outputs based on patterns and / or associations learned by the model through a prior training process. For example, the model can be trained with data to recognize patterns and / or associations, and when processing input data, it follows these patterns and / or associations so that other inputs result in outputs consistent with the recognized patterns and / or associations.

[0018] Many different types of machine learning models and / or machine learning architectures exist. In some of the examples revealed in this paper, neural network (NN) models are used. NN models enable the interpretation of data, where patterns can be recognized. Generally, suitable machine learning models / architectures for the exemplary methods revealed in this paper would be convolutional neural networks (CNNs) and / or deep neural networks (DNNs), where interconnections are not visible outside the model. However, other types of machine learning models can be used additionally or alternatively, such as recurrent neural networks (RNNs), support vector machines (SVMs), gated recurrent units (GRUs), long short-term memory (LSTM) models, and so on.

[0019] Generally, implementing an ML / AI system involves two phases: the learning / training phase and the inference phase. In the learning / training phase, training algorithms are used to train the model based on patterns and / or associations, operating on, for example, training data. Typically, the model contains internal parameters that guide how input data is transformed into output data, such as through a sequence of nodes and connections within the model. In addition, hyperparameters are used as part of the training process to control how the training is performed (e.g., training rate, the number of layers to be used in the machine learning model, etc.). Hyperparameters are defined as training parameters determined before the training process begins.

[0020] Different types of training can be performed based on the type of ML / AI model and / or the desired output. For example, supervised training uses inputs and corresponding desired (e.g., labeled) outputs to select parameters for the ML / AI model that reduce model error (e.g., by iterating through combinations of selected parameters). As used herein, "label" refers to the desired output of the machine learning model (e.g., classification, desired output, etc.). Alternatively, unsupervised training (e.g., used in deep learning, subsets of machine learning, etc.) involves selecting parameters for the ML / AI model from inferring patterns from the inputs (e.g., without the benefit of a desired (e.g., labeled) output).

[0021] In the example presented in this paper, known object heights (e.g., pedestrian height and / or vehicle height) are used to train the ML / AI model. However, any other training algorithms can be used additionally or alternatively. In the example presented in this paper, the object height branch is trained after calculating the object heights (e.g., pedestrian height) of various objects within each frame of the input monocular video.

[0022] In some paradigms, known object widths (e.g., pedestrian widths and / or car widths) can be additionally used to train the ML / AI model. However, any other training algorithms can be used additionally or alternatively. In some paradigms, the object width branch is trained after calculating the object widths (e.g., pedestrian widths) of various objects within each frame of the input monocular video.

[0023] In the examples presented in this paper, the "object height branch" refers to the exemplary branch of the network model used to output object heights (e.g., car height, pedestrian height, etc.). Besides the object height branch, in some examples, one or more of the segmentation backbone network, the Feature Pyramid Network (FPN) for object detection, and / or the Region of Interest (ROI) network generate object detection branches for a neural network model consisting of three convolutional layers.

[0024] Training is performed using hyperparameters that control how learning is done (e.g., learning rate, the number of layers to be used in the machine learning model, etc.).

[0025] Training is performed using training data. In the example described herein, the training data is derived from the publicly available KITTI Vision Benchmark dataset. However, datasets of any type of image, video, and / or visual data can be utilized.

[0026] When shooting monocular video for purposes such as autonomous driving, the input video limits its application to traffic simulation, vehicle trajectory prediction, etc., because the camera scaling factor cannot be effectively recovered from the video.

[0027] Current methods for reconstructing camera parameters and / or scale values ​​from monocular video require knowledge of the object category (e.g., camera height, camera angle, etc.), which will prove unhelpful for applications where these parameters are unknown.

[0028] The exemplary methods and apparatus disclosed herein receive input monocular video and process the input to generate a scale factor without requiring any knowledge of object category parameters. The ability to reconstruct scale from these monocular video cameras allows for the running of simulations of vehicle trajectories from traffic cameras, etc., for use in court cases, such as insurance disputes. The examples disclosed herein utilize scale reconstruction techniques such as image segmentation, lane detection, object detection, etc., to determine camera parameters (e.g., camera height and camera pitch) from the input monocular video. As used herein, "absolute scale reconstruction" can refer to a scale factor obtained from zero and that propagates only in one direction, as opposed to relative scale reconstruction, which can involve multiple directions and any given starting point in a field. Furthermore, in the examples disclosed herein, the scale reconstruction system 100 may hereafter be referred to as "absolute scale reconstruction system" 100.

[0029] Figure 1 illustrates a paradigm scale restoration system 100 implemented in a paradigm network environment 104 according to the teachings of the present invention. The paradigm network environment 104 includes a paradigm user data center 106 and a paradigm group of monocular cameras 102A, 102B, and 102C. The paradigm data center 106 includes a paradigm video database 108, associated with a paradigm scale restoration circuit 110.

[0030] In some examples, the first, second, and / or third monocular cameras 102A, 102B, and 102C can be directly linked to the scale restoration circuit 110 to provide the input monocular video to the scale restoration circuit 110 at different times, eliminating the need for a network environment 104 and / or a video database 108 within the data center 106.

[0031] The example SLR cameras 102A, 102B, and 102C are vehicle-mounted video cameras that use a single image sensor to capture images via a single image path. Alternatively, any other type of camera or image capture device (e.g., infrared camera, fixed-mount camera, portable camera, user-carried camera, etc.) can be used to provide input video to the example scale restoration circuit 110 for scale restoration.

[0032] Although the example in Figure 1 is communicatively coupled to the components via network 104, any one or more networks of one or more types can be used. For example, network 104 can be implemented by any combination of local area networks, wide area networks, wired networks, wireless networks, etc.

[0033] Figure 2 illustrates an exemplary implementation of the scale restoration circuit 110 of Figure 1 operating within an exemplary data center and receiving input from at least one of the monocular video cameras 102A, 102B, and / or 102C via network 104.

[0034] The example data retrieval circuit 205 communicates with the video database 108 of FIG1 and / or directly with at least one of the monocular cameras 102A, 102B, and / or 102C via network 104 to extract monocular input video for scale restoration processing. In some examples, the data retrieval circuit 205 may be configured to retrieve data from the Internet or other sources not on the local area network environment 104.

[0035] The example input image segmentation circuit 210 processes the input video retrieved from the video database 108 to identify road geometry by running each individual video frame through a segmentation backbone network (e.g., Efficient Residual Factorized ConvNet (ERFNet)). The segmentation backbone network analyzes the image and outputs a composite 2D video with road geometry and surrounding objects highlighted for further processing.

[0036] The exemplary lane detection circuit 215 uses a neural network (e.g., a 3D-LaneNet network) to run the segmentation backbone network of the exemplary input image segmentation circuit 210 to identify the road geometry in the 2D output, in order to determine the layout of lanes on the road in the monocular video input. The neural network processes the segmented video frame by frame to predict the 3D layout of lanes on the road for each image. The lane detection circuit 215 estimates the camera height relative to the ground plane by estimating the road projection plane. The road image is projected into a virtual top-view model, from which the relative projection over the entire scene is recovered.

[0037] The example error prediction circuit 220 calculates the error between the prediction and the ground-truth. This error loss is calculated on the top-view model generated by the example lane detection circuit 215, with cross-entropy and offset relative to the anchor. The total camera parameter loss is estimated as the sum of the camera correction parameter loss and the lane detection loss.

[0038] Camera correction parameter loss system uses equations To calculate, where, This refers to the actual tilt angle of the camera. This refers to the estimated camera pitch. This refers to the actual height of the camera, and This refers to the estimated camera height. Lane detection loss is calculated using the equation... This is used to calculate, where {c,l} represents the lane type (e.g., center line, lane delimiter, etc.). The confidence level and / or probability of the anchor point being the center line and / or lane divider, when used in cross-entropy loss calculation, and and This refers to the i-th 3D lane, where each lane is a set of points used for estimating the x and z plane coordinates.

[0039] The example camera parameter refinement circuit 225 uses a cascaded structure to iteratively refine camera parameters (e.g., camera height and / or camera pitch) to minimize the error predicted by the error prediction circuit 220, thereby generating estimated camera height and camera pitch values.

[0040] The example object detection circuit 230 uses a neural network (e.g., Mask R-CNN neural network) to automatically detect all vehicles and pedestrians in each frame of a monocular input video. These detected objects constitute the detection branch of the neural network.

[0041] The example object height loss calculator 235 trains the detection branch of the neural network used by the example object detection circuit 230 using a Gaussian model. The object branch of the neural network is trained using a publicly available dataset (e.g., the KITTI dataset) containing known pedestrian and vehicle heights. The object height loss is then calculated by considering the difference between the estimated object height provided by the object and detection branches of the neural network and the actual object height.

[0042] In some examples, the object height loss calculator 235 can be implemented such that a dataset containing known object heights (e.g., vehicle heights and pedestrian heights) is locally stored in the video database 108 of Figure 1.

[0043] The example relative depth scaling circuit 240 obtains ground masking points from a monocular video and reprojects these points into 3D world space to estimate relative depth values. Ground masking points are determined by projecting all image pixels into 3D space and collecting the eight neighboring points of these coordinates. From these neighboring points, four local surfaces are derived by selecting four pairs of points forming a 90-degree angle. The final surface norm for each world coordinate is the average of its four neighboring surfaces. The angle between the world coordinates and the ground is then considered to determine whether a check point is on the ground.

[0044] The relative depth scaling circuit 240 determines the relative depth of the monocular video by dividing the estimated camera height defined by the camera parameter refinement circuit 225 by the current predicted height for that frame. In some examples, the relative depth scaling circuit 240 may calculate the relative scale individually for each frame or globally.

[0045] The example relative translation scaling circuit 245 takes the estimated depth value given by the example relative depth scaling circuit 240 and uses this depth to project all image points into the three-dimensional world space. The relative translation is then performed using the least squares method, expressed as an equation. To calculate, where, The predicted height of the camera, and Let be the height of the camera in the t-th frame of the input video.

[0046] The iterative scale estimation circuit 250 gathers user input from the example graphical user interface exemplified in Figure 7 and thus adjusts the scale calculation. In some examples, the graphical user interface allows the user to draw a line between any two points in a video frame (e.g., between lane lines) and provide the length of that line (in meters). The iterative scale estimation circuit 250 then calculates the distance error for all lines, taking into account the distance between two selected lines in the input video provided by the user. Distance samples with the lowest mean projection error are collected from the three-dimensional parameter space, and these samples are then averaged to determine the scale value.

[0047] In some examples, user input can be provided to the iterative scale estimation circuit 250 via the graphical user interface of Figure 7 before the automatic scale calculation of the error prediction circuit 220, camera parameter refinement circuit 225, object height loss calculator 235, relative depth scale scaling circuit 240, and / or relative translation scale scaling circuit 245 is initiated.

[0048] The example trajectory drawing circuit 255 draws the vehicle's current trajectory based on a calculated scaling factor and environmental conditions provided by the input monocular video. This trajectory is displayed as a graph on the example graphical user interface shown in Figure 7.

[0049] In some examples, the exemplary data retrieval circuit 205 of Figure 2 includes a mechanism for retrieving monocular input video for processing from a video database and / or a monocular video camera. For example, the mechanism for retrieving monocular input video for processing from a video database and / or a monocular video camera can be implemented by the data retrieval circuit 205. In some examples, the data retrieval circuit 205 can be implemented by machine-executable instructions executable by a processor circuit, such as the machine-executable instructions at least executed by block 302 of Figure 3, which can be implemented by the exemplary processor circuit 1025 of Figure 10, the exemplary processor circuit 1100 of Figure 11, and / or the exemplary field-programmable gate array (FPGA) circuit 1200 of Figure 12. In other examples, the data retrieval circuit 205 is implemented by other hardware logic circuits, hardware-implemented state machines, and / or any other combination of hardware, software, and / or firmware. For example, the data retrieval circuit 205 can be implemented by at least one or more hardware circuits (e.g., processor circuits, discrete and / or integrated analog and / or digital circuits, FPGA, application-specific integrated circuits (ASIC), comparators, operational amplifiers (op-amps), logic circuits, etc.), which are structured to perform the corresponding operations without executing software or firmware, but other structures are equally suitable.

[0050] In some examples, the exemplary input video segmentation circuit 210 of Figure 2 includes a mechanism for processing the input video using a segmentation backbone network to detect objects. For example, the mechanism for processing the input video using a segmentation backbone network to detect objects can be implemented by the input video segmentation circuit 210. In some examples, the input video segmentation circuit 210 can be implemented by machine-executable instructions executable by processor circuitry, such as the machine-executable instructions implemented by at least blocks 402 and / or 404 of Figure 4, which can be implemented by the exemplary processor circuit 1025 of Figure 10, the exemplary processor circuit 1100 of Figure 11, and / or the exemplary field-programmable gate array (FPGA) circuit 1200 of Figure 12. In other examples, the input video segmentation circuit 210 is implemented by other hardware logic circuitry, hardware-implemented state machines, and / or any other combination of hardware, software, and / or firmware. For example, the input image segmentation circuit 210 can be implemented by at least one or more hardware circuits (e.g., processor circuits, discrete and / or integrated analog and / or digital circuits, FPGA, application-specific integrated circuits (ASIC), comparators, operational amplifiers (op-amps), logic circuits, etc.), which are structured to perform corresponding operations without executing software or firmware, but other structures are equally suitable.

[0051] In some examples, the lane detection circuit 215 of Figure 2 includes a mechanism for detecting road lanes in a monocular input video using a segmented backbone network. For example, the mechanism for detecting road lanes in a monocular input video using a segmented backbone network can be implemented by the lane detection circuit 215. In some examples, the lane detection circuit 215 can be implemented by machine-executable instructions executable by processor circuitry, such as the machine-executable instructions at least executed by block 304 of Figure 3, which can be implemented by the example processor circuit 1025 of Figure 10, the example processor circuit 1100 of Figure 11, and / or the example field-programmable gate array (FPGA) circuit 1200 of Figure 12. In other examples, the lane detection circuit 215 can be implemented by other hardware logic circuitry, hardware-implemented state machines, and / or any other combination of hardware, software, and / or firmware. For example, lane detection circuit 215 can be implemented by at least one or more hardware circuits (e.g., processor circuits, discrete and / or integrated analog and / or digital circuits, FPGA, application-specific integrated circuits (ASIC), comparators, operational amplifiers, logic circuits, etc.), which are structured to perform the corresponding operations without executing software or firmware, but other structures are equally suitable.

[0052] In some examples, the exemplary error prediction circuit 220 of Figure 2 includes a mechanism for calculating the estimated camera parameter error. For example, the mechanism for calculating the estimated camera parameter error can be implemented by the error prediction circuit 220. In some examples, the error prediction circuit 220 can be implemented by machine-executable instructions executable by processor circuitry, such as the machine-executable instructions at least executed by block 506 of Figure 5, which can be implemented by the exemplary processor circuit 1025 of Figure 10, the exemplary processor circuit 1100 of Figure 11, and / or the exemplary field-programmable gate array (FPGA) circuit 1200 of Figure 12. In other examples, the error prediction circuit 220 can be implemented by other hardware logic circuitry, hardware-implemented state machines, and / or any other combination of hardware, software, and / or firmware. For example, the error prediction circuit 220 can be implemented by at least one or more hardware circuits (e.g., processor circuits, discrete and / or integrated analog and / or digital circuits, FPGA, application-specific integrated circuits (ASIC), comparators, operational amplifiers, logic circuits, etc.), which are structured to perform the corresponding operations without executing software or firmware, but other structures are equally suitable.

[0053] In some examples, the exemplary camera parameter refinement circuit 225 of Figure 2 includes a mechanism for adjusting the estimated camera parameters (e.g., camera height and camera pitch) to minimize the prediction error. For example, the mechanism for adjusting the estimated camera parameters (e.g., camera height and camera pitch) to minimize the prediction error can be implemented by the camera parameter refinement circuit 225. In some examples, the camera parameter refinement circuit 225 can be implemented by machine-executable instructions executable by processor circuitry, such as the machine-executable instructions at least executed by block 306 of Figure 3, which can be implemented by the exemplary processor circuitry 1025 of Figure 10, the exemplary processor circuitry 1100 of Figure 11, and / or the exemplary field-programmable gate array (FPGA) circuitry 1200 of Figure 12. In other examples, the camera parameter refinement circuit 225 is implemented by other hardware logic circuits, hardware-implemented state machines, and / or any other combination of hardware, software, and / or firmware. For example, the camera parameter refinement circuit 225 may be implemented by at least one or more hardware circuits (e.g., processor circuits, discrete and / or integrated analog and / or digital circuits, FPGAs, application-specific integrated circuits (ASICs), comparators, operational amplifiers (op-amps), logic circuits, etc.) that are structured to perform corresponding operations without executing software or firmware, but other structures are equally suitable.

[0054] In some examples, the exemplary object detection circuit 230 of FIG2 includes a mechanism for detecting objects (e.g., pedestrians, trees, traffic lights, etc.) within a road scene of a given segment of input video. For example, the mechanism for detecting objects (e.g., pedestrians, trees, traffic lights, etc.) within a road scene of a given segment of input video can be implemented by the object detection circuit 230. In some examples, the object detection circuit 230 can be implemented by machine-executable instructions executable by processor circuitry, such as the machine-executable instructions at least executed by block 502 of FIG5, which can be implemented by the exemplary processor circuitry 1025 of FIG10, the exemplary processor circuitry 1100 of FIG11, and / or the exemplary field-programmable gate array (FPGA) circuitry 1200 of FIG12. In other examples, the object detection circuit 230 is implemented by other hardware logic circuits, hardware-implemented state machines, and / or any other combination of hardware, software, and / or firmware. For example, the object detection circuit 230 may be implemented by at least one or more hardware circuits (e.g., processor circuits, discrete and / or integrated analog and / or digital circuits, FPGAs, application-specific integrated circuits (ASICs), comparators, operational amplifiers (op-amps), logic circuits, etc.) that are structured to perform corresponding operations without executing software or firmware, but other structures are equally suitable.

[0055] In some examples, the example object height loss calculator 235 of Figure 2 includes a mechanism for determining the calculated height difference in any given object across each frame of the video. For example, the mechanism for determining the calculated height difference in any given object across each frame of the video can be implemented by the object height loss calculation circuit 235. In some examples, the object height loss calculation circuit 235 can be implemented by machine-executable instructions executable by processor circuitry, such as the machine-executable instructions implemented by at least blocks 504, 506, and / or 508 of Figure 5, which can be implemented by the example processor circuit 1025 of Figure 10, the example processor circuit 1100 of Figure 11, and / or the example field-programmable gate array (FPGA) circuit 1200 of Figure 12. In other examples, the object height loss calculation circuit 235 is implemented by other hardware logic circuits, hardware-implemented state machines, and / or any other combination of hardware, software, and / or firmware. For example, the object height loss calculation circuit 235 may be implemented by at least one or more hardware circuits (e.g., processor circuits, discrete and / or integrated analog and / or digital circuits, FPGAs, application-specific integrated circuits (ASICs), comparators, operational amplifiers (op-amps), logic circuits, etc.) that are structured to perform corresponding operations without executing software or firmware, but other structures are equally suitable.

[0056] In some examples, the exemplary relative depth scaling circuit 240 of Figure 2 includes a mechanism for determining the relative depth value using a projection model for monocular input video. For example, the mechanism for determining the relative depth value using a projection model for monocular input video can be implemented by the relative depth scaling circuit 240. In some examples, the relative depth scaling circuit 240 can be implemented by machine-executable instructions executable by processor circuitry, such as the machine-executable instructions implemented by at least blocks 602 and / or 604 of Figure 6, which can be implemented by the exemplary processor circuitry 1025 of Figure 10, the exemplary processor circuitry 1100 of Figure 11, and / or the exemplary field-programmable gate array (FPGA) circuitry 1200 of Figure 12. In other examples, the relative depth scaling circuit 240 can be implemented by other hardware logic circuitry, hardware-implemented state machines, and / or any other combination of hardware, software, and / or firmware. For example, the relative depth scale scaling circuit 240 can be implemented by at least one or more hardware circuits (e.g., processor circuits, discrete and / or integrated analog and / or digital circuits, FPGA, application-specific integrated circuits (ASIC), comparators, operational amplifiers (op-amps), logic circuits, etc.), which are structured to perform the corresponding operations without executing software or firmware, but other structures are equally suitable.

[0057] In some examples, the exemplary relative translation scaling circuit 245 of Figure 2 includes a mechanism for determining the relative translation value using a projection model for monocular input video. For example, the mechanism for determining the relative translation value using a projection model for monocular input video can be implemented by the relative translation scaling circuit 245. In some examples, the relative translation scaling circuit 245 can be implemented by machine-executable instructions executable by processor circuitry, such as machine-executable instructions implemented by at least blocks 606 and / or 608 of Figure 6, which can be implemented by the exemplary processor circuit 1025 of Figure 10, the exemplary processor circuit 1100 of Figure 11, and / or the exemplary field-programmable gate array (FPGA) circuit 1200 of Figure 12. In other examples, the relative translation scaling circuit 245 can be implemented by other hardware logic circuitry, hardware-implemented state machines, and / or any other combination of hardware, software, and / or firmware. For example, the relative translation scaling circuit 245 can be implemented by at least one or more hardware circuits (e.g., processor circuits, discrete and / or integrated analog and / or digital circuits, FPGA, application-specific integrated circuits (ASIC), comparators, operational amplifiers (op-amps), logic circuits, etc.), which are structured to perform the corresponding operations without executing software or firmware, but other structures are equally suitable.

[0058] In some examples, the exemplary iterative scale estimation circuit 250 of Figure 2 includes a mechanism for estimating the scale of a monocular video using provided user input. For example, the mechanism for estimating the scale of a monocular video using provided user input can be implemented by the iterative scale estimation circuit 250. In some examples, the iterative scale estimation circuit 250 can be implemented by machine-executable instructions executable by processor circuitry, such as the machine-executable instructions at least executed by block 310 of Figure 3, which can be implemented by the exemplary processor circuit 1025 of Figure 10, the exemplary processor circuit 1100 of Figure 11, and / or the exemplary field-programmable gate array (FPGA) circuit 1200 of Figure 12. In other examples, the iterative scale estimation circuit 250 can be implemented by other hardware logic circuitry, hardware-implemented state machines, and / or any other combination of hardware, software, and / or firmware. For example, the iterative scaling circuit 250 can be implemented by at least one or more hardware circuits (e.g., processor circuits, discrete and / or integrated analog and / or digital circuits, FPGAs, application-specific integrated circuits (ASICs), comparators, operational amplifiers (op-amps), logic circuits, etc.), which are structured to perform the corresponding operations without executing software or firmware, but other structures are equally suitable.

[0059] In some examples, the exemplary trajectory plotting circuit 255 of Figure 2 includes a mechanism for plotting the estimated trajectory of an autonomous vehicle in a real-time given input video. For example, the mechanism for plotting the estimated trajectory of an autonomous vehicle in a real-time given input video can be implemented by the trajectory plotting circuit 255. In some examples, the trajectory plotting circuit 255 can be implemented by machine-executable instructions executable by processor circuitry, such as the machine-executable instructions at least executed by block 310 of Figure 3, which can be implemented by the exemplary processor circuit 1025 of Figure 10, the exemplary processor circuit 1100 of Figure 11, and / or the exemplary field-programmable gate array (FPGA) circuit 1200 of Figure 12. In other examples, the trajectory plotting circuit 255 is implemented by other hardware logic circuitry, hardware-implemented state machines, and / or any other combination of hardware, software, and / or firmware. For example, the trajectory plotting circuit 255 can be implemented by at least one or more hardware circuits (e.g., processor circuits, discrete and / or integrated analog and / or digital circuits, FPGA, application-specific integrated circuits (ASIC), comparators, operational amplifiers (op-amps), logic circuits, etc.), which are structured to perform the corresponding operations without executing software or firmware, but other structures are equally suitable.

[0060] Although an exemplary implementation of the scale restoration circuit 110 of Figure 1 is illustrated in Figure 2, one or more of the components, processes, and / or devices illustrated in Figure 1 can be combined, separated, reconfigured, omitted, excluded, and / or implemented in any other manner. Furthermore, the exemplary data retrieval circuit 205, input image segmentation circuit 210, lane detection circuit 215, error prediction circuit 220, camera parameter refinement circuit 225, object detection circuit 230, object height loss calculator 235, relative depth scale scaling circuit 240, relative translation scale scaling circuit 245, iterative scale estimation circuit 250, trajectory plotting circuit 255, and / or, more generally, the scale restoration circuit 110 of Figure 1 can be implemented by any combination of hardware, software, firmware, and / or hardware, software, and / or firmware. Therefore, for example, the example data retrieval circuit 205, the input image segmentation circuit 210, the lane detection circuit 215, the error prediction circuit 220, the camera parameter refinement circuit 225, the object detection circuit 230, the object height loss calculator 235, the relative depth scale scaling circuit 240, the relative translation scale scaling circuit 245, the iterative scale estimation circuit 250, the trajectory plotting circuit 255, and / or, more generally, any of the scale restoration circuits 110 of FIG1 can be implemented by processor circuits, analog circuits, digital circuits, logic circuits, programmable processors, programmable controllers, graphics processing units (GPUs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable logic devices (PLDs), and / or field-programmable logic devices (FPLDs) such as field-programmable gate arrays (FPGAs). When reading any of the device or system claims of this patent to cover purely software and / or firmware implementations, the example data retrieval circuit 205, input image segmentation circuit 210, lane detection circuit 215, error prediction circuit 220, camera parameter refinement circuit 225, object detection circuit 230, object height loss calculator 235, relative depth scale scaling circuit 240, relative translation scale scaling circuit 245, iterative scale estimation circuit 250, trajectory drawing circuit 255, and / or, more generally, at least one of the scale restoration circuit 110 of FIG1 is hereby expressly defined to include non-transitory computer-readable storage devices or storage discs, such as memory, digital video discs (DVDs), compressed discs (CDs), Blu-ray discs, etc., including software and / or firmware.Furthermore, in addition to, or in place of, the components, processes, and / or devices exemplified in Figure 1, such as data retrieval circuit 205, input image segmentation circuit 210, lane detection circuit 215, error prediction circuit 220, camera parameter refinement circuit 225, object detection circuit 230, object height loss calculator 235, relative depth scale scaling circuit 240, relative translation scale scaling circuit 245, iterative scale estimation circuit 250, trajectory plotting circuit 255, and / or, more generally, the scale restoration circuit 110 of Figure 1 may include one or more components, processes, and / or devices, and / or may include one or more of any or all of the exemplified components, processes, and / or devices. As used herein, "communication" includes variations thereof, including direct communication and / or indirect communication via one or more intermediate components, and does not require direct physical (e.g., wired) communication and / or constant communication, but rather additionally includes selective communication at periodic intervals, predetermined intervals, non-periodic intervals, and / or one-off events.

[0061] Although an exemplary implementation of the scale restoration system 100 of Figure 1 is illustrated in Figure 2, one or more of the components, processes, and / or devices illustrated in Figure 2 can be combined, separated, reconfigured, omitted, excluded, and / or implemented in any other manner. Furthermore, the exemplary data retrieval circuit 205, input image segmentation circuit 210, lane detection circuit 215, error prediction circuit 220, camera parameter refinement circuit 225, object detection circuit 230, object height loss calculator 235, relative depth scale scaling circuit 240, relative translation scale scaling circuit 245, iterative scale estimation circuit 250, and / or trajectory drawing circuit 255 can be implemented by any combination of hardware, software, firmware, and / or hardware, software, and / or firmware. Therefore, for example, the exemplary data retrieval circuit 205, the input image segmentation circuit 210, the lane detection circuit 215, the error prediction circuit 220, the camera parameter refinement circuit 225, the object detection circuit 230, the object height loss calculator 235, the relative depth scale scaling circuit 240, the relative translation scale scaling circuit 245, the iterative scale estimation circuit 250, the trajectory plotting circuit 255, and / or, more generally, any of the exemplary scale restoration system 100 can be implemented by one or more analog or digital circuits, logic circuits, programmable processors, programmable controllers, graphics processing units (GPUs), digital signal processors (DSPs), special application integrated circuits (ASICs), programmable logic devices (PLDs), and / or field-programmable logic devices (FPLDs). When reading any of the device or system claims of this patent to cover purely software and / or firmware implementations, at least one of the following is specifically defined to include non-transitory computer-readable storage devices or storage discs, such as memory, digital video discs (DVDs), compressed discs (CDs), Blu-ray discs, etc., including software and / or firmware. Furthermore, in addition to, or in place of, the elements, processes and / or devices exemplified in Figure 2, or replacing the elements, processes and / or devices exemplified in Figure 2, the exemplary scale restoration system 100 of Figure 1 may include one or more elements, processes and / or devices, and / or may include more than one of any or all of the exemplified elements, processes and / or devices.As used herein, "communication" includes variations thereof, including direct communication and / or indirect communication via one or more intermediate components, and does not require direct physical (e.g., wired) communication and / or constant communication, but rather includes selective communication at periodic intervals, predetermined intervals, non-periodic intervals, and / or one-off events. As used herein, "processor circuitry" is defined to include (i) one or more dedicated circuits structured to perform specific operations and comprising one or more semiconductor-based logic devices (e.g., electrical hardware implemented by one or more transistors), and / or (ii) one or more general-purpose semiconductor-based circuits programmed with instructions to perform specific operations and comprising one or more semiconductor-based logic devices (e.g., electrical hardware implemented by one or more transistors). Examples of processor circuitry include programmable microprocessors, field-programmable gate arrays (FPGAs) with instantiable instructions, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), XPUs, or microcontrollers and integrated circuits such as application-specific integrated circuits (ASICs). For example, an XPU can be implemented using a heterogeneous computing system that includes multiple types of processor circuitry (e.g., one or more FPGAs, one or more CPUs, one or more GPUs, one or more DSPs, etc., and / or combinations thereof) and an application programming interface (API) that can assign computational tasks to any one or more of the processor circuitry best suited to perform the computational task(s).

[0062] A flowchart representing the exemplary hardware logic, machine-readable instructions, state machine of hardware implementation, and / or combinations thereof used to implement the scale recovery circuit 110 of FIG1 is shown in FIG3. The machine-readable instructions may be one or more executable programs or portions thereof for execution by a computer processor, such as processor 1025 shown in the exemplary processor platform 1000, and discussed below in conjunction with FIG3. The program may be embodied in software stored on a non-transitory computer-readable storage medium such as a CD-ROM, floppy disk, hard disk drive, DVD, Blu-ray disc, or memory associated with processor 1025; however, the entire program and / or portions thereof may alternatively be executed by a device other than processor 1025 and / or embodied in firmware or dedicated hardware. Furthermore, although the exemplary program is illustrated with reference to the flowcharts exemplified in FIG3, many other methods for implementing the exemplary scale recovery circuit 110 may be used alternatively. For example, the execution order of these blocks can be changed, and / or parts of these blocks can be altered, removed, or combined. Alternatively, any or all of these blocks can be implemented by one or more hardware circuits (e.g., processor circuits, discrete and / or integrated analog and / or digital circuits, FPGAs, application-specific integrated circuits (ASICs), comparators, operational amplifiers (op-amps), logic circuits, etc.), which are structured to perform corresponding operations without executing software or firmware. Processor circuits can be distributed across different network locations and / or local to one or more hardware devices in a single machine (e.g., a single-core processor (e.g., a single-core central processing unit (CPU)), a multi-core processor (e.g., a multi-core CPU), etc.), and can be multiple processors distributed across multiple servers in a server rack, multiple processors distributed across one or more server racks, CPUs and / or FPGAs located in the same package (e.g., the same integrated circuit (IC) package) or in two or more separate packages.

[0063] The machine-readable instructions described herein can be stored in one or more of the following formats: compressed, encrypted, segmented, compiled, executable, packaged, etc. As described herein, machine-readable instructions can be stored as data that can be used to create, manufacture, and / or produce machine-executable instructions (e.g., instruction portions, codes, code representations, etc.). For example, machine-readable instructions can be segmented and stored on one or more storage devices and / or computing devices (e.g., servers). Machine-readable instructions may require one or more installations, modifications, adaptations, updates, combinations, additions, configurations, decryptions, decompressions, unpackings, allocations, reallocations, compilations, etc., so that they can be read, interpreted, and / or executed by computing devices and / or other machines. For example, machine-readable instructions can be stored in multiple locations, which can be individually compressed, encrypted, and stored on separate computing devices, wherein these locations, when decrypted, decompressed, and combined, constitute a set of executable instructions that perform programs as described herein.

[0064] In another example, machine-readable instructions may be stored in a state readable by a computer, but require the addition of libraries (e.g., dynamic link libraries (DLLs)), software development kits (SDKs), application programming interfaces (APIs), etc., to execute these instructions on a specific computing device or other device. In yet another example, the machine-readable instructions and / or corresponding programs may need to be configured (e.g., stored settings, input data, recorded network addresses, etc.) before they can be fully or partially executed. Therefore, the disclosed machine-readable instructions and / or corresponding programs are intended to include such machine-readable instructions and / or programs, regardless of their specific format or state when they are stored, stored, or executed.

[0065] The machine-readable instructions described herein can be represented by any past, present, or future instruction language, scripting language, programming language, etc. For example, machine-readable instructions can be represented using any of the following languages: C, C++, Java, C#, Perl, Python, JavaScript, Hypertext Markup Language (HTML), Structured Query Language (SQL), Swift, etc.

[0066] As described above, the exemplary processes of Figures 3 through 6 can be implemented using executable instructions (e.g., computer and / or machine-readable instructions) stored on a non-transitory computer and / or machine-readable medium, such as a hard disk drive, flash memory, read-only memory, optical disc, digital video disc, cache memory, random access memory, and / or any other storage device or storage disk, wherein the storage time of information is unlimited (e.g., extended period, permanent, transient, temporary buffer, and / or caching of information). As used herein, the term non-transitory computer-readable medium is explicitly defined to include any type of computer-readable storage device and / or storage disk, as well as excluding the propagation of signals and the transmission medium.

[0067] "Including" and "comprising" (and all their forms and tenses) are used herein as open-ended terms. Therefore, whenever a claim uses either form of "including" or "comprising" (e.g., includes, includes, comprising, having, etc.) in its preamble or within any claim statement, it should be understood that additional elements, items, etc., may appear without falling outside the scope of the corresponding claim or statement. As used herein, when the word "at least" is used, for example, as a transition term in the preamble of a claim, it is open-ended, in the same way as the terms "including" and "comprising" are open-ended terms. When used, for example in the form of A, B, and / or C, the term "and / or" refers to any combination or subset of A, B, and C, such as (1) A alone, (2) B alone, (3) C alone, (4) A and B, (5) A and C, (6) B and C, and (7) A and B and C. As used herein, in the context of describing structures, components, items, objects, and / or things, the term "at least one of A and B" is intended to refer to an implementation including (1) at least one A, (2) at least one B, and (3) at least one A and at least one B. Similarly, as used herein, in the context of describing structures, components, items, objects, and / or things, the term "at least one of A or B" is intended to refer to an implementation including (1) at least one A, (2) at least one B, and (3) at least one A and at least one B. As used herein, in the context of describing the performance or execution of a process, instruction, action, activity, and / or step, the term "at least one of A and B" is intended to refer to an execution comprising (1) at least one A, (2) at least one B, and (3) any one of at least one A and at least one B. Similarly, as used herein, in the context of describing the performance or execution of a process, instruction, action, activity, and / or step, the term "at least one of A and B" is intended to refer to an execution comprising (1) at least one A, (2) at least one B, and (3) any one of at least one A and at least one B.

[0068] As used herein, singular references (e.g., "a(a)", "an(an)", "first(first)", "second(second)", etc.) do not exclude plurals. The entity referred to as "a(a)" or "an(an)", as used herein, refers to one or more of that entity. The terms "a(a)" (or "an(an)"), "one or more", and "at least one" are interchangeable herein. Furthermore, although individually listed, multiple mechanisms, elements, or method actions can be performed by, for example, a single unit or processor. Moreover, although individual features may be included in different paradigms or claims, combinations of features that can be combined and whose inclusions in different paradigms or claims do not imply that such combinations are not feasible and / or advantageous.

[0069] Figure 3 is a flowchart of an example machine-readable instruction 300, which can be executed by a processor to perform the example scale restoration circuit 110 of Figure 1 and / or Figure 2 to restore the scale of the input monocular video.

[0070] As illustrated in Figure 3, in block 302, the scale restoration circuit 110 receives monocular video input from the video database 108 of Figure 1. At least one of the monocular cameras 102A, 102B, and / or 102C of Figure 1 provides these input videos via network 104 for storage and retrieval in the video database 108.

[0071] In block 304, camera parameters (e.g., camera height and / or camera pitch) are estimated. An example process for estimating camera parameters is illustrated in Figure 4.

[0072] In block 306, camera parameters (e.g., camera height and / or camera pitch) are iteratively refined. After training the object height branch using known object heights (e.g., pedestrians, cars, etc.) and calculating and minimizing the object consistency loss, the camera parameters estimated from the segmented video are adjusted. An example process for iteratively refining camera parameters is illustrated in Figure 5.

[0073] In block 308, the scale used for relative depth is calculated. For each frame of the input monocular video, the camera depth value is estimated as a moving relative scale value. The ground point system is obtained from the input video and reprojected into the 3D world to determine the depth. An example process for calculating the scale used for relative depth is illustrated in Figure 6.

[0074] In block 310, the scale calculated in block 308 is iteratively refined using provided user input, which is received via the example graphical user interface of Figure 7. Using the graphical user interface, the user draws a line between any two points in a given frame of the input video (e.g., a line between lane markings) and provides a distance measurement. Based on this measurement, the scale factor is adjusted across all frames of the video during the iteration process.

[0075] In block 312, the scaling results are reported for visualization via the graphical user interface shown in Figure 7.

[0076] Figure 4 is a flowchart of machine-readable instructions that can be executed to perform block 304 of Figure 3 to estimate camera parameters (e.g., camera height, camera pitch) from the monocular input video.

[0077] In block 402, each frame of the monocular input video is fed to a segmentation backbone network (e.g., ERFNet), where the neural network analyzes the input image to generate a two-dimensional segmentation of the camera's surroundings.

[0078] In block 404, the 2D segmentation results from block 402 are used to calculate estimated camera parameters (e.g., camera height and camera pitch). The neural network processes the segmented video frame by frame to predict the 3D layout of lanes on the road for each image. The camera height relative to the ground plane is determined by estimating the road projection plane. The road image is projected into a virtual top-view model, and the estimated camera parameters for that video are calculated.

[0079] In block 406, camera parameter loss is calculated by generating a virtual top-view visualization of the surrounding environment using camera parameters (e.g., camera pitch, camera height) estimated by the process in block 404. The error between the prediction and the ground reality is determined, along with the lane detection loss calculated in the top-view model, having cross-entropy and offset relative to the anchor point.

[0080] Figure 5 is a flowchart of machine-readable instructions that can be executed to perform block 306 of Figure 3 to iteratively refine the estimated camera parameters.

[0081] In block 502, the Mask-RCNN deep neural network is used to automatically detect all cars and pedestrians in each frame of a monocular input video. The objects detected in these video scenes constitute the detection branch of the neural network model.

[0082] In block 504, the object height branch is trained using known pedestrian and vehicle heights, as provided by the publicly available KITTI dataset. In some examples, a Gaussian training model is used to train the object height branch.

[0083] In block 506, the reprojection error for each camera object is defined. This reprojection error is calculated by projecting the i-th object onto the video frame using the detected two-dimensional bounding box, and using the estimated camera pitch and camera height values.

[0084] In block 508, the object height consistency loss is calculated. Using the principle that the height of the same vehicle should be the same across all frames of the video, the object height consistency loss is determined by indicating the differences in the estimated object height across the frames.

[0085] Figure 6 is a flowchart of machine-readable instructions that can be executed to perform block 308 of Figure 3 to calculate the scale for relative depth.

[0086] In block 602, obtain the ground mask points from the image segmentation mask.

[0087] In block 604, for each image pixel, the obtained ground mask points are reprojected into the three-dimensional world space.

[0088] In block 606, for each obtained ground masking point, 8 adjacent points are collected.

[0089] In block 608, four local surfaces are derived from these adjacent points by selecting four pairs of points forming a 90-degree angle. The final surface norm for each world coordinate is the average of its four adjacent surfaces. Then, the angle between the world coordinate and the ground is considered to determine whether the check point is on the ground. For each point that is to be considered as a ground occlusion point, the distance between that point and all other image pixels is calculated to derive the camera height and relative depth.

[0090] Figure 7 illustrates an example graphical user interface 700, in which a user can draw a line between any two points in a frame of a monocular input video and provide the distance (in meters) for scale calculation and / or scale refinement. In this graphical user interface, the user can use video controls 710 to skip each individual frame of the input video to select a frame. After selecting a video frame, the user can then select any two points in the video (e.g., two adjacent lane lines) and connect them to draw a line 705. The user must then enter the distance (in meters) between the two selected points and select the auto-correct button 715 to adjust the scale accordingly.

[0091] In some examples, the graphical user interface 700 may also include a trajectory graph 720 (if real-world values ​​have been provided), in which both the autonomous vehicle's actual trajectory and predicted trajectory are drawn in a top-view model for visualization. This trajectory graph 720 will be populated in real-time as the user plays the video.

[0092] Figure 8A illustrates an example road structure in a three-dimensional plane, where a monocular video camera 805, having a relative conversion (TC2R) value 815, is mounted on top of an autonomous vehicle on road 810. In some examples, it is assumed that the camera system is fixed at a roll of 0 degrees relative to a flat ground plane.

[0093] Figure 8B illustrates an example road structure in a two-dimensional plane, with a monocular video camera 805 mounted on top of an autonomous vehicle. The relative transformation (TC2R) value is represented by a camera height (HCAM) value 815 and a camera pitch (θ) angle 820. In some examples, it is assumed that the camera is fixed at 0 degrees of roll relative to a flat ground plane. Objects in the scene (e.g., pedestrians 825) are displayed in front of the monocular video camera, and a reprojected model 830 of this object 825 is displayed above it. In this description, a three-dimensional view of the pedestrian object 825 is shown, along with a display of the calculated object height (as shown by the given projection lines Vt and Vb).

[0094] Figure 9 illustrates the effect of camera parameters (e.g., camera pitch and camera height) on a top-view lane description. Cell 905 describes the monocular input video provided to the scale restoration circuit 110 of Figure 1. Cells 910, 915, and 920 show the original image projections onto the top-view lane model with different camera parameter values. In cell 910, a camera pitch value of 2 degrees is less than the ground reality value; therefore, the lanes in the top-view model appear distorted and non-parallel at the boundaries. In cell 920, the camera pitch value has exceeded the ground reality parameter value, and therefore, the lanes are again distorted and non-parallel. Cell 915 illustrates how the correct camera parameter values ​​(e.g., ground reality) have produced the original top-view image projection, where the lanes are perfectly parallel and there is no image distortion at the boundaries.

[0095] Figure 10 is a block diagram of an example processor platform 1000 that is structured to execute the instructions of Figures 3 to 6 to implement the scale restoration circuit 110 of Figure 1. The processor platform 1000 can be, for example, a server, personal computer, workstation, self-learning machine (e.g., neural network), mobile device (e.g., mobile phone, smartphone, tablet computer such as iPad™), personal digital assistant (PDA), internet tool, DVD player, CD player, digital video recorder, Blu-ray player, game console, personal video recorder, video box, headset (e.g., augmented reality (AR) headset, virtual reality (VR) headset, etc.) or other wearable device, or any other type of computing device.

[0096] The exemplified processor platform 1000 includes processor circuitry 1025. The exemplified processor circuitry 1025 is hardware. For example, processor circuitry 1025 can be implemented by one or more integrated circuits, logic circuits, FPGAs, microprocessors, CPUs, GPUs, DSPs, and / or microcontrollers from any desired family or manufacturer. Processor circuitry can be implemented by one or more semiconductor-based (e.g., silicon-based) devices. In this example, processor circuitry 1025 implements an example data retrieval circuit 205, an example input image segmentation circuit 210, an example lane detection circuit 215, an example error prediction circuit 220, an example camera parameter refinement circuit 225, an example object detection circuit 230, an example object height loss calculator 235, an example relative depth scaling circuit 240, an example relative translation scaling circuit 245, an example iterative scale estimation circuit 250, and an example trajectory plotting circuit 255.

[0097] The illustrated processor circuit 1025 includes local memory 1026 (e.g., cache, register, etc.). The illustrated processor circuit 1025 communicates via bus 1030 with main memory, which includes volatile memory 1015 and non-volatile memory 1020. Volatile memory 1015 can be implemented using synchronous dynamic random access memory (SDRAM), dynamic random access memory (DRAM), RAMBUS® dynamic random access memory (RDRAM®), and / or any other type of random access memory device. Non-volatile memory 1020 can be implemented using flash memory and / or any other desirable type of memory device. Access to main memory 1015 and 1020 is controlled by a memory controller.

[0098] The processor platform 1000 in the example also includes interface circuitry 1020. Interface circuitry 1020 can be implemented by hardware according to any type of interface standard, such as Ethernet interface, Universal Serial Bus (USB) interface, Bluetooth® interface, Near Field Communication (NFC) interface, PCI interface, and / or PCIe interface.

[0099] In the illustrated example, one or more input devices 1022 are connected to interface circuitry 1020. Input devices 1022 allow users to enter data and / or commands into processor circuitry 1012. Input devices can be, for example, audio sensors, microphones, cameras (still or video), keyboards, buttons, mice, touchscreens, trackpads, trackballs, isopoints, and / or voice recognition systems.

[0100] One or more output devices 1050 are also connected to the exemplified interface circuit 1045. The output devices 1024 can use display devices (e.g., light-emitting diodes (LEDs), organic light-emitting diodes (OLEDs), liquid crystal displays (LCDs), cathode ray tube (CRT) displays, in-plane switching (IPS) displays, touch screens, etc.), haptic output devices, printers, and / or speakers. The exemplified interface circuit 1045 therefore typically includes a graphics driver card, a graphics driver chip, and / or a graphics driver processor.

[0101] The exemplified interface circuit 1045 also includes communication devices such as transmitters, receivers, transceivers, modems, residential gateways, wireless access points, and / or network interfaces to facilitate data exchange with external machines (e.g., any type of computing device) via network 1010. Communication can be achieved via, for example, Ethernet connections, digital subscriber line (DSL) connections, telephone line connections, coaxial cable systems, satellite systems, line-of-sight wireless systems, cellular telephone systems, optical connections, and so on.

[0102] The processor platform 1000 of the example also includes one or more mass storage devices 1035 for storing software and / or data. Examples of such mass storage devices 1035 include magnetic storage devices, floppy disk drives, HDDs, CDs, Blu-ray drives, redundant array of disks (RAID) systems, solid-state storage devices such as flash memory devices, and DVD drives.

[0103] Machine-executable instructions 1005, which can be executed by the machine-readable instructions of Figures 3 to 6, can be stored in mass storage device 1035, in volatile memory 1015, in non-volatile memory 1020 and / or on a removable non-transitory computer-readable storage medium such as a CD or DVD.

[0104] Figure 11 is a block diagram of an exemplary implementation of the processor circuit 1025 of Figure 10. In this example, the processor circuit 1025 of Figure 10 is implemented by a microprocessor 1100. For example, the microprocessor 1100 can implement multi-core hardware circuitry, such as a CPU, DSP, GPU, XPU, etc. Although it can contain any number of exemplary cores 1102 (e.g., one core), this exemplary microprocessor 1100 is a multi-core semiconductor device containing N cores. The cores 1102 of the microprocessor 1100 can operate individually or collaboratively to execute machine-readable instructions. For example, machine code corresponding to firmware, embedded software, or software programs can be executed by one of the cores 1102 or by multiple cores 1102 at the same or different times. In some examples, the machine code corresponding to firmware, embedded software, or software programs is split into threads and executed in parallel by two or more of these cores 1102. The software program may correspond to some or all of the machine-readable instructions and / or operations represented by the flowcharts of Figures 3 through 6.

[0105] Core 1102 can communicate via an exemplary first bus 1104. In some examples, the first bus 1104 may implement a communication bus to enable communication with one or more of the cores 1102. For example, the first bus 1104 may implement at least one of an inter-integrated circuit (I2C) bus, a serial-to-parallel interface (SPI) bus, a PCI bus, or a PCIe bus. Additionally or alternatively, the first bus 1104 may implement any other type of computing or electrical bus. Core 1102 can obtain data, instructions, and / or signals from one or more external devices via an exemplary interface circuit 1106. Core 1102 can output data, instructions, and / or signals to one or more external devices via the interface circuit 1106. While the core 1102 of this paradigm includes paradigm local memory 1120 (e.g., a Level 1 (L1) cache that can be split into L1 data cache and L1 instruction cache), the microprocessor 1100 also includes paradigm shared memory 1110 that can be shared by cores (e.g., a Level 2 (L2) cache) for high-speed access to data and / or instructions. Data and / or instructions can be transferred (e.g., shared) by writing to and / or reading from shared memory 1110. The local memory 1120 and shared memory 1110 of each of these cores 1102 may be part of a storage device hierarchy that includes multiple layers of cache and main memory (e.g., main memory 1015, 1020 of FIG. 10). Typically, higher-level memories in this hierarchy exhibit lower access times and have smaller storage capacities than lower-level memories. Changes in the various layers of the cache memory hierarchy can be managed (e.g., reconciliation) through cache consistency policies.

[0106] Each core 1102 can be referred to as a CPU, DSP, GPU, etc., or any other type of hardware circuit. Each core 1102 includes a control unit circuit 1114, an arithmetic and logic (AL) circuit (sometimes referred to as an ALU) 1116, multiple registers 1118, an L1 cache 1120, and an exemplary second bus 1122. Other structures may be present. For example, each core 1102 may include vector unit circuitry, single instruction multiple data (SIMD) unit circuitry, load / store unit (LSU) circuitry, branch / jump unit circuitry, floating point unit (FPU) circuitry, etc. The control unit circuitry 1114 is a semiconductor-based circuit structured within the corresponding core 1102 to control (e.g., coordinate) data movement. The AL circuitry 1116 is a semiconductor-based circuit structured within the corresponding core 1102 to perform one or more mathematical and / or logical operations on the data. Some exemplary AL circuitry 1116 performs integer-based operations. In other examples, AL circuit 1116 also performs floating-point operations. In yet another example, AL circuit 1116 may include a first AL circuit performing integer-based operations and a second AL circuit performing floating-point operations. In some examples, AL circuit 1116 may be referred to as an Arithmetic Logic Unit (ALU). Register 1118 is a semiconductor-based structure used to store data and / or instructions, such as the results of one or more operations performed by the AL circuit 1116 corresponding to core 1102. For example, register 1118 may include vector registers, SIMD registers, general-purpose registers, flag registers, segment registers, machine-specific registers, instruction pointer registers, control registers, debugging registers, memory management registers, machine check registers, etc. Registers 1118 may be arranged in banks, as shown in Figure 11. Alternatively, register 1118 may be organized into any other configuration, format, or structure that includes allocations throughout core 1102 to reduce access time. Second bus 1122 may implement at least one of an I2C bus, an SPI bus, a PCI bus, or a PCIe bus.

[0107] Each core 1102 and / or, more generally, the microprocessor 1100 may include additional and / or alternative structures beyond those shown and described above. For example, there may be one or more clock circuits, one or more power supplies, one or more power gates, one or more cache home agents (CHA) , one or more converged / common mesh stops (CMS) , one or more shifters (e.g., barrel shifters) and / or other circuitry. The microprocessor 1100 is a semiconductor device fabricated to include a plurality of transistors interconnected to implement the structures described above in one or more integrated circuits (ICs) contained in one or more packaged components. The processor circuitry may include one or more accelerators and / or cooperate with one or more accelerators. In some examples, the accelerators are implemented by logic circuitry to perform certain tasks faster and / or more efficiently than those that can be accomplished by a general-purpose processor. Examples of accelerators include ASICs and FPGAs, such as those discussed in this article. GPUs or other programmable devices can also be accelerators. The accelerator can have the processor circuitry onboard, either in the same die package as the processor circuitry or in one or more separate packages.

[0108] Figure 12 is a block diagram of another exemplary implementation of the processor circuit 412 of Figure 10. In this example, the processor circuit 1025 is implemented using an FPGA circuit 1200. The FPGA circuit 1200 can be used, for example, to perform operations that would otherwise be performed by the microprocessor 1100 of Figure 11, which executes the corresponding machine-readable instructions. However, once configured, the FPGA circuit 1200 instantiates the machine-readable instructions in hardware, and thus can often perform these operations much faster than they could be performed by a general-purpose microprocessor executing the corresponding software.

[0109] More specifically, in contrast to the microprocessor 1100 of Figure 11 described above (which is a general-purpose device that can be programmed to execute some or all of the machine-readable instructions represented by the flowcharts of Figures 3 to 6, but whose interconnections and logic circuitry are fixed once fabricated), the FPGA circuit 1200 of the example of Figure 12 includes interconnections and logic circuitry that can be configured and / or interconnected in different ways after fabrication to instantiate, for example, some or all of the machine-readable instructions represented by the flowcharts of Figures 3 to 6. In particular, the FPGA 1200 can be considered as an array of gates, interconnections, and switches. Switches can be programmed to change how the gates are interconnected by these interconnections to effectively constitute one or more dedicated logic circuits (unless and until the FPGA 1200 is reprogrammed). The configured logic circuitry causes the gates to cooperate in different ways to perform operations on data received from the input circuitry. Those operations can correspond to some or all of the software represented by the flowcharts of Figures 3 to 6. In this way, the FPGA circuit 1200 can be structured to efficiently instantiate some or all of the machine-readable instructions represented by the flowcharts of Figures 3 to 6 as dedicated logic circuits to perform operations corresponding to those software instructions in a dedicated manner similar to that of an ASIC. Therefore, the FPGA circuit 1200 can perform those same operations much faster than a general-purpose microprocessor can perform some or all of the operations corresponding to the machine-readable instructions of Figures 3 to 6.

[0110] In the example of Figure 12, the FPGA circuit 1200 is structured to be programmed (and / or reprogrammed one or more times) by an end user using a hardware description language (HDL) such as Verilog. The FPGA circuit 1200 of Figure 12 includes exemplary input / output (I / O) circuitry 1202 to receive and / or output data from / to exemplary configuration circuitry 1204 and / or external hardware (e.g., external hardware circuitry) 1206. For example, configuration circuitry 1204 may implement interface circuitry that can obtain machine-readable instructions to configure FPGA circuitry 1200, or portions thereof. In some such examples, configuration circuitry 1204 may obtain machine-readable instructions from a user, a machine (e.g., hardware circuitry (e.g., programming or dedicated circuitry)) that can implement artificial intelligence / machine training (AI / ML) models to generate instructions, etc. In some examples, external hardware 1206 may implement the microprocessor 1100 of Figure 11. The FPGA circuit 1200 also includes an array of exemplary logic gate circuits 1208, a plurality of exemplary configuration interconnects 1210, and exemplary storage circuits 1212. The logic gate circuits 1208 and interconnects 1210 are configurable to instantiate one or more operations, which may correspond to at least a portion of the machine-readable instructions of Figures 3 to 6 and / or other desired operations. The logic gate circuits 1208 shown in Figure 12 are fabricated in groups or blocks. Each block contains semiconductor-based electrical structures that can be configured as logic circuits. In some examples, these electrical structures include logic gates (e.g., AND gates, OR gates, NOR gates, etc.) that provide basic building blocks for logic circuits. Electrically controlled switches (e.g., transistors) are present within each logic gate circuit 1208 such that the configuration of the electrical structures and / or logic gates can form a circuit to perform the desired operations. The logic gate circuit 1208 may include other electrical structures, such as lookup tables (LUTs), temporary registers (e.g., flip-flops or latches), multiplexers, etc.

[0111] The interconnect 1210 in the example example is a conductive path, trace, via, etc., which may include an electronically controlled switch (e.g., a transistor) whose state can be changed by programming (e.g., using an HDL instruction language) to enable or disable one or more connections between logic gate circuits 1208 to program the desired logic circuit.

[0112] The storage circuit 1212 in the illustrated example is structured to store the results of one or more operations performed by the corresponding logic gates. The storage circuit 1212 can be implemented using temporary registers, etc. In the illustrated example, the storage circuit 1212 is distributed within the logic gate circuit 1208 to facilitate access and increase execution speed.

[0113] The example FPGA circuit 1200 in Figure 12 also includes example dedicated operating circuitry 1214. In this example, dedicated operating circuitry 1214 includes special-purpose circuitry 1216, which can be invoked to perform common functions to avoid the need to program those functions in the field. Examples of such special-purpose circuitry 1216 include memory (e.g., DRAM) controller circuitry, PCIe controller circuitry, clock circuitry, transceiver circuitry, memory, and multiplexer-accumulator circuitry. Other types of special-purpose circuitry may be present. In some examples, FPGA circuitry 1200 may also include example general-purpose programmable circuitry 1218, such as example CPU 1220 and / or example DSP 1222. Other general-purpose programmable circuitry 1218 may appear additionally or alternatively, such as GPUs, XPUs, etc., which can be programmed to perform other operations.

[0114] While Figures 11 and 12 illustrate two exemplary implementations of the processor circuit 1025 of Figure 10, many other methods are considered. For example, as mentioned above, modern FPGA circuits can include one or more on-board CPUs, such as the example CPU 1220 of Figure 12. Therefore, the processor circuit 1025 of Figure 10 can be additionally implemented by combining the example microprocessor 1100 of Figure 11 and the example FPGA circuit 1200 of Figure 12. In some such hybrid examples, the first part of the machine-readable instructions represented by the flowcharts of Figures 3 through 6 can be executed by one or more of the core 1102 of Figure 11, and the second part of the machine-readable instructions represented by the flowcharts of Figures 3 through 6 can be executed by the FPGA circuit 1200 of Figure 12.

[0115] In some examples, the processor circuitry 1025 of FIG10 can be housed in one or more package components. For example, the processor circuitry 1100 and / or the FPGA circuitry 1100 of FIG11 can be housed in one or more package components. In some examples, the XPU can be implemented by the processor circuitry 1025 of FIG10, which can be housed in one or more package components. For example, the XPU may include a CPU in one package component, a DSP in another package component, a GPU in yet another package component, and an FPGA in yet another package component.

[0116] Exemplary methods, apparatuses, systems, and manufacturing objects for restoring scale from monocular input video are disclosed herein. Further examples and combinations thereof include the following:

[0117] Example 1 includes an apparatus comprising a non-transitory computer-readable medium, instructions at the apparatus, and logic circuitry for executing the instructions, for at least: segmenting an input image from a monocular video to detect objects in the camera's field of view; estimating camera parameters from the segmented input image; iteratively refining the estimated camera parameters using object height; calculating a scale for the video; iteratively refining the scale based on user input; and reporting the scale-up results for visualization.

[0118] Example 2 includes the non-transient computer-readable media of Example 1, wherein the monocular video camera provides direct input to the scale restoration circuitry.

[0119] Example 3 includes the non-transitory computer-readable media of Example 1, wherein, when such instructions are executed, the input image segmentation system is further triggered to use a segmentation backbone network for this purpose.

[0120] Example 4 includes the non-transient computer-readable media of Example 1, wherein, when such instructions are executed, the video scale is further triggered to be calculated using the first and second camera parameters.

[0121] Example 5 includes a non-transitory computer-readable medium of either Example 1 or 4, wherein, when such instructions are executed, further adjustments to the parameters of the first and second cameras are triggered based on the projection model.

[0122] Example 6 includes the non-transitory computer-readable media of Example 1, wherein, when such instructions are executed, a further report of scaling results is triggered via a graphical user interface.

[0123] Example 7 contains the non-transitory computer-readable media of Example 1, wherein the height of the object is obtained from a data set.

[0124] Example 8 contains a non-transitory computer-readable medium of either Example 1 or 7, wherein, when such instructions are executed, a branch is further triggered to use the object height to train a neural network model.

[0125] Example 9 includes a non-transitory computer-readable medium of either Example 7 or 8, wherein, when such instructions are executed, they further trigger the use of a trained branch of the neural network model to adjust the parameters of the first and / or second camera.

[0126] Example 10 includes the non-transient computer-readable media of Example 1, wherein the user input for iterative scaling is provided via a graphical user interface.

[0127] Example 11 includes an apparatus for restoring scale from monocular video, the apparatus comprising: interface circuitry for accessing input image from monocular video; and processor circuitry comprising one or more of the following: at least one of a central processing unit, a graphics processing unit, or a digital signal processor, the at least one of the central processing unit, the graphics processing unit, or the digital signal processor having control circuitry for controlling data movement within the processor circuitry, arithmetic and logic circuitry for performing one or more first operations according to instructions, and one or more registers for storing the one or more first operations and the results of the instructions in the apparatus; a field-programmable gate array (FPGA), the FPGA including logic gate circuitry, a plurality of configurable interconnects, and storage circuitry, the logic gate circuitry... The circuit and interconnects are used to perform one or more second operations, and the storage circuit is used to store the results of the one or more second operations; or the application-specific integrated circuit (ASIC) includes logic gate circuits for performing one or more third operations; the processor circuit is used to perform at least one of the first operations, the second operations, or the third operations to instantiate: an image segmentation circuit for segmenting an input image from the monocular video to detect objects in the camera's field of view; a camera parameter refinement circuit for iteratively refining the estimated camera parameters using object height; a relative depth scalar circuit for calculating the scale for the video; an iterative scale estimation circuit for iteratively refining the scale based on user input; and a trajectory plotting circuit for reporting the scale scaling results for visualization.

[0128] Example 12 includes the apparatus of Example 11, wherein the monocular video camera provides direct input to the scale restoration circuit.

[0129] Example 13 includes the apparatus of Example 11, wherein, when the instructions are executed, the input image segmentation system is further triggered to use a segmentation backbone network to perform the segmentation.

[0130] Example 14 includes the apparatus of Example 11, wherein, when the instructions are executed, the video scale is further triggered to be calculated using the first and second camera parameters.

[0131] Example 15 includes an apparatus of either Example 11 or 14, wherein, when such instructions are executed, further adjustments to the parameters of the first and second cameras are triggered based on the projection model.

[0132] Example 16 includes the apparatus of Example 11, wherein, when the instructions are executed, a further trigger is made to report the scaling results via a graphical user interface.

[0133] Example 17 includes the apparatus of Example 11, wherein the height of the object is obtained from a dataset.

[0134] Example 18 includes a device that includes either Example 11 or 17, wherein, when such instructions are executed, a branch is further triggered to use the object height to train the neural network model.

[0135] Example 19 includes an apparatus of either Example 17 or 18, wherein, when such instructions are executed, a trained branch of the neural network model is further triggered to adjust the parameters of the first and / or second camera.

[0136] Example 20 includes the apparatus of Example 11, wherein the user input for iterative scaling is provided via a graphical user interface.

[0137] Example 21 contains a method for rescaling from a monocular video, the method comprising: estimating camera parameters from a monocular input video; iteratively refining the estimated camera parameters; calculating a scale for relative depth; iteratively refining the scale with provided user input; and reporting the scale scaling result for visualization.

[0138] Example 22 incorporates the method of Example 21, wherein the monocular video camera provides direct input to the scale restoration circuit.

[0139] Example 23 includes the method of Example 21, wherein, when such instructions are executed, the input image segmentation system is further triggered to use a segmentation backbone network for this purpose.

[0140] Example 24 includes the method of Example 21, wherein, when such instructions are executed, the video scale is further triggered to be calculated using the first and second camera parameters.

[0141] Example 25 includes the method of either Example 21 or 24, wherein, when such instructions are executed, further adjustments to the parameters of the first and second cameras are triggered based on the projection model.

[0142] Example 26 includes the method of Example 21, wherein, when such instructions are executed, a further trigger is made to report the scaling results via a graphical user interface.

[0143] Example 27 includes the method of Example 21, where the height of the object is obtained from the data set.

[0144] Example 28 includes the method of either Example 21 or 27, wherein, when such instructions are executed, a branch is further triggered to use the object height to train the neural network model.

[0145] Example 29 includes the method of either Example 27 or 28, wherein, when such instructions are executed, a trained branch of the neural network model is further triggered to adjust the parameters of the first and / or second camera.

[0146] Example 30 incorporates the method of Example 21, wherein the user input for iterative scaling is provided via a graphical user interface.

[0147] Example 31 includes an apparatus for reconstructing scale from monocular video, the apparatus comprising: means for estimating camera parameters from monocular input video; means for iteratively refining the estimated camera parameters; means for calculating scale for relative depth; means for iteratively refining the scale using provided user input; and means for reporting the scale scaling result for visualization.

[0148] Example 32 includes the apparatus of Example 31, wherein the means for estimating camera parameters from monocular input video further includes the use of a segmented backbone network.

[0149] Example 33 includes the apparatus of Example 31, wherein the means for iteratively refining the estimated camera parameters further includes the use of a lane detection network.

[0150] Example 34 includes the apparatus of Example 31, wherein the means for calculating the scale for relative depth further includes the calculation of first and second camera parameters.

[0151] Example 35 includes an apparatus of either Example 31 or 34, wherein, when the instructions are executed, at least one of the parameters of the first and second cameras is further triggered to be adjusted according to the projection model.

[0152] Example 36 includes the apparatus of Example 31, wherein the means for iteratively refining the scale using provided user input further includes the use of a graphical user interface.

[0153] Example 37 includes the apparatus of Example 31, wherein the means for reporting the scaling results for visualization purposes further includes the use of a graphical user interface.

[0154] From the above, it can be understood that exemplary methods, devices, and manufactured objects have been revealed, which are expanded from monocular video restoration scales (e.g., absolute scales) for applications such as traffic accident simulation. For some applications such as autonomous vehicles and traffic cameras, monocular video cameras are superior to stereoscopic video cameras due to their low cost and ease of use.

[0155] Although some exemplary methods, apparatuses, and articles of manufacture have been disclosed herein, the scope of this patent is not limited thereto. Rather, this patent covers all methods, apparatuses, and articles of manufacture that fall substantially within the scope of the claims of this patent.

[0156] The following claims are incorporated herein by reference, together with each of the claims themselves as an individual embodiment of the invention.

[0157] 100: Scale Restoration System 102A: The first SLR camera 102B: Second SLR camera 102C: The third SLR camera 104: Internet 106: Data Center 108: Video Database 110: Scale Restoration Circuit 205: Data retrieval circuit 210: Input image segmentation circuit 215: Lane detection circuit 220: Error Prediction Circuit 225: Camera parameter refinement circuit 230: Object detection circuit 235: Object Height Loss Calculator 240: Relative depth scale scaling circuit 245: Relative translation scale scaling circuit 250: Iterative Scale Estimation Circuit 255: Trajectory Plotting Circuit 700: Graphical User Interface 705: Line 710: Video Control 715: Automatic calibration button 720: Track Map 805: Single-lens reflex video camera 810: Road 815: Relative Transformation (T C2R) Value 820: Camera pitch (θ) angle 825: Pedestrian 830: Reprojection Model Cells 905, 910, 915, 920 1000: Processor Platform 1005: Machine-executable instructions 1010: Internet 1015: Volatile Memory 1020: Non-volatile memory 1025: Processor Circuit 1026: Local Memory 1030: Busbar 1035: Mass storage device 1040: Input device 1045: Interface Circuit 1050: Output device 1100: Microprocessor 1102: Core 1104: First busbar 1106: Interface Circuit 1110: Level 2 (L2) Cache 1114: Control Unit Circuit 1116: Arithmetic and Logic (AL) Circuits 1118: Temporary Register 1120: Level 1 (L1) Cache Memory 1122: Second busbar 1200: FPGA circuit 1202: Input / Output (I / O) Circuit 1204: Circuit Configuration 1206: External Hardware 1208: Logic gate circuit 1210: Interconnection 1212: Storage circuit 1214: Dedicated operating circuit 1216: Special Purpose Circuits 1218: General Purpose Programmable Circuits 1220: Central Processing Unit (CPU) 1222: Digital Signal Processor (DSP)

Claims

1. A non-transitory computer-readable medium, comprising: Instructions are provided to cause at least one processor circuit to perform at least the following: segmenting an input image from a monocular video to detect objects in the camera's field of view; estimating at least one camera parameter from the segmented input image; iteratively refining the estimated at least one camera parameter based on the height associated with the detected object; calculating a scale for the video based on the estimated at least one camera parameter; iteratively refining the scale based on user input from a graphical user interface, the user input specifying the actual distance between two points in the input image; generating a vehicle trajectory map based on the scale; and displaying the vehicle trajectory map through the graphical user interface.

2. As in request item 1, a non-transitory computer-readable medium, wherein, These instructions are used to cause one or more of the at least one processor circuitry to: provide the segmented image to a neural network trained to predict the three-dimensional (3D) layout of lanes on a road based on road geometry identified in the segmented image; and estimate the parameters of the at least one camera based on the predicted three-dimensional layout of lanes on the road.

3. As in request item 1, a non-transitory computer-readable medium, wherein, These instructions are used to cause one or more of the processor circuits to segment the input image based on the segmented backbone network.

4. As in request item 1, a non-transitory computer-readable medium, wherein, The at least one camera parameter includes a first camera parameter and a second camera parameter, and the instructions are used to cause one or more of the at least one processor circuits to calculate the scale based on the first camera parameter and the second camera parameter.

5. As in request item 3, a non-transitory computer-readable medium, wherein, These instructions are used to cause one or more of the at least one processor circuits to adjust the parameters of the first camera and the parameters of the second camera based on the projection model.

6. As in claim 2, a non-transitory computer-readable medium, wherein, These instructions are used to cause one or more of the at least one processor circuitry to iteratively estimate the at least one camera parameter to reduce a combination of a first loss and a second loss, the first loss being based on an error associated with the estimation of the at least one camera parameter and the second loss being based on an error associated with the predicted three-dimensional layout of lanes on the road.

7. As in request item 1, a non-transitory computer-readable medium, wherein, These instructions are used to cause one or more of the at least one processor circuitry to determine the height of the detected object based on a neural network, the branches of which are trained based on a plurality of object heights.

8. As in request 4, a non-transitory computer-readable medium, wherein, The first camera parameter is the camera height, and the second camera parameter is the camera pitch.

9. An apparatus for restoring scale from monocular video, comprising: Interface circuit; Machine-readable instructions; And at least one programmable circuitry programmed based on the machine-readable instructions to: segment an input image from the monocular video to detect objects in the camera's field of view; estimate at least one camera parameter from the segmented input image; iteratively refine the estimated at least one camera parameter based on the height associated with the detected object; calculate a scale for the video based on the estimated at least one camera parameter; iteratively refine the scale based on user input from a graphical user interface specifying the actual distance between two points in the input image; generate a vehicle trajectory map based on the scale; and display the vehicle trajectory map through the graphical user interface.

10. The apparatus of claim 9, wherein, One or more of the at least one programmable circuitry perform the following: providing the segmented image to a neural network trained to predict the three-dimensional (3D) layout of lanes on a road based on road geometry identified in the segmented image; and estimating the parameters of the at least one camera based on the predicted three-dimensional layout of lanes on the road.

11. The apparatus as claimed in claim 9, wherein, One or more of the programmable circuits are used to segment the input image based on a segmentation backbone network.

12. The apparatus of claim 9, wherein, The at least one camera parameter includes a first camera parameter and a second camera parameter, and one or more of the at least one programmable circuits calculate the scale based on the first camera parameter and the second camera parameter.

13. The apparatus of claim 12, wherein, One or more of the at least one programmable circuitry adjust the parameters of the first camera and the parameters of the second camera based on a projection model.

14. The apparatus of claim 10, wherein, One or more of the at least one programmable circuitry iteratively estimates the at least one camera parameter to reduce a combination of a first loss and a second loss, the first loss being based on an error associated with the estimation of the at least one camera parameter and the second loss being based on an error associated with the predicted three-dimensional layout of lanes on the road.

15. The apparatus of claim 9, wherein, One or more of the programmable circuits determine the height of the detected object based on a neural network, the branches of which are trained based on a plurality of object heights.

16. The apparatus of claim 12, wherein, The first camera parameter is the camera height, and the second camera parameter is the camera pitch.

17. A method for resizing self-monocular video, the method comprising: The input image from the monocular video is segmented to detect objects in the camera's domain; at least one camera parameter is estimated from the segmented input image; The estimated camera parameters are iteratively refined based on the height associated with the detected object; a scale for the video is calculated based on the estimated camera parameters; the scale is iteratively refined based on user input from a graphical user interface, the user input specifying the actual distance between two points in the input image; a vehicle trajectory map is generated based on the scale; and the vehicle trajectory map is displayed through the graphical user interface.

18. As in request item 17, wherein, This step of segmenting the input image is based on the segmentation backbone network.

19. The method as described in request item 17, wherein, The at least one camera parameter includes a first camera parameter and a second camera parameter, and the step of calculating the scale is based on the first camera parameter and the second camera parameter.

20. The method as described in request item 19, wherein, The step of iteratively refining the estimated at least one camera parameter includes adjusting the first camera parameter and the second camera parameter based on the projection model.

Images