Adaptive optimal view selection for vehicle display

By generating and selecting a virtual camera viewpoint with the highest view score, the problem that the view does not meet the needs of complex scenes during vehicle operation in the prior art is solved, and more efficient view presentation and user satisfaction are achieved.

CN118942359APending Publication Date: 2024-11-12FORD GLOBAL TECH LLC
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202410520179.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-05-12
Filing Date
2024-04-28
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

During vehicle operation, it is difficult for the prior art to automatically select the best viewpoint to meet view standards in complex driving scenarios, resulting in time-consuming and unsatisfactory user adjustments.

Method used

The alternative viewpoint of the virtual camera is generated by the processor, based on the viewpoint switching conditions and user preferences, a candidate viewpoint with the highest view score is selected as the best viewpoint, and a view of the 3D virtual driving space is presented on the display device.

Benefits of technology

It improves the efficiency of view selection and user satisfaction, reduces the time and resource consumption of manual adjustment, and adapts to the view needs in complex driving scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118942359A_ABST
    Figure CN118942359A_ABST
Patent Text Reader

Abstract

The invention provides adaptive optimal viewpoint selection for a vehicle display. Devices and methods disclosed herein provide an optimal viewpoint from which a virtual camera views a scene in an environment around a self-aware vehicle. A first value is determined that indicates an extent to which a first view from a default viewpoint of the virtual camera satisfies a first view observation target and a second view observation target. A second value is determined that indicates an extent to which a second view from an alternative viewpoint of the virtual camera satisfies the first view observation target and the second view observation target. If the second value exceeds the first value by an amount greater than a specified standard value, the alternative viewpoint is selected for the virtual camera. If the second value does not exceed the first value by an amount greater than the specified standard value, the default viewpoint is selected for the virtual camera.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure includes techniques for selecting an optimal viewpoint from which to obtain a view to be presented on a vehicle display. Background Art

[0002] Vehicle systems may include perception, prediction, and planning subsystems. The perception subsystem typically includes multiple on-board sensors to capture a view of the vehicle's surroundings. The prediction subsystem may predict the trajectory of an observed object detected in the captured view. The planning subsystem may plan the vehicle's trajectory based on the predicted trajectory of the observed object detected in the captured view. Advanced driver assistance systems (ADAS) may provide a virtual view of the real-time driving situation to the vehicle operator based on vehicle motion and path planning. While the vehicle is operating, operating scenarios may appear and / or change without any change in the vehicle's motion and / or planned path. In such cases, the default view in the vehicle display may not be optimal for meeting a given occupant viewing objective. Summary of the invention

[0003] According to the present invention, a device is provided, which has: a processor; a memory, which is communicatively coupled to the processor and stores processor-executable instructions, and the processor-executable instructions configure the processor to: present a default view of a 3D virtual driving space from a default viewpoint of a virtual camera on a display device of a vehicle; generate an alternative viewpoint of the virtual camera by constructing a viewpoint vector extending from a position of the virtual camera in the 3D virtual driving space to a position on a ground plane in a top-down plan view of a real-world driving space; change the viewpoint of the virtual camera from the default viewpoint to the alternative viewpoint based on a viewpoint switching condition; and provide a view of the 3D virtual driving space from the alternative viewpoint to a display interface, thereby presenting the view of the 3D virtual driving space from the alternative viewpoint on the display device.

[0004] According to an embodiment, the processor is also configured to: obtain corresponding first view scores of corresponding corresponding views of corresponding corresponding candidate alternative viewpoints from the virtual camera; and select the corresponding candidate alternative viewpoint with the highest score among the corresponding first view scores as the alternative viewpoint.

[0005] According to an embodiment, the processor is also configured to: identify, from the alternative viewpoint, an observation object including the vehicle on the ground plane within the field of view (FoV) of the virtual camera; and evaluate the alternative viewpoint based on motion parameters of the observation object within the FoV according to the viewpoint switching condition.

[0006] According to an embodiment, the processor is also configured to: obtain a first count of a first observed object on the ground plane within the field of view of the virtual camera from the alternative viewpoint; obtain a second count of a second observed object on the ground plane within the field of view (FoV) of the virtual camera from the default viewpoint; and test the alternative viewpoint by comparing the first count with the second count according to the viewpoint switching condition.

[0007] According to an embodiment, the present invention is also characterized by a scene table, which maps corresponding driving scenes to corresponding corresponding default viewpoints, wherein the processor is also configured to: receive real-time motion data and a planned path of the vehicle; identify a scene corresponding to the real-time motion data and the planned path; and identify a default viewpoint corresponding to the scene, thereby identifying the default viewpoint.

[0008] According to an embodiment, the processor is also configured to: identify an observation object including the vehicle on the ground plane; assign a motion parameter value to the observation object; receive a first path view preference via a user interface; select a first viewing area based on the first path view preference and the motion parameter value; and generate a candidate alternative viewpoint in the first viewing area.

[0009] According to an embodiment, the processor is further configured to: receive a second path view preference via the user interface; select a second viewing area based on the second path view preference and the motion parameter value; and generate the candidate alternative viewpoint in the second viewing area.

[0010] According to an embodiment, the processor is also configured to: generate the candidate alternative viewpoint in the first viewing area by constructing a viewpoint vector from the center of the first face of the first viewing area to the vehicle on the ground plane; identify the observation object within the FoV of the virtual camera from the candidate alternative viewpoint; test the candidate alternative viewpoint based on the motion parameters of the observation object on the ground plane within the FoV of the virtual camera as seen from the candidate alternative viewpoint according to the viewpoint switching condition; based on the test, generate a first local candidate alternative viewpoint to identify a first local maximum score; and select the local candidate alternative viewpoint on the first face based on the first local maximum score.

[0011] According to an embodiment, the processor is also configured to: receive an adjustment to the viewpoint switching condition via a user interface; retest the candidate replacement viewpoint based on the adjustment; and change the viewpoint of the virtual camera to a position on the second surface of the first viewing area based on a result of the retest.

[0012] According to an embodiment, the processor is further configured to: identify an observation object within the FoV of the virtual camera on the second surface of the first viewing area; score the viewpoint based on the observation object within the FoV of the virtual camera; and evaluate the viewpoint switching condition based on the adjustment.

[0013] According to the present invention, a method is provided, the method comprising: presenting a default view of a 3D virtual driving space from a default viewpoint of a virtual camera on a display device of a vehicle; generating an alternative viewpoint of the virtual camera by constructing a viewpoint vector extending from a position of the virtual camera in the 3D virtual driving space to a position on a ground plane in a top-down plan view of a real-world driving space; changing the viewpoint of the virtual camera from the default viewpoint to the alternative viewpoint based on a viewpoint switching condition; and providing a view of the 3D virtual driving space from the alternative viewpoint to a display interface, thereby presenting the view of the 3D virtual driving space from the alternative viewpoint on the display device.

[0014] In one aspect of the present invention, the method includes: obtaining corresponding first view scores of corresponding corresponding views of corresponding corresponding candidate alternative viewpoints from the virtual camera; and selecting a corresponding candidate alternative viewpoint having a highest score among the corresponding first view scores as the alternative viewpoint.

[0015] In one aspect of the present invention, the method includes: identifying, from the alternative viewpoint, an observation object including the vehicle on the ground plane within the field of view (FoV) of the virtual camera; and testing the alternative viewpoint based on motion parameters of the observation object within the FoV according to the viewpoint switching condition.

[0016] In one aspect of the present invention, the method includes: obtaining a first count of a first observed object on the ground plane within the field of view of the virtual camera from the alternative viewpoint; obtaining a second count of a second observed object on the ground plane within the field of view (FoV) of the virtual camera from the default viewpoint; and testing the alternative viewpoint by comparing the first count with the second count according to the viewpoint switching condition.

[0017] In one aspect of the present invention, the method includes: receiving real-time motion data and a planned path of the vehicle; identifying a scene corresponding to the real-time motion data and the planned path; and identifying a default viewpoint corresponding to the scene, thereby identifying the default viewpoint.

[0018] In one aspect of the present invention, the method includes: identifying an observation object including the vehicle on the ground plane; assigning a motion parameter value to the observation object; receiving a first path view preference via a user interface; selecting a first viewing area based on the first path view preference and the motion parameter value; and generating a candidate alternative viewpoint in the first viewing area.

[0019] In one aspect of the invention, the method includes: receiving a second path view preference via the user interface; selecting a second viewing area based on the second path view preference and the motion parameter value; and generating the candidate alternative viewpoint in the second viewing area.

[0020] In one aspect of the present invention, the method includes: generating the candidate alternative viewpoint in the first viewing area by constructing a viewpoint vector from the center of the first face of the first viewing area to the vehicle on the ground plane; identifying an observation object within the FoV of the virtual camera from the candidate alternative viewpoint; testing the candidate alternative viewpoint based on motion parameters of the observation object on the ground plane within the FoV of the virtual camera as seen from the candidate alternative viewpoint according to the viewpoint switching condition; based on the test, generating a first local candidate alternative viewpoint to identify a first local maximum score; and selecting a local candidate alternative viewpoint on the first face based on the first local maximum score.

[0021] In one aspect of the present invention, the method includes: receiving an adjustment to the viewpoint switching condition via a user interface; retesting the candidate replacement viewpoint based on the adjustment; and changing the viewpoint of the virtual camera to a position on a second surface of the first viewing area based on a result of the retest.

[0022] In one aspect of the present invention, the method includes: identifying an observation object within the FoV of the virtual camera on the second side of the first viewing area; scoring the viewpoint based on the observation object within the FoV of the virtual camera; and evaluating the switching condition based on the adjustment. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a block diagram of an exemplary vehicle system;

[0024] Figure 2 It is shown Figure 1 a block diagram showing further details of the vehicle system shown;

[0025] Figure 3 is a block diagram of an adaptive optimal viewpoint selection device;

[0026] Figure 4AIt is a top-down orthographic view of the ground plane of the 3D virtual driving space;

[0027] Figure 4B is an illustration of a 3D virtual driving space including a spherical grid dividing the space into surfaces;

[0028] Figure 4C is a diagram showing the construction of a viewpoint vector;

[0029] Figure 4D It is shown Figure 4B Illustration of the area of ​​the spherical grid;

[0030] Figure 5 is a flow chart of a method for identifying an optimal viewpoint;

[0031] Figure 6 is a flow chart of a method for identifying an optimal viewpoint;

[0032] Figure 7 is a flow chart of a method for scoring views to determine an optimal viewpoint; and

[0033] Figure 8 is a flow chart of a method for notifying a user of a change in a planned path caused by viewpoint switching. DETAILED DESCRIPTION

[0034] As described herein, an optimal viewpoint is an alternative to a default viewpoint that can be identified and provides a view that satisfies a view criterion to a greater extent than a view from the default viewpoint. Thus, the present disclosure provides systems and methods for determining which of a default view and possible alternative views is the optimal view for a user to view a current situation.

[0035] like Figure 1 As seen, system 100 includes a host vehicle 101, which in turn includes a computer 104, which is communicatively coupled to various elements including sensors 108, subsystems or components 110 (such as steering, propulsion and braking), a human machine interface (HMI) 112 and a communication module 114 via a communication network (such as a vehicle network 106).

[0036] The vehicle computer 104 (and the remote server 118 discussed below) includes a processor and a memory. The memory of the computer 104, such as described herein, includes one or more forms of computer 104 readable media and stores instructions executable by the vehicle computer 104 for performing various operations, such that the vehicle computer is configured to perform various operations including as disclosed herein.

[0037] For example, the vehicle computer 104 may be a general-purpose computer 104 having a processor and memory as described above, and / or may include an electronic control unit ECU or controller for a specific function or set of functions, and / or a dedicated electronic circuit including an ASIC (Application Specific Integrated Circuit) manufactured for a specific operation, such as an ASIC for processing sensor 108 data and / or transmitting sensor 108 data. In another example, the vehicle computer 104 may include an FPGA (Field Programmable Gate Array), which is an integrated circuit manufactured to be configurable by a user. Typically, hardware description languages ​​such as VHDL (Very High Speed ​​Integrated Circuit Hardware Description Language) are used in electronic design automation to describe digital and mixed signal systems such as FPGAs and ASICs. For example, an ASIC is manufactured based on VHDL programming provided before manufacturing, and the logic components inside the FPGA can be configured based on, for example, VHDL programming stored in a memory electrically connected to the FPGA circuit. In some examples, a combination of a processor, an ASIC, and / or an FPGA circuit may be included in the computer 104. Furthermore, the vehicle computer 104 may include multiple computers 104 in the vehicle, such as multiple ECUs (electronic control units) or the like, which operate together to perform the operations attributed herein to the vehicle computer 104 .

[0038] The memory may be of any type, for example, a hard drive, a solid state drive, a server 118, or any volatile or non-volatile medium. The memory may store the collected data sent from the sensor 108. The memory may be a separate device from the computer 104, and the computer 104 may retrieve the information stored by the memory via a communication network in the vehicle, such as the vehicle network 106 (e.g., through a CAN bus, a wireless network, etc.). Alternatively or in addition, the memory may be part of the computer 104, for example, as a memory of the computer 104.

[0039] The computer 104 may include programming to operate one or more components 110, such as vehicle braking, propulsion (e.g., one or more of an internal combustion engine, an electric motor, a hybrid engine, etc.), steering, climate control, interior and / or exterior lights, etc., and to determine whether and when the computer 104 (rather than a human operator) controls such operations. In addition, the computer 104 may be programmed to determine whether and when a human operator controls such operations. The computer 104 may include more than one processor or be communicatively coupled to more than one processor, such as via a vehicle network 106 (such as a communication bus) as further described below, such as included in components 110 (such as sensors 108, electronic control units (ECUs), etc.) included in the vehicle for monitoring and / or controlling various vehicle components (e.g., a powertrain controller, a brake controller, a steering controller, etc.).

[0040] The computer 104 is typically arranged to communicate over a vehicle network 106, which may include a communication bus in the vehicle, such as a controller area network (CAN), etc., and / or other wired and / or wireless mechanisms. The vehicle network 106 is a communication network via which messages can be exchanged between various devices in the vehicle (e.g., sensors 108, components 110, computers 104, etc.). The computer 104 may typically be programmed to send and / or receive messages to and / or from other devices in the vehicle (e.g., any or all of the ECUs, sensors 108, actuators, components 110, communication modules, human-machine interfaces (HMIs) 112, etc.) via the vehicle network 106. For example, subsystems of various components 110 (e.g., components 110 may be controlled by corresponding ECUs) and / or controllers 108 may provide data to the computer 104 via the vehicle network 106.

[0041] In addition, in the case where the computer 104 actually includes multiple devices, the vehicle network 106 can be used for communication between the devices represented as the computer 104 in this disclosure. For example, the vehicle network 106 may include a controller area network (CAN) in which messages are communicated via a CAN bus, or a local interconnect network (LIN) in which messages are communicated via a LIN bus. In some embodiments, the vehicle network 106 may include a network in which other wired communication technologies and / or wireless communication technologies (e.g., Ethernet, In some embodiments, the vehicle network 106 may represent a combination of multiple networks that may be of different types that support communication between devices in the vehicle. For example, the vehicle network 106 may include: a CAN (or CAN bus), in which some devices in the vehicle communicate via the CAN bus; and a wired or wireless local area network, in which some devices in the vehicle communicate according to an Ethernet or Wi-Fi communication protocol.

[0042] The vehicle 101 typically includes a variety of sensors 108. A sensor 108 is a device that can obtain one or more measurements of one or more physical phenomena. Some sensors 108 detect the internal state of the vehicle, such as wheel speed, wheel orientation, and engine and transmission variables. Some sensors 108 detect the location or orientation of the vehicle, such as a global positioning system GPS sensor 108; an accelerometer, such as a piezoelectric or micro-electromechanical system MEMS; a gyroscope, such as a rate gyroscope, a ring laser gyroscope, or a fiber gyroscope; an inertial measurement unit IMU; and a magnetometer. Some sensors 108 detect the external world, such as a radar sensor 108, a scanning laser rangefinder, a light detection and ranging lidar device, and an image processing sensor 108 (such as a camera). A lidar device detects the distance to an object by emitting a laser pulse and measuring the flight time for the pulse to travel to the object and back. Some sensors 108 are communication devices, such as a vehicle-to-infrastructure V2I or a vehicle-to-vehicle V2V device. The operation of the sensor 108 may be affected by obstructions (e.g., dust, snow, insects, etc.). Typically, but not necessarily, the sensor 108 includes a digital-to-analog converter to convert the sensed analog data into a digital signal, which can be provided to the digital computer 104, for example, via a network.

[0043] The sensors 108 may include various devices and may be configured to sense the environment, provide data about the machine, etc. in various ways. For example, the sensors 108 may be mounted to stationary infrastructure elements on, above, or near the road. In addition, various controllers in the vehicle may act as sensors 108 to provide data via the vehicle network 106 or bus, such as data related to vehicle speed, location, subsystem and / or component 110 status, etc. In addition, other sensors 108 in or on the infrastructure such as vehicles, stationary infrastructure elements, etc. may include cameras, short-range radars, long-range radars, lidars, and / or ultrasonic transducers, weight sensors 108, accelerometers, motion detectors, etc., i.e., sensors 108 that provide various data. To provide only a few non-limiting examples, the sensor 108 data may include data for determining the location of the component 110, the location of the object, the speed of the object, the type of the object, the slope of the road, the temperature, the presence or amount of moisture, the fuel level, the data rate, etc.

[0044] The computer 104 may include programming to command one or more actuators to operate one or more vehicle subsystems or components 110, such as vehicle braking, propulsion, or steering. That is, the computer 104 may actuate control of speed in the vehicle by controlling one or more of an internal combustion engine, an electric motor, a hybrid engine, etc., and / or may actuate control of braking, steering, climate control, interior and / or exterior lights, etc. The computer 104 may include more than one processor or be communicatively coupled to more than one processor, such as via a vehicle network 106, such as included in a component 110 (such as a sensor 108, an electronic control unit (ECU), etc.) for monitoring and / or controlling various vehicle components (e.g., an ECU, etc., such as a powertrain controller, a brake controller, a steering controller, etc.).

[0045] The vehicle may include an HMI 112 (human machine interface), such as one or more of a display, a touch screen display, a microphone, a speaker, etc. A user may provide input to a device such as the computer 104 via the HMI 112. The HMI 112 may communicate with the computer 104 via the vehicle network 106, for example, the HMI 112 may send messages to the computer 104 including user input provided via a touch screen, a microphone, a camera that captures gestures, etc., and / or may display output, such as via a screen, a speaker, etc. In addition, the operation of the HMI 112 may be performed by a portable user device (not shown) (such as a smart phone, etc.) that communicates with the vehicle computer 104, such as via Bluetooth, etc.

[0046] The computer 104 may be configured to communicate via a vehicle-to-vehicle communication module 114 or to interface with devices external to the vehicle, for example, vehicle-to-vehicle V2V, vehicle-to-infrastructure or outside V2X or vehicle-to-outside (including cellular communications C-V2X wireless communications (cellular), DSRC, etc.) via a wide area network 116 and / or to another vehicle, to an infrastructure element (typically via direct radio frequency communication) and / or (typically via a network) to a remote server 118. The module may include one or more mechanisms by which the vehicle's computer 104 may communicate, including any desired combination of wireless (e.g., cellular, wireless, satellite, microwave, and radio frequency) communication mechanisms and any desired network topology or multiple topologies when multiple communication mechanisms are used. Exemplary communications provided via the module may include cellular, Bluetooth, IEEE 802.11, dedicated short-range communications DSRC, cellular V2X CV2X, etc.

[0047] The computer 104 may be programmed to communicate with one or more remote sites (such as a remote server 118) via the wide area network 116. The wide area network 116 may include one or more mechanisms by which the vehicle computer 104 can communicate with, for example, the remote server 118. For example, the vehicle 101 may include a wireless transceiver (i.e., a transmitter and / or a receiver) to send and receive messages outside the vehicle 101. Thus, the network may include one or more of a variety of wired or wireless communication mechanisms, including any desired combination of wired (e.g., cable and fiber optic) and / or wireless (e.g., cellular, wireless, satellite, microwave, and radio frequency) communication mechanisms and any desired network topology or multiple topologies when multiple communication mechanisms are used. Exemplary communication networks include wireless communication networks that provide data communication services, such as using Bluetooth, Bluetooth Low Energy BLE, IEEE 802.11, vehicle-to-vehicle V2V or vehicle-to-external V2X (such as cellular V2X CV2X, dedicated short-range communication DSRC, etc.), local area networks, and / or wide area networks 116 including the Internet.

[0048] Server 118 may include one or more computing devices accessible via wide area network 116 , eg, having respective processors and memory and / or associated data storage devices.

[0049] Figure 2 is a block diagram illustrating a vehicle system 100 including an advanced driver assistance system (ADAS) 200 in an example implementation. The ADAS 200 includes a vehicle perception subsystem 130 and a vehicle motion and planning subsystem 150. The vehicle motion and planning subsystem 150 provides real-time vehicle motion data 137, 147 and a planned path 157.

[0050] The view management and storage system 145 can implement a 3D virtual driving space 212. The virtual driving space 212 is a virtual 3D view space of the real world 3D space surrounding the vehicle 101 advancing along the path. A "view" is an observation of the real world space taken by a camera. The range of observation is determined by the FoV of the camera placed at a point in the real world space (positioned in pitch, yaw, and roll). The point in space at which the camera is positioned and placed is the viewpoint. The view observed from the viewpoint of the physical camera can be captured by the physical camera, that is, sensed as an image of the scene in the FoV.

[0051] A "virtual view" is a view synthesized, constructed, or otherwise derived from one or more images of one or more views of the real world captured by a physical camera. The view of the virtual driving space 212 can be constructed from raw data from imaging and / or ranging sensors 108 (e.g., cameras 5, 15, 20, 25, 30, and / or radar) and inertial navigation systems 19, map data, satellites 62, images from traffic and surveillance cameras 60, aerial drone cameras, and / or from other vehicles that can provide their sensor data.

[0052] Virtual views may also be constructed from other virtual views. For example, a single view of a real-world space may be captured as an image by a physical camera. An image processing system may process the image to project the view from the plane of the camera imager onto many different view planes, thereby allowing the view to be presented from many different angles. Thus, many (e.g., thousands) of different virtual views of the real-world space may be generated from a single view. Many of the single views may likewise be re-projected, synthesized, or otherwise combined with other views and virtual views to generate even more virtual views of the real-world space.

[0053] Every view (virtual or real) has a viewpoint, regardless of whether a physical camera captures the view. Given a virtual view, the corresponding viewpoint can be estimated or "reverse engineered", for example, by analyzing the viewpoint and comparing the anchor points in the view with the anchor points in a reference view. In other words, a view that is constructed from other views and therefore not captured by a camera can be analyzed to estimate the viewpoint that the camera would have if the camera captured that view. The estimated viewpoint is the virtual viewpoint.

[0054] Reference now Figure 3 In the example described herein, the real world driving space 212 is modeled as a sphere 333 that encloses a volume of space around the vehicle. The space within the sphere 333 is represented by a collection of virtual views from many different viewpoints on the surface of the sphere 333. The reason for estimating the viewpoint of the virtual view is to determine where the virtual view fits within the spherical volume of space around the vehicle 101 (defined by the sphere 333), and to be able to retrieve the view based on the viewpoint.

[0055] The views that fit within the sphere 333 are stored in the data store 216 and indexed to their corresponding estimated viewpoints. The views are also extensively processed to provide "descriptors" that describe various aspects of the view content. The data store 216 containing the views, together with the database that indexes the views by viewpoint, constitutes a "virtual driving space". A "virtual camera" is a hypothetical camera, i.e., implemented by computer program instructions, which "captures" a view of the virtual driving space by searching the data store 216 in response to a query to retrieve the corresponding view. The virtual camera is "positioned" at a viewpoint on the sphere 333 by providing the viewpoint as a search query. In other words, the virtual camera is a query system for the view storage space. Queries are expressed in terms of viewpoints.

[0056] In some cases, a view with specific characteristics is sought, rather than a view from a specific viewpoint. For example, the ADAS 200 may specify a default viewpoint. The default viewpoint is a predetermined viewpoint selected to provide a view that meets general goals in a typical driving scenario. In a complex driving situation, the view from the default ADAS viewpoint may not have characteristics that meet a more specific goal (i.e., a goal based on specific characteristics of the current driving situation that may be more complex than a typical driving scenario).

[0057] In those complex situations, the user may seek a better vantage point (viewpoint) to view the virtual driving space. For example, the ADAS 200 view presentation subsystem (not shown) typically provides a view from a default viewpoint for presentation on the display surface of the display device 230. In this case, the user may not know if there are any better vantage points, and there may not be any vantage points better than the default viewpoint. To find out, the user may try to manipulate the user interface through various viewpoints to try to tune into a "better" view. Figure 2 ADAS 200 is shown with a display that allows a user to manually change the virtual viewpoint by interacting with display device 230. The user may find manual searching inconvenient and time consuming, which may lead to user dissatisfaction.

[0058] The processor may search the virtual driving space by providing successive corresponding viewpoints as queries and evaluating the corresponding successive views according to a criterion defining "better" until a view that satisfies the criterion is retrieved. Even then, the processor would not know whether some other view in the virtual driving space satisfies the criterion to a greater extent than the retrieved view. To find the answer, the processor would have to evaluate the views from every viewpoint on the sphere 333. This exhaustive search consumes time and processing resources and is incompatible with the real-time requirements of ADAS applications.

[0059] The present disclosure provides a system and method for searching for an optimal viewpoint in an efficient manner to reduce the time and / or resource consumption of searching for a better view. Whether a candidate optimal viewpoint is an optimal viewpoint is determined by comparing the degree to which a view from a candidate optimal viewpoint satisfies a criterion with the degree to which a view from a default viewpoint satisfies the criterion. Of the two views, the view whose other view satisfies the criterion to a greater extent is the optimal view, and its viewpoint is the optimal viewpoint.

[0060] The present disclosure provides devices and methods by which a user can adjust the criteria for defining a "better" view. In a first method, a user can select which of two extreme values ​​in a measurable path characteristic the user prefers to use to identify the best view, i.e., the highest measured value (maximum) or the lowest measured value (minimum). The device identifies the best view by finding a maximum function or a minimum function that outputs based on which of the minimum or maximum values ​​was selected. In a second method, a user can adjust the search to favor views based on the complexity (or immediacy) of the view content by changing the value of a hyperparameter (time), which determines the extent to which the content of the view represents events that may occur relatively soon versus relatively distant (future) events and scores the views accordingly.

[0061] The camera viewpoint is usually defined in terms of a camera coordinate system 13. The viewpoint is specified by both a point and the view from that point. "Camera viewpoint" in this document means the spatial position (x, y, z) and spatial orientation (θ1, θ2, θ 3) (pitch, yaw, roll). The view management and storage system 145 may apply conventional techniques to convert coordinates from the camera coordinate system 13 to the real world coordinate system 10, and vice versa. The viewpoint of the virtual camera 214 is specified by a position specified by x, y, z coordinates that may be in the coordinate system 10 of the virtual driving space 212. The coordinate system 10 of the virtual driving space 212 may correspond to the coordinate system 10 in the real world driving space. The viewpoint is further specified by the pitch θ1, yaw θ2, and roll θ3 of the camera relative to the x-axis, y-axis, and z-axis that give its position.

[0062] Table 1 below shows an example portion of a data set 265 in the form of a table providing a vehicle operation data set, which is a specified value set of vehicle motion and path parameters. Each data set defines a driving scene. ADAS can use Table 1 to determine a default viewpoint. Example driving scenes A to D are defined by corresponding corresponding motion and path parameter value sets and corresponding corresponding default viewpoints of virtual camera 214. The vehicle system 100 compares the received real-time motion data sets 137, 147 and planned path data sets 157 with the corresponding scene motion data sets 137, 147 and planned path data sets 157 to determine which default viewpoint the vehicle will use. The vehicle system provides the default viewpoint specified by the data set in the scene data set 265 to the virtual camera 214 corresponding to the identified scene.

[0063] Table I

[0064]

[0065] For example, scene A is defined by a vehicle speed range of less than 25 MPH 137, a straight line planned path 157, and a planned speed of less than 25 MPH 147. In this scene, the default viewpoint of the virtual camera 214 is defined by the x, y, z position (-6, 0, -6) and orientation (pitch, yaw, roll) θ1(0), θ2(45°), θ3(0).

[0066] Scene B is defined by a vehicle speed range 137 greater than 25 MPH and less than 55 MPH, a straight line planned path 157, and a planned vehicle speed 147 greater than 25 MPH and less than 55 MPH. In this scene, the default viewpoint of the virtual camera 214 is defined by a position (-10, 0, 8) and an orientation θ1(0), θ2(30°), θ3(0).

[0067] Scenario C is defined by a vehicle speed 137 range greater than 55 MPH, a straight line planned path 157, and a planned speed 147 greater than 55 MPH. In this scenario, the default viewpoint of the virtual camera 214 is defined by position (-8, 0, 6) and orientation (pitch, yaw, roll) θ1(0), θ2(25°), θ3(0).

[0068] Scenario D specifies a speed range of less than 5 MPH, which may correspond to the vehicle slowing down to make a specified standard right turn in its planned path, after which the vehicle motion planning requires a speed range between 25 MPH and 44 MPH. In this scenario, the default virtual camera viewpoint is defined by position (-10, -8, 8) and orientation (pitch, yaw, roll) θ1(0), θ2(35°), θ3(75).

[0069] Figure 33 is a block diagram of an apparatus 300 for viewpoint selection and automatic viewpoint switching. The apparatus 300 includes a memory 370, a data set 365, a processor 310, a view management interface 318, a display interface 330, and a user interface 380. The memory 370 stores processor-executable instructions that configure the processor 310 to perform methods, processes, and functions including those described herein.

[0070] The processor 310 is configured to communicate with the view management and storage system 145 via the virtual camera 214 to retrieve a view of the virtual driving space 212 by providing a virtual camera viewpoint to the virtual camera 214. The virtual camera 214 "captures" an image from the virtual driving space 212 and provides it to the processor 310 or the display interface 330 for presentation on the display device 230. The device 300 is also configured to present the view on a transparent display of a head-up display (HUD) (not depicted) of the vehicle 101.

[0071] As described above, a viewpoint is specified by both a point and a view from that point. For a virtual camera's viewpoint, the point is given as the location of the virtual camera relative to a point (x, y, z) on the surface of a grid defining a virtual sphere 333. In this context, the "view" of a virtual camera is an area (typically a cone area) of the ground plane 203 that is coextensive with the field of view (FoV) of the virtual camera. The FoV of a virtual camera is the horizontal and vertical spatial extent of the virtual camera's virtual imager at a given distance from the ground plane 203 along the camera's optical axis. For ease of describing the present subject matter, the example virtual camera 214 is considered to have a fixed FoV, but virtual cameras that do not have a fixed FoV are suitable for use in the disclosed examples.

[0072] The "view" from a specified point is given as the angular spatial orientation of the virtual camera with six degrees of freedom of motion (yaw, pitch, roll) relative to its optical axis. When the optical axis is pointed at the ground plane 203, the camera's FoV defines the angular range (or conic section) of the ground plane that can be imaged by the camera relative to the intersection of the optical axis and the ground plane. If objects on the ground plane 203 are within the camera's field of view (FoV), they are in the view captured by the camera. The ground plane 203 is a 2D plane in the virtual driving space 212 that represents the surface of the earth over an area so small that the curvature is negligible.

[0073] The term "optimal" in this context means best or preferred with reference to a set of available alternatives. An optimal viewpoint is a preferred viewpoint (ie, a viewpoint that satisfies defined criteria to a greater extent than an alternative).

[0074] As described above, the view management interface 318 queries the virtual driving space 212 by providing a viewpoint to the virtual camera 214. The processor 310 may identify a default viewpoint based on the motion data 137, 147 and the planned path 157. The processor 310 may provide the default viewpoint to the view management interface 318. The view management interface 318 may search the virtual driving space 212 for a default view by providing the default viewpoint to the virtual camera 214. The default view may be presented on the display device 230. The processor 310 is configured to evaluate the default view retrieved by the view management interface 318 to determine whether the default view best meets the specified goal. The processor 310 is also configured to receive a signal indicating the user's viewing preference via the display interface 330.

[0075] The processor 310 is configured to receive the motion data 137, 147 and the planned path 157 from the vehicle system 100 as the vehicle 101 moves along its planned path. The processor 310 uses the motion data 137, 147 and the planned path 157 to identify a default viewpoint to be used by the view management interface 318 to retrieve a view for presentation on the display device 230. The processor 310 obtains a default view or a description of a default view based on the default viewpoint.

[0076] Table II shows that dataset 365 may be stored in a table format, for example, a scenario table mapping respective driving scenarios to corresponding respective default viewpoints is provided below.Dataset 365 partially replicates dataset 265 (see Table I above).

[0077] Processor 310 uses data set 365 to identify a current default viewpoint by comparing the values ​​designated as real-time motion data 137, 147 and the values ​​designated as planned path 157 data to the values ​​designated as motion data and planned path data in each of the scenes in data set 365. Processor 310 looks up the default viewpoint that corresponds to the identified scene.

[0078] Additionally, the processor 310 may look up any corresponding viewpoint constraints and a default tuning value T from the scene dataset 365 . Such data is not included in the dataset 265 of the view management interface 318 .

[0079] Table II

[0080]

[0081] The tuning value T can be implemented in one of at least three different ways. For example, the tuning value T can be implemented as a hyperparameter, a penalty, or a threshold, depending on the details of the application.

[0082] A hyperparameter is a parameter whose value controls (tunes) the performance of the search for the best viewpoint. While the hyperparameter values ​​themselves cannot be learned, what a given user considers to be the "best performance" of the device 300 can be learned, and this can be related to the hyperparameter value, which can then be set for the given user. The penalty function imposes a penalty on views that violate constraints (such as constraints on view complexity C). The threshold is the magnitude that must be exceeded before the default viewpoint can be switched to an alternative viewpoint.

[0083] like Figure 3 As shown, device 300 includes user interface 380. User interface 380 includes controls for adjusting tuning value T (386) and controls for selecting a path view preference (PREF1 at 382 or PREF2 at 384), where PREF1 represents a preference for viewing the highest resistance path and PREF2 represents a preference for viewing the lowest resistance path. For purposes of this specification, "path resistance" is the quality of a vehicle path that tends to impede vehicle 101 from advancing along a path due to resistance from obstacles (such as other vehicles that intercept a portion of a planned path at a time when vehicle 101 is expected to intercept that portion of the planned path) such that vehicle 101 may be prevented from advancing.

[0084] For example, at an intersection where vehicle 101 plans to cross two lanes of traffic to make a left turn, the view of the intersection from the driver's left window shows objects coming oncoming (i.e., moving along a path (lane) toward vehicle 101). The view of the intersection from the driver's right window shows objects moving along a path (lane) away from vehicle 101. Objects moving toward vehicle 101 are potential interceptors of vehicle 101 if vehicle 101 proceeds to the intersection, regardless of which path the vehicle might take. Objects moving away from vehicle 101 are not potential interceptors of vehicle 101 if vehicle 101 proceeds to the intersection, regardless of which path the vehicle might take.

[0085] Thus, the path (lane) along which the vehicle is traveling toward vehicle 101 is a high resistance path (relative to the vehicle moving into the intersection). The path (lane) along which the vehicle is traveling away from vehicle 101 is a low resistance path (relative to the vehicle moving into the intersection). In this case, some users may prefer a view of the path with oncoming objects (i.e., the path of highest resistance). Other users may prefer a view of the path of least resistance, for example, when considering a right turn into one lane rather than waiting for traffic to decrease before crossing two lanes.

[0086] In some implementations, controls 382, ​​384, 386 may be software implemented. In those implementations, a user's preference for one type of view over another may be learned by recording the user's interactions with views on a display device view over time and analyzing the interactions using suitable techniques. Thus, T may be adjusted dynamically based on the learned data collected over time. A user's preference for optimizing the best viewpoint selection process by preferring views with minimum values ​​(lowest number of minTTI objects, i.e., PREF1) or maximum values ​​(highest number of minTTI objects, i.e., PREF 2) may also be learned by observing the viewer's response to the presented views. In other implementations, settings may be adjusted quickly in response to specific situations.

[0087] Figure 4A is an orthographic view looking down at the ground plane of the 3D virtual driving space 212. In one example implementation, this is an orthographic bird's eye view (BEV). The BEV shows the locations of objects within a 360 degree radius of the vehicle 101 as points, e.g., observation objects 412, 413, 418, 414, and may also show the path taken by the objects to reach their current locations, e.g., 430, 432. An orthographic bird's eye view is a view of a scene projected onto a plane, where the projection line may be considered perpendicular to the plane.

[0088] The BEV may be constructed by extracting features from the view provided by the imaging sensor 108 and fusing them with the point cloud from the range sensor. For example, red, green, and blue (RGB) pixel data from the imaging sensor may be fused with the lidar point cloud, and the fused data may be transformed into an orthogonal BEV.

[0089] The aggregated point cloud in the BEV can be evaluated relative to the motion of the vehicle 101 in the real-world driving space, and moving objects (such as vehicles and pedestrians) can generally be described as moving along two orthogonal axes of a coordinate system described, for example, with respect to the vehicle 101: parallel to the vehicle 101, as indicated at 430, or perpendicular to the vehicle (such as at an intersection), as indicated at 432.

[0090] Figure 4B 3 is an illustration of a 3D virtual driving space including a sphere 333 whose surface is divided into faces 402 by a grid. The center 404 of each face is the position (x, y, z) of the virtual camera 214. The processor 310 can adjust the number of faces to provide more or fewer faces 402 by adjusting the grid.

[0091] Figure 4C yes Figure 4A4 shows a perspective view of the ground plane 203, illustrating the construction of the viewpoint vector. The viewpoint vector is a vector that defines the viewpoint of the camera, i.e., a vector that extends from a center position of the face to a position on the ground plane (e.g., a position of an object on the ground plane). From the viewpoint defined by the viewpoint vector 410, an observed object (e.g., 413) on the ground plane 203 is within the FoV of the virtual camera 214 if the dot product of its normal 411 and the viewing direction of the virtual camera 214 is negative. Viewpoints are scored by evaluating the view based on the number of observed objects that meet the temporal criteria within the FoV.

[0092] Figure 4D Regions 406 of the sphere 333 are shown. In some implementations, the processor 310 selects regions of the sphere 333 based on traffic patterns and path layouts in a top view (e.g., at 405 of FIG. 4 ). Region selection based on traffic patterns (which are typically defined by orthogonal paths) constrains the number of faces of the sphere 333 that can provide optimal viewpoints, because the view from the optimal viewpoint will include objects moving along at least one of the paths defined by the traffic patterns. A region 406 includes multiple faces 402. To perform a grid search, the processor 310 positions the virtual camera 214 (invisible) at the center 404 of the face 402 of the region 406. The yaw, pitch, and roll can be determined by constructing a viewpoint vector as described herein. Once the viewpoint is given, the field of view determines what objects (e.g., 101, 413) will be within the field of view FoV of the virtual camera 214.

[0093] Figure 5 is a flow chart of a method 500 for adaptive optimal viewpoint switching. At block 502, the processor 310 determines a virtual camera default viewpoint, for example, using Table II as described above.

[0094] At box 504, the processor 310 retrieves the view from the virtual camera 214 default viewpoint and evaluates the function to determine the extent to which the view meets the criteria. In other words, the function scores the view. The view is scored according to the motion parameters of the objects in the default view. The first example motion parameter is the intercept time (TTI). TTI is defined as the time it takes for an object on the path that intercepts the path of the vehicle 101 to travel from its current position to the interception point, assuming that the speed and path of the object and vehicle 101 do not change. The procedure for calculating the intercept time takes into account the initial positions of the object and vehicle 101 as well as their speed and direction of travel. The intersection point is given by the following expression:

[0095]

[0096]

[0097] The remaining time for the object to reach the interception point with the vehicle 101 is calculated based on equations 1 and 2. The processor 310 calculates the TTI for each object in the default view. Among all the calculated TTIs for the view, the processor 310 identifies the lowest TTI. This is the "minTTI" which is the second motion parameter. The processor 310 scores the default view according to a first criterion, which is a function of the first motion parameter and the second motion parameter, which in this example is the percentage of objects in the view whose TTI=minTTI:

[0098] f minTTI (x, y, z, yaw, pitch, roll) = #minTTI objects / #TTI objects (3)

[0099] Processor 310 may also evaluate the view according to a second objective criterion that is a function of the first motion parameter:

[0100] C (x, y, z, yaw, pitch, roll) = #TTI object (4)

[0101] The scores according to the first criterion and the second criterion may be combined to provide a first score, which is a default view score.

[0102] S = f minTTI (x, y, z, yaw, pitch, roll) + C (x, y, z, yaw, pitch, roll) (5)

[0103] At block 506, processor 310 performs the following operations by constructing Figure 4C The viewpoint vector shown is used to generate an alternative virtual camera viewpoint.

[0104] At block 508, processor 310 retrieves a view from an alternative virtual camera viewpoint and scores the view according to the same objective criteria and motion parameters by which the default view was evaluated. The score according to the first objective criterion is given by:

[0105] f minTTI(x, y, z, yaw, pitch, roll) = #minTTI objects / #TTI objects (6)

[0106] Processor 310 may also evaluate the view according to a second objective criterion:

[0107] C(x, y, z, yaw, pitch, roll) = #TTI object (7)

[0108] Processor 310 may combine the first and second standard scores for the alternative viewpoints to provide a second score, which is an alternative view score:

[0109] S = fminTTI(x, y, z, yaw, pitch, roll) + C(x, y, z, yaw, pitch, roll) (8)

[0110] At block 510, the processor 310 evaluates a viewpoint switching condition comparing the first score to the second score:

[0111] f minTTI (x, y, z, yaw, pitch, roll) + C (x, y, z, yaw, pitch, roll) > f 默认 (...)+T+C(...) (9)

[0112] A viewpoint switching condition specifies a condition as an inequality that must be satisfied before the viewpoint can be switched from the default viewpoint to the alternate viewpoint.

[0113] At block 512, processor 310 determines whether a viewpoint switching condition is satisfied. In implementations where the tuning value T is a threshold, if the second score is greater than the first score by an amount exceeding the threshold T, then the viewpoint switching condition is satisfied.

[0114] If the viewpoint switching condition is satisfied, the processor 310 provides the alternative viewpoint to the display interface 330 to present a view from the alternative viewpoint on the display 230 .

[0115] If the viewpoint switching condition is not met, the processor 310 returns to block 506 and generates another alternative viewpoint. The processor 310 repeats blocks 506 to 512 until the viewpoint switching condition is met at block 512. In this case, at block 514, the processor 310 provides the alternative virtual camera viewpoint to the display interface 330 so that the display device 230 presents a view of the driving environment from the alternative viewpoint.

[0116] As a threshold, the tuning value T affects how many iterations of blocks 506 to 512 will be performed before the viewpoint switching condition is met. For example, if T is set relatively high, the alternative view score must be higher than the default view score by this higher value. In other words, the threshold is set higher for the alternative viewpoint to meet the switching condition. This means that the processor 310 may have to evaluate many alternative viewpoints to find one viewpoint that provides a view with a high enough score to exceed the default view score plus the threshold. This may result in waiting time as the device 300 loops through blocks 506 to 512 to find a viewpoint that provides a view with a high enough score.

[0117] On the other hand, if T is set very low, then the view from the alternative viewpoint does not need to exceed the default view score by very much to satisfy the viewpoint switching condition. However, in this case, the alternative view that satisfies the condition may not be much better than the default view. The user can adjust control 386 to find their preferred balance between a significantly better view with a longer execution time and a shorter execution time with a view that is only slightly better.

[0118] Figure 6 is a flow chart of a method 600 for finding the best alternative to a default viewpoint.

[0119] At block 602, the processor 310 positions the virtual camera 214 at the center point 404 of the first face 402 of the virtual sphere 333. In an example implementation, based on the above description of Figure 4A The traffic patterns observed in the overhead view are used to identify a viewing area 406 where the sphere 333 is well positioned to provide a candidate alternative viewpoint. For example, a viewing area that gives the virtual camera 214 a view of objects moving along a path toward the vehicle 101 is more suitable for searching for alternative viewpoints than a viewing area that does not give the virtual camera 214 any view of the path or only a partial view of the path.

[0120] At block 604, processor 310 constructs a viewpoint vector (eg, Figure 4C 101). In some implementations, the position of the vehicle on the ground plane may be received from the vehicle motion prediction and path planning subsystem 150. In other implementations, a view of the ground plane or another descriptor source may provide object descriptors that give the positions of objects on the ground plane, including the position of the vehicle 101.

[0121] At block 606 , the processor 310 retrieves a view of the virtual driving space 212 by providing an alternative viewpoint to the virtual camera 214 .

[0122] At block 608, processor 310 scores the alternative views based on the values ​​of the motion parameters of the objects in the alternative views:

[0123] #minTTI objects / #TTI objects+#TTI(10)

[0124] At block 610, processor 310 evaluates the viewpoint switching condition based on the alternative view score and the default view score. If the viewpoint switching condition is not met, processor 310 proceeds to block 612 to determine whether the last face of sphere 33 has been evaluated, and if not, returns to block 602 and selects the center 404 of the next face 402 in region 406. Otherwise, method 600 ends.

[0125] If the viewpoint switching condition is met in block 610 , the processor proceeds to block 614 .

[0126] At block 614, the processor 310 constructs a local viewpoint vector around the center of the face, where "local" means on the same face as the viewpoint that satisfies the viewpoint switch condition.

[0127] At block 616, processor 310 generates local candidate replacement viewpoints based on the local candidate replacement viewpoint vectors. The goal is to move the virtual camera around the face, evaluating the scores to find a local maximum, ie, a point where the score peaks and then drops at the next point.

[0128] At block 618 , the processor 310 scores the views from the local candidate alternative viewpoints.

[0129] At block 620, processor 310 determines whether the score is a local maximum. If so, the local candidate alternative viewpoint is identified as the best viewpoint.

[0130] In this case, at block 622, processor 310 provides the best viewpoint to display interface 230, and the method ends. Thereafter, display 230 presents the view from the alternative viewpoint (best viewpoint).

[0131] If the viewpoint switching condition is not met, the processor 310 proceeds to box 612. At box 612, the processor 310 determines whether the face is the last face in the viewing area. If not, the processor 310 returns to box 602 and positions the virtual camera at the center of the next face of the viewing area to generate another alternative viewpoint. The method repeats until the alternative viewpoint meets the viewpoint switching criteria at box 610, or the method reaches the last face at box 612. In this case, the method can end after completing the search for the selected area. This search constrained to a certain area may be useful when the vehicle is traveling along a highway with relatively few intersections or no intersections and the area of ​​interest relative to other objects is constrained to, for example, one side of the vehicle.

[0132] In some implementations, for example, in an urban environment with many intersections and / or complex traffic patterns, a global search (i.e., region-by-region) may be useful where the region of interest relative to other objects includes both sides of the vehicle and may include the rear of the vehicle. In this case, the method may select another face on the viewing area and search based on Figure 7 The illustrated method 700 is repeated.

[0133] In some implementations, the tuning parameter T is a hyperparameter. For example, the value of T can be expressed in seconds. Adjusting T sets a cutoff value for how many seconds in the future the minTTI of an object in a view can be without excluding the view from consideration. As an extreme example, if the minimum interception remaining time (minTTI) for any object in the first view is, for example, 1 hour (3600 seconds), and there are two objects with that minTTI and two other objects in the view, the score can be 0.5 (2 / 4). If the minTTI in the second view is 2 seconds, and one object has a minTTI and the other three objects have a TTI greater than the minTTI, the score for the second view can be (1 / 4), i.e., 0.25. In this case, the first view will score higher than the second view, and therefore the first view will "win" in the search for the highest scoring view. Setting T to 30 seconds (the minTTI in a view cannot be greater than 30 seconds) will exclude the first view from consideration in the scoring function because its minTTI is greater than 30 seconds. The second view will be considered because its minTTI is less than 30 seconds. The higher the time setting, the more views will be considered and the longer it will take to execute the scoring function. This will also tend to produce views with higher object counts. On the other hand, the lower the time setting, the faster the scoring function will execute. This will tend to produce views with lower object counts. Setting hyperparameter values ​​can tune the search to strike a balance based on user preference and / or software choice.

[0134] Figure 7 is a flow chart of a method 700 for identifying an optimal viewpoint.

[0135] At block 702 , the processor 310 models the virtual driving space 212 as a virtual sphere 333 .

[0136] At box 706 , the processor 310 receives a top-down orthographic (or “plan”) view 405 of the 2D ground plane 203 that includes the real-world driving space of the vehicle 101 .

[0137] An example of a suitable top-down orthographic view is a bird's eye view corrected for lens distortion.The top-down plan view may also be provided by a range-finding sensor such as a LiDAR sensor, or by a combination of a range-finding sensor and an imaging sensor.

[0138] At block 708, the processor 310 evaluates the top plan view 405 to identify objects to be observed on the ground plane, referred to herein as "observation objects", such as objects 412, 413, 414, and 418. The observation objects 412, 413, 414, 418 may be vehicles, pedestrians, motorcycles, bicycles, or other objects including the vehicle 101 detected in the raw images captured by the physical sensors (e.g., the imaging and / or ranging sensors 108). Analysis of the images identifies the objects as moving along a path (e.g., path 432) that intersects a path (e.g., path 430 of the vehicle 101) in the real-world driving space of the vehicle 101.

[0139] At block 710, processor 310 positions virtual camera 214 at point 404 of face 402. This defines the position (x, y, z) of virtual camera 214. The virtual camera position only partially defines the viewpoint, since an unconstrained camera at a given position in space has six degrees of freedom of movement, as Figure 4C 10. At position 404 alone, there may be thousands or more candidate alternative viewpoints, depending on the resolution selected. Processor 310 "points" the camera in the direction of the view to be captured.

[0140] Rather than evaluating views from every possible combination of yaw, pitch, and roll at location 404, at block 712, processor 310 constructs viewpoint vector 410 extending from point 404 to the observed object (e.g., object 413), such as FIG. 4A to FIG. 4D The vectors define the yaw, pitch, and roll of the virtual camera 214 . The combination of the position (x, y, z) and the yaw, pitch, and roll specifies a candidate alternative viewpoint for the virtual camera 214 .

[0141] At block 714, processor 310 generates candidate replacement viewpoints based on the viewpoint vector.

[0142] At 716, the processor 310 identifies all observed objects (in Figure 4B ). The processor 310 obtains a count of the observed objects within the FoV. In an example implementation, the count is a second fraction.

[0143] At 718 , the processor 310 calculates the time to intercept (TTI) for each observed object in the FoV.

[0144] At block 720, processor 310 identifies the lowest TTI among the TTIs calculated at block 718. The lowest TTI is referred to herein as "minTTI". Processor 310 determines the percentage of objects in the FoV whose TTI = minTTI. In other words, processor 310 calculates #minTTI objects / #TTI objects. This number is the first candidate alternative viewpoint criterion (f minTTI )'s first score.

[0145] At block 722 , the processor 310 tests a viewpoint switching condition based on the first score and the second score.

[0146] At block 724, the processor 310 determines whether a viewpoint switching condition is satisfied. The determination at block 724 is performed by testing the candidate alternative viewpoints to determine whether the viewpoint switching condition is satisfied. The viewpoint switching condition is given by:

[0147] f minTTI (x, y, z, yaw, pitch, roll) + C (x, y, z, yaw, pitch, roll) > f 默认 (...)+T+C(...) (12)

[0148] where f minTTI (x, y, z, yaw, pitch, roll) is a function that outputs a first score, and C(x, y, z, yaw, pitch, roll) is a function that outputs a second score for the candidate alternative viewpoint, and f 默认 (...)+T+C(...) The default viewpoint is evaluated according to the same functional criteria applied to the candidate alternative viewpoints and also takes into account the tuning value T.

[0149] Since the default viewpoint is known, the processor 310 does not construct a viewpoint vector for the default viewpoint. The processor 310 places the virtual camera 214 at a position on the sphere 333 corresponding to the known default viewpoint. The processor 310 executes blocks 716 to 720, where the virtual camera 214 is located at the default viewpoint to calculate the first score and the second score for the default viewpoint.

[0150] To evaluate, the condition processor 310 determines whether the sum of the first score and the second score of the candidate alternative viewpoint exceeds the sum of the first score and the second score of the default viewpoint by an amount given by T.

[0151] If the viewpoint switching condition is not met, the processor 310 proceeds to block 725. At block 725, the processor 310 determines whether this is the last region to be evaluated. If so, the method ends.

[0152] If not, the processor 310 returns to block 710 and generates another candidate alternative viewpoint by positioning the virtual camera 214 at the center 404 of the center face of the next region. The process repeats until the evaluation at block 724 returns "yes", indicating that the candidate alternative viewpoint satisfies the viewpoint switching condition.

[0153] In this case, the processor 310 proceeds to block 726. At block 726, the processor 310 evaluates local viewpoints, i.e., viewpoints around the center point on the same face. The goal is to find the local maximum score. According to the given criteria of minTTI and C, the position on the face that provides the highest score is the best position to view the observed object.

[0154] At block 728, the highest scoring candidate alternative viewpoint on the facet is selected as the best viewpoint.

[0155] At block 730 , the processor 310 sends the best viewpoint to the display interface 330 , thereby presenting a view from the best viewpoint on the display device.

[0156] Figure 8 800 is a flow chart of a method 800 for alerting a vehicle occupant of a viewpoint update. Some vehicles include a touch screen display and / or a dashboard display and a head-up display (HUD). The ADAS feature may detect the occurrence of an abnormal event that triggers an update of the default viewpoint. In some implementations, the abnormal event is the identification of a best viewpoint that is not the default viewpoint. For example, a driving situation may occur in which the number of detected objects in a path that is not in the default view increases from a number below a threshold to a number above a threshold. The event may trigger a change from the default viewpoint to a viewpoint that gives an updated view, which includes a path with a large number of objects. In this case, the updated view provides situation information to the vehicle occupant. The method 800 alerts the vehicle occupant of the updated view.

[0157] At block 805 , processor 310 detects a viewpoint update.

[0158] At block 810 , the processor 310 retrieves an updated view from the updated viewpoint. At block 820 , the updated view is presented on the ADAS display.

[0159] At box 825, the processor 310 starts a timer. At box 830, the processor 310 receives an indication of a gaze direction of the vehicle occupant. At box 835, the processor 310 evaluates the gaze direction to determine if the gaze is on the ADAS display.

[0160] If not, at block 860, processor 310 checks the timer to determine if the time has expired. If the time has not expired, processor 310 returns to block 830 to recheck the user's gaze, and at 835, determines if the user's gaze is on the display device.

[0161] Blocks 835, 860, and 830 repeat until the timer expiration inquiry at block 860 is affirmative, or the processor 310 determines at block 835 that the vehicle occupant has directed their gaze toward the ADAS display, whichever comes first. If at block 860, the user has not directed their gaze toward the ADAS display by the time the timer expires, the processor 310 proceeds to block 865. At block 865, the processor 310 displays the updated view on a heads-up display (HUD) where the vehicle occupant will more easily notice it.

[0162] At block 837, a determination is made as to whether the event that triggered the updated viewpoint has expired. If not, the method waits at block 837 until the viewpoint expires. If it has, the processor 310 proceeds to block 815. At block 815, the processor 310 changes the ADAS viewpoint to its last default viewpoint. At block 850, a view from the default viewpoint is retrieved. At block 855, the view is presented, and the method returns to 805 and waits until another viewpoint update is detected.

[0163] Computing devices such as those discussed herein typically each include instructions that can be executed by one or more computing devices (such as those identified above) and used to implement the blocks or steps of the processes described above. For example, the process blocks discussed above may be embodied as computer-executable instructions.

[0164] The computer executable instructions may be compiled or interpreted by a computer program created using a variety of programming languages ​​and / or technologies, including, but not limited to, Java, TM , C, C++, Visual Basic, Java Script, Perl, HTML, etc. In general, a processor (e.g., a microprocessor) receives instructions, such as from a memory, a computer-readable medium, etc., and executes these instructions to perform one or more processes, including one or more of the processes described herein. Such instructions and other data can be stored and transmitted using a variety of computer-readable media. Files in a computing device are typically a collection of data stored on a computer-readable medium such as a storage medium, a random access memory, etc.

[0165] Computer-readable media include any media that participate in providing data (e.g., instructions) that can be read by a computer. Such media can take many forms, including but not limited to non-volatile media, volatile media, etc. Non-volatile media include, for example, optical or magnetic disks and other persistent memories. Volatile media include dynamic random access memory (DRAM) that usually constitutes main memory. Common forms of computer-readable media include, for example, floppy disks, floppy disks, hard disks, tapes, any other magnetic media, CD-ROMs, DVDs, any other optical media, punch cards, paper tapes, any other physical media with hole patterns, RAMs, PROMs, EPROMs, flash EEPROMs, any other memory chips or cassettes, or any other media from which a computer can read.

[0166] In the accompanying drawings, the same reference numerals indicate the same elements. In addition, some or all of these elements may be changed. With respect to the media, processes, systems, methods, etc. described herein, it should be understood that, although the steps of such processes, etc. have been described as occurring in a certain ordered sequence, such processes may be practiced by performing the steps described in an order different from the order described herein. It should also be understood that certain steps may be performed simultaneously, other steps may be added, or certain steps described herein may be omitted. In other words, the description of the process herein is provided for the purpose of illustrating certain embodiments and should never be interpreted as limiting the claimed invention.

[0167] Therefore, it should be understood that the above description is intended to be illustrative rather than restrictive. After reading the above description, many embodiments and applications other than the examples provided will be apparent to those skilled in the art. The scope of the present invention should not be determined with reference to the above description, but with reference to the appended claims together with the full scope of equivalents to which these claims are assigned. It is contemplated and expected that future developments will occur in the technology discussed herein, and the disclosed systems and methods will be incorporated into such future embodiments. In short, it should be understood that the present invention is capable of modification and variation and is limited only by the appended claims.

[0168] All terms used in the claims are intended to be given their broadest reasonable constructions and their ordinary meanings as understood by those skilled in the art unless an explicit indication to the contrary is made herein.

Claims

1. A method, comprising: presenting, on a display device of the vehicle, a default view of the 3D virtual driving space from a default viewpoint of the virtual camera; Searching the 3D virtual driving space to obtain candidate alternative viewpoints; selecting a candidate alternative viewpoint; as well as changing the viewpoint of the virtual camera from the default viewpoint to a selected candidate alternative viewpoint; as well as The view of the 3D virtual driving space from the selected candidate alternative viewpoint is provided to a display interface, thereby presenting the view of the 3D virtual driving space from the selected candidate alternative viewpoint on the display device.

2. The method of claim 1, comprising: scoring views from the candidate alternative viewpoints to provide scored views; as well as One of the candidate alternative viewpoints is selected based on the scored views.

3. The method of claim 1, comprising: identifying, from the selected candidate alternative viewpoint, an observed object including the vehicle on a ground plane of a bird's eye view (BEV) of the 3D virtual driving space within a field of view (FoV) of the virtual camera; as well as The selected candidate replacement viewpoints are tested based on motion parameters of the observed object within the FoV according to a viewpoint switching condition.

4. The method of claim 1, comprising: obtaining a first count of first observed objects on a ground plane of a bird's eye view (BEV) of the 3D virtual driving space within a field of view of the virtual camera from the selected candidate alternative viewpoint; obtaining, from the selected candidate alternative viewpoint, a second count of second observed objects on the ground plane of the 3D virtual driving space within a field of view (FoV) of the virtual camera; as well as The selected candidate replacement viewpoint is tested according to a viewpoint switching condition by comparing the first count with the second count.

5. The method of claim 1, further comprising: Receiving real-time motion data and a planned path of the vehicle; as well as The default viewpoint is identified based on the real-time motion data and a planned path of the vehicle.

6. The method of claim 1, comprising: identifying an observation object including the vehicle on a ground plane of a bird's-eye view (BEV) of the 3D virtual driving space; Assigning motion parameter values ​​to the observed object; receiving, via a user interface, a first path view preference; selecting a first viewing area based on the first path view preference and the motion parameter value; as well as The candidate alternative viewpoints are generated from the first viewing area.

7. The method of claim 1, comprising: generating the candidate alternative viewpoint from a first viewing area by constructing a viewpoint vector from a center of a first face of the first viewing area to the vehicle on a ground plane; as well as Observation objects within the FoV of the virtual camera are identified from the selected candidate alternative viewpoints.

8. The method of claim 1, comprising: switching according to a viewpoint switching condition to test the selected candidate alternative viewpoint based on motion parameters of an observed object on a ground plane of a bird's eye view (BEV) of the 3D virtual driving space within the FoV of the virtual camera as seen from the selected candidate alternative viewpoint; Based on the testing, generating a first local candidate alternative viewpoint to identify a first local maximum score; and A local candidate replacement viewpoint is selected based on the first local maximum score.

9. The method of claim 1, comprising: Test the selected candidate alternative viewpoint based on the viewpoint switching condition' receiving an adjustment to the viewpoint switching condition; retesting the selected candidate alternative viewpoint based on the adjustment; and Based on the retesting, the viewpoint of the virtual camera is changed from the default viewpoint to the selected candidate replacement viewpoint.

10. The method of claim 1, comprising: identifying an observation object within the FoV of the virtual camera from a viewpoint on a second side of the first viewing area; as well as The viewpoints are scored based on the observed objects within the FoV of the virtual camera.

11. A computing device programmed to perform the method of any one of claims 1 to 10.

12. A vehicle comprising the computing device of claim 11.

Citation Information

Cited By

  • Transparent vehicle body system of large mining vehicle and large mining vehicle

    CN122354351A