Vehicle and control unit as well as methods for operating the vehicle
The control device with augmented reality and sensor technology enables precise autonomous vehicle parking by capturing and processing three-dimensional maps for accurate trajectory planning and execution, addressing the inefficiencies in existing systems.
Patent Information
- Application Number
- DE102022105155
- Authority / Receiving Office
- DE · DE
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2022-03-04
- Publication Date
- 2026-01-15
- Estimated Expiration
- 2042-03-04
AI Technical Summary
Existing autonomous vehicle parking systems lack accurate and efficient methods for determining and executing parking maneuvers using augmented reality, particularly in complex environments.
A control device with augmented reality capabilities that utilizes sensors, including cameras, to capture and process three-dimensional environmental maps, allowing for precise user input and trajectory planning, enabling accurate control of the vehicle's movement to a target position or along a planned path.
Enhances the accuracy and efficiency of autonomous vehicle parking by providing a detailed three-dimensional representation of the environment, allowing for precise user input and control, thereby improving the vehicle's navigation and maneuvering capabilities.
Smart Images

Figure 00000000_0000_ABST
Abstract
Description
[0001] The invention relates to a vehicle and a control device as well as a method for operating the vehicle.
[0002] DE 10 2018 221 186 A1 relates to a method for autonomous parking. In this method, a communication link is first established between a motor vehicle and a mobile device. An autonomous maneuver is then determined to move the vehicle from its current position to a target parking position. The communication link receives an initial image signal captured by the mobile device's camera and / or a map of the vehicle's position, calculated by the mobile device based on this initial image signal. The calculated map is a three-dimensional representation of the vehicle's surroundings. The autonomous parking maneuver is then adjusted based on the received initial image signal and / or the received map.The three-dimensional representation is determined based on the first image signal and at least one second image signal recorded by a camera of the motor vehicle.
[0003] US 2021 0 311 472 A1 also concerns a procedure for autonomous parking.
[0004] US 2019 / 0220002 A1 reveals aspects of an operating concept.
[0005] The methods, the control unit and the vehicle according to the independent claims result in an improved operating concept.
[0006] A method for operating an autonomous vehicle in an augmented reality control device provides that a representation of the environment of the control device is acquired with at least one sensor of the control device, a representation of the environment of the vehicle acquired with at least one sensor of the vehicle is provided, three-dimensional coordinates of a three-dimensional map representation as a mesh of the environment of the autonomous vehicle are determined depending on the representation of the environment of the control device and the representation of the environment of the vehicle, in particular with a collaborative, simultaneous position determination and mapping by the control device and the autonomous vehicle, on the basis of which, with a semantic segmentation of the three-dimensional coordinates, areas traversable by the vehicle are recognizable.A two-dimensional representation of the autonomous vehicle's environment is displayed on a control device's display, and either a user input is recognized which, in the two-dimensional representation, is assigned to a point representing a target position for the autonomous vehicle, wherein the point from the two-dimensional representation is mapped to one of the three-dimensional coordinates of the map representation, and wherein the three-dimensional coordinate is sent to the vehicle for control purposes, in particular via a communication module of the control device, or a user input is recognized which, in the two-dimensional representation, is assigned to a set of points representing a trajectory along which the autonomous vehicle is to move, wherein at least one point from the set in the two-dimensional representation is mapped to one of the three-dimensional coordinates of the map representation.and wherein the three-dimensional coordinate for controlling the vehicle is sent to the vehicle, in particular via a communication module of the control unit. According to the invention, the at least one sensor of the control unit comprises a camera, wherein a digital image is captured with the camera, and wherein the representation of the environment of the control unit is determined depending on the digital image. The pixels of the digital image represent the environment of the control unit particularly well.
[0007] Preferably, three-dimensional coordinates of a trajectory planned by the vehicle are received, in particular by the vehicle, wherein the three-dimensional coordinates are mapped to points of the two-dimensional representation of the vehicle's environment that represent the trajectory, wherein the two-dimensional representation is displayed on the display of the control device, wherein the trajectory is superimposed on the two-dimensional representation on the display of the control device.
[0008] Preferably, the at least one sensor of the control device is configured to detect acceleration and / or velocity and / or position of the control device, whereby the position of the control device in the three-dimensional map representation and / or the three-dimensional coordinates of the three-dimensional map representation are determined depending on the acceleration and / or velocity and / or position of the control device. This improves the accuracy of the assignment of pixels of the digital image to the three-dimensional coordinates of the map representation.
[0009] A method for operating an autonomous vehicle from within the vehicle provides that a representation of the environment of the control unit, acquired by at least one sensor of a control device for the vehicle, is provided; a representation of the environment of the vehicle is acquired by at least one sensor of the vehicle; three-dimensional coordinates of a three-dimensional map representation are determined as a mesh of the environment of the autonomous vehicle depending on the representation of the environment of the control device and the representation of the environment of the vehicle, in particular by collaborative, simultaneous position determination and mapping by the control device and the autonomous vehicle; areas traversable by the vehicle are recognized by semantic segmentation of the three-dimensional coordinates; one of the three-dimensional coordinates is received, in particular by the control device, which is assigned to a point.which represents a target position for the autonomous vehicle on a display of the control unit in a two-dimensional representation of an environment of the autonomous vehicle, wherein, depending on the three-dimensional coordinate and the drivable surfaces, a trajectory for controlling the vehicle to reach the target position is determined, in particular in a control unit of the vehicle, and a motor system of the vehicle is controlled, depending on the trajectory, to move the vehicle on the trajectory to reach the target position, or at least one of the three-dimensional coordinates is received, in particular by the control unit, which is assigned to a point of a set of points that represent a trajectory for the autonomous vehicle on a display of the control unit in a two-dimensional representation of an environment of the autonomous vehicle.wherein, depending on the three-dimensional coordinates and the traversable surfaces, a trajectory for controlling the vehicle to reach the target position is determined, particularly in a control unit of the vehicle, and a motor system of the vehicle is controlled, depending on the trajectory, to move the vehicle along the trajectory to reach the target position. According to the invention, the at least one sensor of the vehicle comprises a camera, wherein a digital image is captured by the camera, and wherein the representation of the vehicle's surroundings is determined depending on the digital image. The pixels of the digital image represent the vehicle's surroundings particularly well.
[0010] Preferably, three-dimensional coordinates of the trajectory planned by the vehicle are transmitted to the control unit, particularly via a communication module in the vehicle. This allows the planned route to be superimposed on the display in the control unit.
[0011] Preferably, the at least one sensor of the vehicle is configured to detect acceleration and / or velocity and / or position of the vehicle, whereby the vehicle's position in the three-dimensional map representation and / or the three-dimensional coordinates of the three-dimensional map representation are determined depending on the vehicle's acceleration and / or velocity and / or position. This improves the accuracy of the mapping of pixels of the digital image to the three-dimensional coordinates of the map representation.
[0012] A control device for operating an autonomous vehicle with augmented reality comprises a display, at least one sensor and a communication module, wherein the display is configured to show a two-dimensional representation of an environment of the vehicle, wherein the at least one sensor is configured to capture a two-dimensional representation of an environment of the control device, wherein the communication module is configured to communicate with a corresponding communication module of the vehicle, and wherein the display, the at least one sensor and the communication module are configured to perform the steps in the method.
[0013] The at least one sensor comprises a camera. The at least one sensor is preferably configured to detect acceleration and / or velocity and / or position of the control device.
[0014] A vehicle that can be controlled by an augmented reality control device comprises a control unit, a motor system, at least one sensor and a communication module, wherein the at least one sensor is configured to capture a two-dimensional representation of the vehicle's environment, wherein the control unit, the communication module and the at least one sensor are configured to perform the steps in the method for determining a trajectory, wherein the control unit is configured to control the motor system to move the vehicle along the trajectory.
[0015] The at least one sensor comprises a camera. The at least one sensor is preferably configured to detect acceleration and / or velocity and / or position of the vehicle.
[0016] Further advantageous embodiments can be found in the following description and the drawing. The drawing shows: Fig. 1 a schematic representation of a vehicle and a control device for it, Fig. 2 a sequence diagram of a procedure for operating the vehicle.
[0017] In Fig. 1 is a vehicle 100 and a control unit 102 for it, shown schematically.
[0018] Vehicle 100 can be controlled via the control unit 102 using augmented reality. Augmented reality means, for example, that a digital image captured by a camera is overlaid with calculated additional information, or that user input, with which the user selects one or more pixels of the digital image, is captured. The digital image preferably represents a scenario that is taking place in the real world at the moment the user makes the selection and / or enters the user input. Instead of representing the scenario through the captured digital image itself, another two-dimensional representation of the scenario can also be provided, in particular a perspective view or top view of the scenario calculated from different digital images representing the same scenario at essentially the same time.User input is enabled, for example, by determining a pixel of the digital image as a point in response to user input, whereby the pixel selected by a user by tapping the digital image on a touch-sensitive display while the digital image is being displayed is recognized.
[0019] The vehicle 100 comprises a control unit 104, a motor 106, at least one camera 108, at least one sensor 110 designed to detect an acceleration of the vehicle 100, at least one sensor 112 designed to detect a speed of the vehicle 100, at least one sensor 114 designed to detect a position of the vehicle 100.
[0020] The camera 108 is designed to capture a digital image of the vehicle 100's surroundings. This means that the vehicle 100 includes at least one sensor designed to capture a two-dimensional representation of the vehicle 100's surroundings.
[0021] The control unit 104 is designed to determine a representation of the vehicle's environment based on the digital image of the vehicle's surroundings.
[0022] The vehicle 100 includes a communication module 116, which is designed to communicate with the control unit 102.
[0023] The control unit 104, the communication module 116 and the at least one sensor are configured to perform steps in a procedure described below for determining a trajectory on which the vehicle 100 is to move, in particular autonomously.
[0024] The control unit 104 is designed to control the motor system 106 to move the vehicle 100, in particular autonomously, along the trajectory.
[0025] The control unit 102 for operating the autonomous vehicle 100 with augmented reality comprises a display 118, at least one camera 120, at least one sensor 122 for detecting an acceleration of the control unit 102, at least one sensor 124 for detecting a speed of the control unit 102 and at least one sensor 126 for detecting a position of the control unit 102.
[0026] The camera 120 is designed to capture a digital image of the environment of the control device 102. This means that the control device 102 includes at least one sensor designed to capture a two-dimensional representation of the environment of the control device 102.
[0027] The control unit 102 includes a communication module 128, which is designed to communicate with the vehicle 100.
[0028] Display 118 is designed to show a two-dimensional representation of the environment surrounding vehicle 100. In this example, vehicle 100 itself is also shown in display 118. It is also possible to display a view from inside vehicle 100 of its surroundings.
[0029] The control unit 104 is designed to communicate with the motor unit 106, the sensors 108, 110, 112, 114 and the communication module 116 via a data connection 134.
[0030] The display 118, the at least one sensor and the communication module 116 are configured to perform steps in the procedure described below.
[0031] Display 118 is designed in an example to recognize a user input which, in the two-dimensional representation, is assigned to a point 130 that represents a target position for the autonomous vehicle 100.
[0032] The display 118 is designed in an example to recognize a user input which is assigned in the two-dimensional representation of a set of points 132 that represent a trajectory on which the autonomous vehicle 100 is to move.
[0033] In this example, the control unit 102 and the vehicle 100 are configured to determine three-dimensional coordinates of a three-dimensional map representation of the environment of the autonomous vehicle 100.
[0034] The three-dimensional map representation comprises a network through which the three-dimensional coordinates are assigned to each other.
[0035] In the example, the control device 102 and the vehicle 100 are configured to determine the three-dimensional map representation depending on the representation of the environment of the control device 102 and the representation of the environment of the vehicle 100 with a collaborative, simultaneous position determination and mapping.
[0036] The control unit 102 includes at least one processor 138.
[0037] Processor 138, for example, is configured to map point 130 from the two-dimensional representation to one of the three-dimensional coordinates of the map representation. Communication module 128, for example, is configured to send the three-dimensional coordinate for controlling vehicle 100 to vehicle 100.
[0038] The processor 138, for example, is designed to map at least one point from the set 132 from the two-dimensional representation to one of the three-dimensional coordinates of the map representation.
[0039] The communication module 128, for example, is designed to send the three-dimensional coordinate for controlling vehicle 100 to vehicle 100.
[0040] In this example, communication module 128 is configured to communicate with the corresponding communication module 116 of vehicle 100 via a communication link 136. Processor 138, in this example, is configured to communicate with sensors 120, 122, 124, 126 and communication module 128 via a data link 140.
[0041] Fig. Figure 2 shows a sequence diagram of an exemplary implementation of the procedure.
[0042] In this example, the procedure for operating the autonomous vehicle 100 takes place partly in the control unit 102 with augmented reality and partly in the vehicle 100.
[0043] In step 200, a representation of an environment of the control device 102 is captured with at least one sensor 120, 122, 124, 126 of the control device 102.
[0044] In step 202, a representation of the vehicle 100's environment is captured using at least one sensor 108, 110, 112, 114 of the vehicle 100. For example, a digital image is captured with camera 120.
[0045] It may be provided that an acceleration and / or a speed and / or a position of the control device 102 is detected.
[0046] In step 204, the representation of the environment of the vehicle 100 is provided to the control device 102 and / or the representation of the environment of the control device 102 is provided to the vehicle 102.
[0047] In this example, these steps are repeatedly performed as part of a collaborative, simultaneous positioning and mapping process 206 by the control device 102 and the autonomous vehicle 100. For example, the representation of the environment of the control device 102 is determined based on the digital image. It may be possible to determine the position of the control device 102 in the three-dimensional map representation and / or the three-dimensional coordinates of the three-dimensional map representation based on the acceleration and / or velocity and / or orientation of the control device 102.
[0048] It may be provided that the acceleration and / or speed and / or position of vehicle 100 is recorded. It may be provided that the position of vehicle 100 in the three-dimensional map representation and / or the three-dimensional coordinates of the three-dimensional map representation are determined depending on the acceleration and / or speed and / or position of vehicle 100.
[0049] In this method, the three-dimensional coordinates of the three-dimensional map representation of the environment of the autonomous vehicle 100 are determined depending on the representation of the environment of the control device 102 and the representation of the environment of the vehicle 100. It may be provided that the three-dimensional coordinates are determined depending on the acceleration and / or the velocity and / or the attitude and / or the position of the vehicle 100, in particular at a time when data has been acquired by at least one sensor, which is used for determining the representation of the environment of the control device 102.It may be provided that the three-dimensional coordinates are determined depending on the acceleration and / or the velocity and / or the position and / or the location of the control device 102, in particular at a time when the data used to determine the representation of the environment of the control device 102 have been acquired by the at least one sensor.
[0050] In step 208, based on the three-dimensional coordinates or the three-dimensional map representation, a semantic segmentation of the three-dimensional coordinates is used to determine 100 drivable areas for the vehicle.
[0051] In step 210, a two-dimensional representation of the environment of the autonomous vehicle 100 is displayed on the display 118 of the control unit 102. This means that the two-dimensional representation is displayed on the display 118 of the control unit 102.
[0052] In step 212, user input is recognized.
[0053] In a first example, the user input is recognized, which in the two-dimensional representation is assigned to point 130, representing the target position for the autonomous vehicle 100.
[0054] In a second example, the user input is recognized, which is assigned in the two-dimensional representation to a set of points 132 that represent the trajectory on which the autonomous vehicle 100 is to move.
[0055] Then step 214 is executed.
[0056] In step 214, in the first example, point 130 is mapped from the two-dimensional representation to one of the three-dimensional coordinates of the map representation. Raycasting is used for this purpose, for example.
[0057] In step 214, in the second example, at least one point from the set 132 is mapped from the two-dimensional representation to one of the three-dimensional coordinates of the map representation.
[0058] Then step 216 is executed.
[0059] In step 216, at least one three-dimensional coordinate is sent to vehicle 100 for control purposes.
[0060] In the first example, the three-dimensional coordinate is sent which is assigned to point 130, which on a display of the control device 102 in the two-dimensional representation of the environment of the autonomous vehicle 100 represents the target position for the autonomous vehicle 100.
[0061] In the second example, at least one of the three-dimensional coordinates assigned to one of the points from the set of 132 is sent. The second example may involve determining the three-dimensional coordinates of several or all points from the set of 132 and sending them to vehicle 100 for navigation purposes. This means that the trajectory requested by the user is transmitted partially or completely.
[0062] Then step 218 is executed.
[0063] In step 218, the trajectory for controlling the vehicle 100 to reach the target position is determined depending on the three-dimensional coordinate and the drivable surfaces, in particular in the control unit 104 of the vehicle 100.
[0064] In the second example, it can be provided that the trajectory is determined depending on the transmitted three-dimensional coordinates of the trajectory desired by the user and the drivable areas.
[0065] In step 220, three-dimensional coordinates of the trajectory planned by vehicle 100 to reach the target point are transmitted from vehicle 100 to the control unit 102. In this example, three-dimensional coordinates of the trajectory planned by vehicle 100 are sent to the control unit 102, specifically using the communication module 116 of vehicle 100.
[0066] In step 222, the three-dimensional coordinates are mapped to points of the two-dimensional representation of the environment of the vehicle 100, which represent the trajectory.
[0067] In step 224, the trajectory of the two-dimensional representation is superimposed on the display 118 of the control unit 102.
[0068] In step 226, the motor 106 of the vehicle 100 is controlled depending on the trajectory to move the vehicle 100 along the trajectory to reach the target position.
Claims
[1] Method for operating an autonomous vehicle (100) in an augmented reality control device (102), wherein a representation of an environment of the control device (102) is captured with at least one sensor (120, 122, 124, 126) of the control device (102) (200), a representation of the vehicle's environment (100) is provided (204) by means of at least one sensor (108, 110, 112, 114) of the vehicle (100), three-dimensional coordinates of a three-dimensional map representation as a network of the environment of the autonomous vehicle (100) depending on the representation of the environment of the control device (102) and the representation of the environment of the vehicle (100) in particular with a collaborative, simultaneous position determination and mapping by the control device (102) and the autonomous vehicle (100) is determined (206), on the basis of which, with a semantic segmentation of the three-dimensional coordinates, areas traversable by the vehicle (100) are recognizable (208), a two-dimensional representation of the environment of the autonomous vehicle (100) is displayed on a display (118) of the control device (102) (210), and either a user input is recognized (212) which is assigned in the two-dimensional representation to a point (130) representing a target position for the autonomous vehicle (100), wherein the point (130) from the two-dimensional representation is mapped onto one of the three-dimensional coordinates of the map representation (214), and wherein the three-dimensional coordinate is sent to the vehicle (100) for control purposes, in particular by means of a communication module (128) of the control device (102) (216), or a user input is recognized (212) which is assigned in the two-dimensional representation to a set of points (132) representing a trajectory along which the autonomous vehicle (100) is to move,wherein at least one point from the set (132) from the two-dimensional representation is mapped onto one of the three-dimensional coordinates of the map representation (214), and wherein the three-dimensional coordinate for controlling the vehicle (100) is sent to the vehicle (100) in particular by means of a communication module (128) of the control device (102) (216), wherein the at least one sensor of the control device (102) comprises a camera (120), wherein a digital image is captured by the camera (120) (200), and wherein the representation of the environment of the control device (102) is determined depending on the digital image (206). [2] Method according to claim 1, characterized by, that three-dimensional coordinates of a trajectory planned by the vehicle (100) are received in particular by the vehicle (100) (220), wherein the three-dimensional coordinates are mapped to points of the two-dimensional representation of the environment of the vehicle (100) (222) that represent the trajectory, wherein the two-dimensional representation is displayed on the display (118) of the control device (102) (210), wherein the trajectory of the two-dimensional representation is superimposed on the display (118) of the control device (102) (224). [3] Method according to any one of claims 1 to 2, characterized by, that the at least one sensor (122, 124, 126) of the control device (102) is configured to detect an acceleration and / or a velocity and / or a position of the control device (102), wherein a position of the control device (102) in the three-dimensional map representation and / or the three-dimensional coordinates of the three-dimensional map representation are determined depending on the acceleration and / or the velocity and / or the position of the control device (102) (206). [4] Method for operating an autonomous vehicle (100) in the vehicle (100), wherein a representation (200) of an environment of the control unit (104) is provided (204) acquired by at least one sensor (120, 122, 124, 126) of a control unit (102) for the vehicle (100), a representation of an environment of the vehicle (100) is acquired by at least one sensor (108, 110, 112, 114) of the vehicle (100) (202), and three-dimensional coordinates of a three-dimensional map representation as a mesh of the environment of the autonomous vehicle (100) are determined depending on the representation of the environment of the control unit (102) and the representation of the environment of the vehicle (100), in particular by collaborative, simultaneous position determination and mapping by the control unit (102) and the autonomous vehicle (100). (206), areas traversable by the vehicle (100) can be identified using a semantic segmentation of the three-dimensional coordinates (208),one of the three-dimensional coordinates is received, in particular by the control device (102) (216), which is assigned to a point (130) that represents a target position for the autonomous vehicle (100) on a display of the control device (102) in a two-dimensional representation of an environment of the autonomous vehicle (100), wherein, depending on the three-dimensional coordinate and the drivable surfaces, a trajectory for controlling the vehicle (100) to reach the target position is determined, in particular in a control unit (104) of the vehicle (100) (218), and a motor system (106) of the vehicle (100) is controlled, depending on the trajectory, to move the vehicle (100) on the trajectory to reach the target position (226), or at least one of the three-dimensional coordinates is received, in particular by the control device (102) (216), which is assigned to a point of a set of points (132),which on a display of the control unit (102) in a two-dimensional representation of an environment of the autonomous vehicle (100) represent a trajectory for the autonomous vehicle (100), wherein, depending on the three-dimensional coordinate and the drivable surfaces, a trajectory for controlling the vehicle (100) to reach the target position is determined, in particular in a control unit (104) of the vehicle (100) (218), and a motor system (106) of the vehicle (100) is controlled, depending on the trajectory, to move the vehicle (100) on the trajectory to reach the target position (226), wherein the at least one sensor of the vehicle (100) comprises a camera (108), wherein a digital image is captured with the camera (108) (202), and wherein the representation of the environment of the vehicle (100) is determined depending on the digital image (206). [5] Method according to claim 4, characterized by, that three-dimensional coordinates of the trajectory planned by the vehicle (100) are sent to the control unit (102) in particular by means of a communication module (116) of the vehicle (100) (220). [6] Method according to any one of claims 4 to 5, characterized by , that the at least one sensor (110, 112, 114) of the vehicle (100) is configured to detect an acceleration and / or a velocity and / or a position of the vehicle (100) (202), wherein a position of the vehicle (100) in the three-dimensional map representation and / or the three-dimensional coordinates of the three-dimensional map representation are determined depending on the acceleration and / or the velocity and / or the position of the vehicle (100) (206). [7] Control device (102) for operating an autonomous vehicle (100) with augmented reality, characterized by, that the control device (102) comprises a display (118), at least one sensor (120, 122, 124, 126) and a communication module (128), wherein the display is configured to show a two-dimensional representation of an environment of the vehicle (100), wherein the at least one sensor is configured to detect a two-dimensional representation of an environment of the control device (102) and comprises a camera (120), wherein the communication module is configured to communicate with a corresponding communication module of the vehicle (100), wherein the display, the at least one sensor and the communication module are configured to perform the steps in the method according to one of claims 1 to 3. [8] Control device (102) according to claim 7, characterized by , that a sensor (122, 124, 126) of the control device (102) is configured to detect an acceleration and / or a velocity and / or a position of the control device (102). [9] Vehicle (100) that can be controlled by an augmented reality control device (102), characterized by , that the vehicle (100) comprises a control unit (104), a motor (106), at least one sensor (108, 110, 112, 114) and a communication module (116), wherein the at least one sensor (108, 110, 112, 114) is configured to capture a two-dimensional representation of the vehicle's (100's) environment and comprises a camera (108), wherein the control unit (104), the communication module (116), and the at least one sensor (108, 110, 112, 114) are configured to perform the steps in the method according to one of claims 4 to 6 for determining a trajectory, wherein the control unit (104) is configured to control the motor (106) to move the vehicle (100) along the trajectory. [10] Vehicle (100) according to claim 9, characterized in that the at least one sensor (110, 112, 114) of the vehicle (100) is configured to detect an acceleration and / or a speed and / or a position of the vehicle (100).
Citation Information
Patent Citations
Autonomous parking procedures
DE102018221186A1
Monitoring and Planning a Movement of a Transportation Device
US20210311472A1