Vehicle control devices
The vehicle control device accurately determines the AR glasses' position using SLAM technology to separate interior and exterior feature points, improving positioning accuracy and content superimposition on AR glasses.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- TOYOTA JIDOSHA KK
- Filing Date
- 2023-03-03
- Publication Date
- 2026-05-19
AI Technical Summary
Existing AR glasses systems struggle with inaccurate positioning due to feature points from both inside and outside the vehicle cabin being mixed, leading to insufficient accuracy in determining the relative position between the AR glasses and external objects.
A vehicle control device that includes an extraction unit to identify feature points, a determination unit to distinguish between interior and exterior regions based on movement vector discontinuity, and an identification unit to accurately determine the AR glasses' position using SLAM technology.
Enables precise identification of the AR glasses' position and superimposition of content at desired external positions, enhancing accuracy and reducing development costs.
Smart Images
Figure 0007861665000001 
Figure 0007861665000002 
Figure 0007861665000003
Abstract
Description
Technical Field
[0001] This disclosure relates to a vehicle control device.
Background Art
[0002] Patent Document 1 discloses a wearable terminal (AR (Augmented Reality) glasses) that is worn on the head of a passenger and includes a display unit that is disposed in front of the passenger's eyes in the worn state.
Prior Art Documents
Patent Documents
[0003]
Patent Document 1
Summary of the Invention
Problems to be Solved by the Invention
[0004] In order to display content on the display unit of the AR glasses so that content such as an arrow is superimposed on an arbitrary position of the scenery outside the vehicle cabin that the user views through the AR glasses, it is necessary to specify the position of the AR glasses (the relative position between the object outside the vehicle cabin and the AR glasses). Here, when a sensor is provided on the AR glasses so that an object outside the vehicle cabin enters the detection range of the sensor, it is possible in principle to extract feature points from the detection result by the sensor and specify the position of the AR glasses from the information of the extracted feature points.
[0005] However, the feature points extracted from the detection result by the sensor include a first feature point corresponding to an object located inside the vehicle cabin (for example, a pillar of the vehicle) and a second feature point corresponding to an object located outside the vehicle cabin (for example, a pedestrian), and they are mixed. Therefore, if the position of the AR glasses is specified using all the information of the feature points extracted from the detection result by the sensor, there is a problem that the accuracy of specifying the position of the AR glasses is extremely insufficient.
[0006] This disclosure is made in consideration of the above facts, and aims to provide a vehicle control device that can accurately determine the position of AR glasses using sensors provided on the AR glasses. [Means for solving the problem]
[0007] A vehicle control device according to the first embodiment includes: an extraction unit that extracts a plurality of feature points from within the detection range of a sensor based on the detection results of a sensor provided on the AR glasses; a determination unit that determines the boundary between a first region corresponding to the interior of the vehicle and a second region corresponding to the exterior of the vehicle within the detection range based on the discontinuity of the movement vectors of the plurality of feature points when the vehicle is moving; and a identification unit that identifies the position of the AR glasses based on the determination result of the determination unit. fruit, The determination unit determines that, among the plurality of feature points, the first feature point whose direction and magnitude of the movement vector do not correspond to the movement of the vehicle, and which is the first feature point that is the shortest distance from the second feature point whose direction and magnitude of the movement vector correspond to the movement of the vehicle, is the feature point on the boundary. .
[0008] In the first embodiment, the multiple feature points extracted by the extraction unit from within the detection range of the sensor provided in the AR glasses include a mixture of first feature points corresponding to objects located inside the vehicle and second feature points corresponding to objects located outside the vehicle. However, the movement vector of the first feature point corresponds to the direction and magnitude of the user wearing the AR glasses, while the movement vector of the second feature point corresponds to the direction and magnitude of the vehicle's movement when the vehicle is moving. Therefore, when the vehicle is moving, the movement vectors of the feature points become discontinuous (at least one of the direction and magnitude of the movement vector changes clearly) across the boundary between the area corresponding to the inside of the vehicle and the area corresponding to the outside of the vehicle within the sensor's detection range.
[0009] The first embodiment utilizes this, in which the determination unit determines the boundary between a first region corresponding to the interior of the vehicle and a second region corresponding to the exterior of the vehicle, based on the discontinuity of the movement vectors of multiple feature points during vehicle movement. The identification unit then identifies the position of the AR glasses based on the determination result by the determination unit. This makes it possible for the identification unit to identify the position of the AR glasses separately for the first and second feature points. Therefore, according to the first embodiment, it is possible to accurately identify the position of the AR glasses using sensors provided on the AR glasses.
[0010] Furthermore, in the first embodiment, multiple feature points are discriminated into a first feature point or a second feature point based on whether the direction and magnitude of the movement vector correspond to the movement of the vehicle, and the feature point among the first feature points that has the smallest distance from the second feature point is determined to be a feature point on the boundary. This makes it possible to accurately determine the boundary between the first region corresponding to the interior of the vehicle and the second region corresponding to the exterior of the vehicle with a simple process.
[0012] The 2 In the first embodiment, the identifying unit determines the relative position between the vehicle and the AR glasses from information of a first feature point located within the first region within the detection range, and determines the relative position between the vehicle and an object outside the vehicle from information of a second feature point located within the second region within the detection range.
[0013] The 2 In this embodiment, the position of the AR glasses (the relative position between the AR glasses and an object outside the vehicle) is divided into the relative position between the vehicle and the AR glasses, and the relative position between the vehicle and the object outside the vehicle. The relative position between the vehicle and the AR glasses is determined from the information of the first feature point, and the relative position between the vehicle and the object outside the vehicle is determined from the information of the second feature point. This makes it possible to accurately determine the position of the AR glasses using sensors installed on the AR glasses.
[0014] The 3 The manner of is, 2 In this embodiment, the identifying unit determines the relative position between the vehicle and the AR glasses and the relative position between the vehicle and an object outside the vehicle interior using a localization algorithm in SLAM (Simultaneous Localization And Mapping) technology.
[0015] The 3In an aspect, the specific unit determines the relative position between the vehicle and the AR glasses and the relative position between the vehicle and an object outside the vehicle compartment by means of an algorithm for self-position estimation in existing SLAM technology. Thereby, the specific unit can be easily configured, and the development cost of the vehicle control device according to the present disclosure can be reduced.
[0016] First 4 In an aspect, in the first aspect, based on the position of the AR glasses specified by the specific unit, content is superimposed on a position outside the vehicle visible through the AR glasses at a predetermined position. A display control unit for displaying content at a position corresponding to the predetermined position in a display unit provided on the AR glasses is further included.
[0017] First 4 According to an aspect, the content to be displayed on the display unit of the AR glasses can be superimposed at an arbitrary position with respect to the scenery outside the vehicle visible through the AR glasses.
Effect of the Invention
[0018] The present disclosure has an effect that it becomes possible to accurately specify the position of the AR glasses using a sensor provided on the AR glasses.
Brief Description of the Drawings
[0019] [Figure 1] It is a block diagram showing a schematic configuration of a vehicle control system according to an embodiment. [Figure 2] It is a flowchart showing an example of glass position estimation / display control processing. [Figure 3] It is a perspective view showing a state where a user wearing AR glasses is seated on a vehicle seat. [Figure 4] It is a block diagram showing another schematic configuration of a vehicle control system.
Modes for Carrying Out the Invention
[0020] Hereinafter, an example of an embodiment of the present disclosure will be described in detail with reference to the drawings. Figure 1 shows a vehicle control system 10 according to this embodiment. The vehicle control system 10 includes AR glasses 12 worn on the user's head, a mobile terminal 30 consisting of a smartphone, tablet, or the like held by the user, and a vehicle-side system 52 mounted on a vehicle 50.
[0021] The AR glasses 12 include a wireless communication unit 14 that communicates wirelessly with a mobile terminal 30 or the like, a display unit 16 (details will be described later), and a camera 18 that photographs the area in front of the AR glasses 12. Although not shown in the illustration, the AR glasses 12 have left and right light-transmitting glass parts attached to a frame to which the bases of the left and right temples are attached. Each glass part is provided with a display unit 16 capable of displaying images on its inner surface (the surface facing the eyes of the user wearing the AR glasses 12).
[0022] The display unit 16 is a see-through type so that light entering the glass portion from the outer surface of the glass portion passes through the display unit 16 and enters the eyes of the occupant wearing the AR glasses 12. As a result, when an image is displayed on the display unit 16, the occupant wearing the AR glasses 12 sees the image displayed on the display unit 16 (virtual image) superimposed on the actual field of view through the glass portion (for example, the actual image in front of the vehicle 50).
[0023] As shown in Figure 3, in this embodiment, the camera 18 is attached to the temple of the AR glasses 12, but the placement of the camera 18 is not limited to this, and the camera 18 may be attached to the glass portion of the AR glasses 12 or elsewhere.
[0024] The mobile terminal 30 includes a CPU (Central Processing Unit), memory such as ROM (Read Only Memory) and RAM (Random Access Memory), a non-volatile storage unit such as an HDD (Hard Disk Drive) and SSD (Solid State Drive), and a wireless communication unit 32. A control program is stored in the storage unit. The mobile terminal 30 functions as an extraction unit 34, a determination unit 36, a identification unit 38, and a display control unit 40 when the CPU executes the control program, and performs the glass position identification / display control processing (Figure 2) described later.
[0025] The extraction unit 34 extracts multiple feature points from within the shooting range of the camera 18, which is installed in the AR glasses 12, based on the results of the camera 18. The determination unit 36 determines the boundary between a first region corresponding to the inside of the vehicle and a second region corresponding to the outside of the vehicle, based on the discontinuity of the movement vectors of the multiple feature points when the vehicle 50 is moving. The identification unit 38 then identifies the position of the AR glasses 12 based on the determination result from the determination unit 36.
[0026] Based on the position of the AR glasses 12 identified by the identification unit 38, the display control unit 40 displays content on the display unit 16 of the AR glasses 12 at a position corresponding to the predetermined position (a position instructed by the vehicle-side system 52) so that the content is superimposed on the view outside the vehicle seen through the AR glasses 12. In the embodiment shown in Figure 1, the mobile terminal 30 functions as an example of a vehicle control device according to this disclosure.
[0027] The vehicle-side system 52 includes one or more ECUs (Electronic Control Units) (not shown), a wireless communication unit 54 that communicates wirelessly with a mobile terminal 30, a surrounding environment sensor 56 that detects the conditions of the environment surrounding the vehicle 50, and a vehicle speed sensor 60 that detects the vehicle speed of the vehicle 50. The surrounding environment sensor 56 may be at least one of a camera, radar, and LiDAR (Light Detection and Ranging or Laser Imaging Detection and Ranging). The vehicle speed information, which represents the vehicle speed of the vehicle 50 detected by the vehicle speed sensor 60, is transmitted to the mobile terminal 30 via the wireless communication unit 54 as needed (for example, periodically).
[0028] A predetermined ECU included in the vehicle-side system 52 functions as a content display instruction unit 58. The content display instruction unit 58 monitors whether the timing has come to display content on the display unit 16 of the AR glasses 12, and when it determines that the timing has come, it transmits content information, including the type of content to be displayed on the display unit 16 of the AR glasses 12 and its 3D position, to the mobile terminal 30 via the wireless communication unit 54.
[0029] For example, the content display instruction unit 58 monitors whether the vehicle 50 has approached a location where it should turn by comparing information on the vehicle 50's planned route, obtained from the navigation device mounted on the vehicle 50, with the vehicle 50's current position, obtained from the GNSS (Global Navigation Satellite System) sensor mounted on the vehicle 50. If it determines that the vehicle 50 has approached a location where it should turn, the content display instruction unit 58 sends content information to the mobile terminal 30 instructing it to superimpose an arrow indicating the direction of the vehicle's turn onto the road ahead.
[0030] Next, as an explanation of the operation of this embodiment, the glasses position identification / display control processing performed on the mobile terminal 30 when the user is in the vehicle 50 and wearing the AR glasses 12 on their head will be described with reference to Figure 2. The glasses position identification / display control processing is triggered, for example, when a predetermined operation is performed by the user, who is in the vehicle 50 and wearing the AR glasses 12, via the mobile terminal 30 to instruct the activation of the glasses position identification / display control processing.
[0031] In step 70, the extraction unit 34 acquires image information from the AR glasses 12 representing the image captured by the camera 18 (an image including a region corresponding to the interior of the vehicle and a region corresponding to the exterior of the vehicle). In step 72, the extraction unit 34 extracts feature points from the image information acquired in step 70. In this embodiment, it is assumed that the vehicle 50 is stationary when steps 70 and 72 are performed, and while the vehicle 50 is stationary, it is not possible to distinguish whether the feature points extracted in step 72 correspond to objects located inside the vehicle or objects located outside the vehicle.
[0032] In the next step 74, the extraction unit 34 determines whether the vehicle 50 is in motion based on whether the vehicle speed of the vehicle 50, as indicated by the vehicle speed information acquired by the mobile terminal 30 from the vehicle system 52, is greater than 0 km / h. If the determination in step 74 is negative, the process returns to step 70, and steps 70 to 74 are repeated until the determination in step 74 is affirmed. If the determination in step 74 is affirmed, the process proceeds to step 76. In step 76, the extraction unit 34 acquires image information from the AR glasses 12, similar to step 70 described above, and in step 78, the extraction unit 34 extracts feature points from the acquired image information, similar to step 72 described above.
[0033] Furthermore, in this embodiment, the determination process (steps 82, 84) for determining the boundary between the interior area corresponding to the interior of the vehicle and the exterior area corresponding to the outside of the vehicle, within the shooting range of the camera 18, is performed at a first timing (the first execution of the determination process) when the processes from step 80 onwards are executed for the first time, and at a second timing after a predetermined time has elapsed since the previous execution of the determination process. Therefore, in the next step 80, the determination unit 36 determines whether the current timing is the first timing or the second timing. Note that the predetermined time defining the second timing is not limited to being fixed, but may be changed according to the vehicle speed, for example (for example, the predetermined time may be made smaller as the vehicle speed increases).
[0034] If the determination in step 80 is affirmative, the process proceeds to step 82. In step 82, the determination unit 36 calculates a movement vector for each of the feature points extracted in step 78. Figure 3 shows an example of the movement vectors for individual feature points, indicated by arrows.
[0035] As shown in Figure 3 as an example, the first feature point P1, located within the vehicle interior area, has a movement vector magnitude of 0 while the user's head, who is wearing the AR glasses 12, is stationary. When the user performs an action such as turning their head, the movement vector changes in direction and magnitude according to the user's action. On the other hand, the second feature point P2, located within the vehicle exterior area, has a movement vector whose direction is from the vanishing point in the image captured by the camera 18 towards the individual feature point P2, and the magnitude of the movement vector corresponds to the vehicle speed of the vehicle 50. Therefore, while the vehicle 50 is in motion, the direction and magnitude of the movement vectors at each feature point become discontinuous across the boundary between the vehicle interior area and the vehicle exterior area (at least one of the direction and magnitude of the movement vector clearly changes across the boundary).
[0036] In the next step 84, this is used to determine the boundary between the first region, which corresponds to the interior of the vehicle, and the second region, which corresponds to the exterior of the vehicle, based on the discontinuity of the movement vectors of multiple feature points as the vehicle 50 moves. Specifically, the process of determining the boundary between the first region and the second region is implemented by, for example, the following process.
[0037] Specifically, the determination unit 36 first determines whether the movement vector of each feature point corresponds to the direction and magnitude of the movement of the vehicle 50, thereby classifying each feature point into either a first feature point where at least one of the direction and magnitude of the movement vector does not correspond to the movement of the vehicle 50, or a second feature point where the direction and magnitude of the movement vector correspond to the movement of the vehicle 50. The determination unit 36 also calculates the distance for each classified first feature point to the nearest second feature point in the image captured by the camera 18.
[0038] Next, the determination unit 36 sets a search area smaller in area than the captured image by the camera 18, and repeats the process of searching for and extracting the first feature point that is located within the set search area and is the closest to the second feature point, while shifting the position of the search area on the captured image. The determination unit 36 then determines that the first feature point with the closest distance to the second feature point extracted at each position of the search area is a feature point on the boundary between the first and second regions, and sets the boundary between the first and second regions. As an example, in Figure 3, an example of the boundary between the first and second regions that is set is shown with a thick line and the symbol "62". Through the above process, the area within the shooting range of the camera 18 is divided into a first region corresponding to the interior of the vehicle and a second region corresponding to the exterior of the vehicle.
[0039] In step 86, the identification unit 38 determines the relative position between the vehicle 50 and the AR glasses 12 using a self-localization algorithm in SLAM technology based on information from the first feature point in the first region. Then, in the next step 88, the identification unit 38 determines the relative position between the vehicle and an object outside the vehicle using a self-localization algorithm in SLAM technology based on information from the second feature point in the second region.
[0040] In step 90, the display control unit 40 determines whether or not the content display instruction unit 58 of the vehicle-side system 52 has instructed the AR glasses 12 to display content. If the determination in step 90 is negative, the process returns to step 74, and steps 74 to 90 are repeated. If content information is received from the content display instruction unit 58, the determination in step 90 is affirmed, and the process proceeds to step 92.
[0041] In step 92, the display control unit 40 calculates the display position of the content on the display unit 16 of the AR glasses 12 in order to superimpose the content onto the real image viewed by the user through the AR glasses 12 at a three-dimensional position indicated by the content information received from the vehicle-side system 52. This calculation of the display position is performed based on the relative position between the object outside the vehicle and the AR glasses 12, which is obtained from the relative position between the vehicle 50 and the AR glasses 12 identified in step 86, and the relative position between the vehicle and the object outside the vehicle, identified in step 88.
[0042] In step 94, the display control unit 40 controls the AR glasses 12 so that the content is displayed on the display unit 16 of the AR glasses 12 at the display position calculated in step 92. As a result, the content (virtual image) displayed on the display unit 16 of the AR glasses 12 is perceived as being superimposed on the real image viewed by the user through the AR glasses 12 at the three-dimensional position indicated by the content information. After the processing in step 94 is completed, the process returns to step 74, and the processing from step 74 onward is repeated.
[0043] As described above, in this embodiment, the extraction unit 34 extracts multiple feature points from within the shooting range of the camera 18, which is provided on the AR glasses 12, based on the results of the camera 18. The determination unit 36 determines the boundary between a first region corresponding to the interior of the vehicle and a second region corresponding to the exterior of the vehicle, based on the discontinuity of the movement vectors of the multiple feature points when the vehicle 50 is moving. The identification unit 38 then identifies the position of the AR glasses 12 based on the determination result from the determination unit 36. This makes it possible to accurately determine the position of the AR glasses 12 using the camera 18 provided on the AR glasses 12.
[0044] Furthermore, in this embodiment, the determination unit 36 determines that, among the plurality of feature points extracted by the extraction unit 34, the first feature point whose direction and magnitude of the movement vector do not correspond to the movement of the vehicle 50, and which is the shortest distance from the second feature point whose direction and magnitude of the movement vector correspond to the movement of the vehicle 50, is a feature point on the boundary between the first region corresponding to the interior of the vehicle and the second region corresponding to the exterior of the vehicle. This makes it possible to accurately determine the boundary between the first region corresponding to the interior of the vehicle and the second region corresponding to the exterior of the vehicle with a simple process.
[0045] Furthermore, in this embodiment, the identification unit 38 determines the relative position between the vehicle 50 and the AR glasses 12 from information of a first feature point located within a first region corresponding to the interior of the vehicle within the shooting range of the camera 18, and determines the relative position between the vehicle 50 and an object outside the vehicle from information of a second feature point located within a second region corresponding to the exterior of the vehicle within the shooting range of the camera 18. This makes it possible to accurately determine the position of the AR glasses 12 using the camera 18 provided on the AR glasses 12.
[0046] Furthermore, in this embodiment, the identification unit 38 determines the relative position between the vehicle 50 and the AR glasses 12, and the relative position between the vehicle 50 and an object outside the vehicle interior, using a self-localization algorithm in SLAM technology. This makes it easy to configure the identification unit 38 and reduces the development cost of the control program for making the mobile terminal 30 function as a vehicle control device according to this disclosure.
[0047] Furthermore, in this embodiment, the display control unit 40 displays content at a position corresponding to a predetermined location on the display unit 16 provided on the AR glasses 12, based on the position of the AR glasses 12 identified by the identification unit 38, so that the content is superimposed at a predetermined position on the view outside the vehicle seen through the AR glasses 12. This makes it possible to superimpose the content displayed on the display unit 16 of the AR glasses 12 at any position on the view outside the vehicle seen through the AR glasses 12.
[0048] In the above embodiment, an example of a sensor in this disclosure was described in which a camera 18 provided on AR glasses 12 is applied. However, the sensor in this disclosure may be, for example, a lidar or a distance image sensor.
[0049] Furthermore, while the above embodiment describes a method of determining the relative position between the vehicle and an object outside the vehicle compartment from an image detected by a camera 18 provided on the AR glasses 12, this disclosure is not limited thereto. For example, the relative position between the vehicle and an object outside the vehicle compartment may be determined from the results of detecting the area in front of the vehicle using sensors such as cameras and lidars mounted on the vehicle.
[0050] Furthermore, although the above embodiment describes a configuration in which the mobile terminal 30 functions as an example of a vehicle control device according to the present disclosure, the present disclosure is not limited thereto. For example, as shown in Figure 4, any ECU included in the vehicle-side system 52 may be configured to function as an example of a vehicle control device according to the present disclosure. In this case, as shown in Figure 4, the ECU can be configured to communicate directly with the AR glasses 12, thereby eliminating the need for the mobile terminal 30. [Explanation of symbols]
[0051] 10. Vehicle control systems 12 AR Glasses 16 Display section 18. Camera (Sensor) 30. Mobile terminals (vehicle control devices) 34 Extraction part 36 Judgment section 38 Specific part 40 Display Control Unit 50 vehicles 52 Vehicle-side systems 58 Content display instruction unit 60 Vehicle speed sensor
Claims
1. An extraction unit extracts multiple feature points from within the detection range of a sensor provided in the AR glasses, based on the detection results from the sensor. A determination unit that determines the boundary between a first region corresponding to the interior of the vehicle and a second region corresponding to the exterior of the vehicle, based on the discontinuity of the movement vectors of the plurality of feature points during vehicle movement, A determination unit that identifies the position of the AR glasses based on the determination result from the determination unit, Includes, The determination unit determines that, among the plurality of feature points, the first feature point is one in which at least one of the direction and magnitude of the movement vector does not correspond to the movement of the vehicle, and the distance from the first feature point is the smallest to the second feature point in which the direction and magnitude of the movement vector correspond to the movement of the vehicle, and the first feature point is the feature point on the boundary.
2. The vehicle control device according to Claim 1, wherein the identifying unit determines the relative position between the vehicle and the AR glasses from information of a first feature point located within the first region within the detection range, and determines the relative position between the vehicle and an object outside the vehicle compartment from information of a second feature point located within the second region within the detection range.
3. The vehicle control device according to claim 2, wherein the identifying unit identifies the relative position between the vehicle and the AR glasses and the relative position between the vehicle and an object outside the vehicle interior using a self-position estimation algorithm in SLAM technology.
4. The vehicle control device according to claim 1, further comprising a display control unit that displays content in a display unit provided on the AR glasses at a position corresponding to the predetermined position, based on the position of the AR glasses identified by the specific unit, such that the content is superimposed in a predetermined position on the scenery outside the vehicle that is viewed through the AR glasses.