Environment recognition device

Abstract

The environment recognition device of the present application has: a determination section that determines a traveling direction of a vehicle based on a position of a shift operation section of the vehicle, i.e., a shift position; an image acquisition section that acquires images captured by a plurality of photographing sections provided at a front portion, a rear portion, and side portions of the vehicle; an image selection section that selects, among the images captured by the plurality of photographing sections, the images captured by the photographing sections at the front portion or the rear portion and the side portions that are the traveling direction of the vehicle; a feature point detection section that detects feature points from the selected images, respectively; and an environment recognition section that detects a corresponding point for each of the feature points, estimates a self position based on the corresponding point, and generates an environment map.

Description

Technical Field

[0001] This invention relates to an environmental identification device. Background Technology

[0002] Previously, Visual-SLAM (Simultaneous Localization and Mapping, hereinafter referred to as "VSLAM") technology was known as a technique for estimating the current position of an object in three dimensions and generating an environmental map based on images such as image data obtained from an imaging device. For example, Patent Document 1 discloses a method that selects effective feature points by combining VSLAM with edge detection, thereby removing unnecessary feature points and ensuring a certain number of feature points, thus achieving an efficient VSLAM technique.

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2023-148235 Summary of the Invention

[0006] The problem the invention aims to solve

[0007] However, in such existing technologies, the entire imaging system mounted on the vehicle is used to create an environmental map of the vehicle's surroundings, resulting in a large storage capacity.

[0008] The present invention was made in view of the above-mentioned problems, and its main objective is to provide an environment identification device that can reduce storage capacity when generating environment maps.

[0009] means for solving problems

[0010] The environmental recognition device of the present invention comprises: a determination unit that determines the direction of travel of a vehicle based on the position of the vehicle's transmission operation unit, i.e., the gear shift position; an image acquisition unit that acquires images captured by a plurality of shooting units disposed at the front, rear, and side of the vehicle; an image selection unit that selects, from the images captured by the plurality of shooting units, images captured by the shooting units at the front and side of the vehicle in the direction of travel, or images captured by the shooting units at the rear and side of the vehicle in the direction of travel; a feature point detection unit that detects feature points in the selected images; and an environmental recognition unit that detects corresponding points for each feature point, estimates its own position based on the corresponding points, and generates an environmental map.

[0011] Invention Effects

[0012] According to the environmental identification device of the present invention, storage capacity can be reduced when generating environmental maps. Attached Figure Description

[0013] Figure 1 This is a perspective view showing an example of a portion of the passenger compartment of the vehicle according to the first embodiment being shown in perspective.

[0014] Figure 2 This is a top view of an example of a vehicle according to the first embodiment.

[0015] Figure 3 This is a block diagram illustrating an example of the structure of the vehicle according to the first embodiment.

[0016] Figure 4 This is a block diagram illustrating an example of the functional structure of the ECU in the vehicle according to the first embodiment.

[0017] Figure 5 This diagram illustrates an example of the relationship between the forward and backward switching of vehicle 1 in the first embodiment and the use of the camera unit.

[0018] Figure 6 This is a flowchart illustrating an example of the environmental identification processing steps involved in the first embodiment.

[0019] Figure 7 This is a diagram used to illustrate the second method of the second embodiment.

[0020] Figure 8 This is a flowchart illustrating an example of the environmental identification processing steps involved in the second embodiment. Detailed Implementation

[0021] Hereinafter, exemplary embodiments of the present invention are disclosed. The structure of the embodiments shown below, as well as the effects, results, and outcomes resulting from such structures, are examples. The present invention can be implemented using structures other than those disclosed in the following embodiments, and at least one of various effects and derived effects based on the basic structure can be obtained.

[0022] (First Implementation)

[0023] The vehicle involved in this embodiment can be a car driven by an internal combustion engine (internal combustion engine vehicle), a car driven by an electric motor (electric vehicle, fuel cell vehicle, etc.), or a car driven by both (hybrid vehicle). Furthermore, the vehicle can be equipped with various transmission devices, internal combustion engines, and various devices (systems, components, etc.) required for driving the electric motor. Additionally, the method, number, and layout of the devices related to the driving of the wheels in the vehicle can be configured in various ways.

[0024] Figure 1 This is a perspective view showing an example of a portion of the passenger compartment of the vehicle according to the first embodiment being shown in perspective. For example... Figure 1 As shown, vehicle 1 includes a body 2, a steering unit 4, an acceleration control unit 5, a braking control unit 6, a transmission control unit 7, and a monitoring device 11. The body 2 has a passenger compartment 2a for occupants. Inside the passenger compartment 2a, with the driver (as a passenger) in seat 2b, the steering unit 4, acceleration control unit 5, braking control unit 6, and transmission control unit 7 are provided. The steering unit 4 is, for example, a steering wheel protruding from the instrument panel 24. The acceleration control unit 5 is, for example, an accelerator pedal located under the driver's feet. The braking control unit 6 is, for example, a brake pedal located under the driver's feet. The transmission control unit 7 is, for example, a gear lever protruding from the center console.

[0025] The monitoring device 11 is, for example, located in the center of the instrument panel 24 in the vehicle width direction (i.e., the left-right direction). The monitoring device 11 may also have functions such as a navigation system or an audio system. The monitoring device 11 includes a display device 8, a sound output device 9, and an operation input unit 10. Furthermore, the monitoring device 11 may also include various operation input units such as switches, dials, joysticks, and buttons.

[0026] The display device 8 is composed of LCD (Liquid Crystal Display), OELD (Organic Electroluminescent Display), etc., and is capable of displaying various images based on image data. The sound output device 9 is composed of speakers, etc., and outputs various sounds based on sound data. The sound output device 9 can also be installed in different locations within the carriage 2a, other than the monitoring device 11.

[0027] The operation input unit 10 is composed of a touch panel or the like, allowing occupants to input various information. Furthermore, the operation input unit 10 is located on the display screen of the display device 8, allowing images displayed on the display device 8 to be seen. Thus, the operation input unit 10 enables occupants to visually confirm the images displayed on the display screen of the display device 8. The operation input unit 10 accepts various information input from occupants by detecting touch operations on the display screen of the display device 8.

[0028] Figure 2 This is a top view of an example of the vehicle according to the first embodiment. Figure 1 and Figure 2 As shown, vehicle 1 is a four-wheeled automobile, etc., with two front wheels 3F on the left and right and two rear wheels 3R ​​on the left and right. All or part of the four wheels 3 are capable of steering.

[0029] Vehicle 1 is equipped with a plurality of imaging units 15 (vehicle-mounted cameras). In this embodiment, vehicle 1 is equipped with, for example, four imaging units 15a to 15d. The imaging unit 15 is a digital camera having an imaging element such as a CCD (Charge Coupled Device) or a CIS (CMOS Image Sensor). The imaging unit 15 is capable of capturing images of the area around vehicle 1 at a specified frame rate. Moreover, the imaging unit 15 outputs the captured images of the area around vehicle 1. Each imaging unit 15 has a wide-angle lens or a fisheye lens, and in the horizontal direction, it can capture a range of, for example, 140° to 220°. In addition, the optical axis of the imaging unit 15 is sometimes set to face obliquely downward.

[0030] Specifically, the camera unit 15a is located, for example, at the rear end 2e of the vehicle body 2, and is installed on the wall below the rear window of the trunk door 2h. Furthermore, the camera unit 15a can capture images of the area behind the vehicle 1 in the surrounding area. The camera unit 15b is located, for example, at the right end 2f of the vehicle body 2, and is installed on the right side door mirror 2g. Furthermore, the camera unit 15b can capture images of the area to the side of the vehicle 1 in the surrounding area. The camera unit 15c is located, for example, at the front end 2c of the vehicle body 2, i.e., at the front side of the vehicle 1 in the longitudinal direction, and is installed on the front bumper, front grille, etc. Furthermore, the camera unit 15c can capture images of the area in front of the vehicle 1 in the surrounding area. The camera unit 15d is located, for example, at the left end 2d of the vehicle body 2, i.e., at the left side in the width direction, and is installed on the left side door mirror 2g. Furthermore, the camera unit 15d can capture images of the area to the side of the vehicle 1 in the surrounding area.

[0031] Figure 3 This is a block diagram illustrating an example of the structure of the vehicle according to Embodiment 1. Next, using... Figure 3 An example of the structure of the vehicle 1 involved in this embodiment will be described.

[0032] like Figure 3 As shown, vehicle 1 has a steering system 13, a braking system 18, a steering angle sensor 19, an acceleration sensor 20, a shift position sensor 21, a wheel speed sensor 22, an in-vehicle network 23, and an ECU (Electronic Control Unit) 14.

[0033] The monitoring device 11, steering system 13, braking system 18, steering angle sensor 19, acceleration sensor 20, shift position sensor 21, wheel speed sensor 22, and ECU 14 are electrically connected via an in-vehicle network 23, which serves as an electrical communication line. The in-vehicle network 23 consists of CAN (Controller Area Network) and other components.

[0034] The steering system 13 is an electric power steering system, SBW (Steer By Wire) system, etc. The steering system 13 includes an actuator 13a and a torque sensor 13b. Furthermore, the steering system 13 is electrically controlled by the ECU 14, causing the actuator 13a to actuate and apply additional torque to the steering unit 4 to supplement the steering force, thereby turning the wheels 3. The torque sensor 13b detects the torque applied to the steering unit 4 by the driver and sends the detection result to the ECU 14.

[0035] The braking system 18 includes ABS (Anti-lock Brake System) to control the locking of the brakes of the vehicle 1, ESC (Electronic Stability Control) to suppress the sideslip of the vehicle 1 during cornering, an electric braking system that enhances braking force to assist braking, and BBW (Brake By Wire).

[0036] The braking system 18 includes an actuator 18a and a brake sensor 18b. The braking system 18 is electrically controlled by an ECU 14, and applies braking force to the wheels 3 via the actuator 18a. Based on the rotational difference between the left and right wheels 3, the braking system 18 detects signs of brake lock-up, wheel spin, and sideslip, and performs controls to suppress brake lock-up, wheel spin, and sideslip. The brake sensor 18b is a displacement sensor that detects the position of the brake pedal, which is a movable part of the brake operating unit 6, and sends the detection result of the brake pedal position to the ECU 14.

[0037] The steering angle sensor 19 is a sensor that detects the amount of steering of the steering part 4, such as the steering wheel. In this embodiment, the steering angle sensor 19 is composed of a Hall element or the like, and detects the rotation angle of the rotating part of the steering part 4 as the amount of steering, and sends the detection result to the ECU 14.

[0038] Acceleration sensor 20 is a displacement sensor that detects the position of the accelerator pedal, which is a movable part of acceleration operation unit 5, and sends the detection result to ECU 14.

[0039] The shift position sensor 21 is a sensor that detects the position (hereinafter referred to as "shift position") of the movable part (lever, arm, button, etc.) of the transmission control unit 7 and sends the detection result to the ECU 14. In this embodiment, the shift position sensor 21 detects D (forward position), R (reverse position), etc. as shift positions.

[0040] The wheel speed sensor 22 has a Hall element and is a sensor that detects the amount of rotation of the wheel 3 and the number of revolutions of the wheel 3 per unit time, and sends the detection result to the ECU 14.

[0041] The ECU 14, composed of a computer and other components, is responsible for the overall control of the vehicle 1 through hardware and software collaboration. Specifically, the ECU 14 includes a CPU (Central Processing Unit) 14a, a ROM (Read Only Memory) 14b, a RAM (Random Access Memory) 14c, a display control unit 14d, a sound control unit 14e, and an SSD (Solid State Drive) 14f. The CPU 14a, ROM 14b, and RAM 14c can also be housed on the same circuit board.

[0042] CPU 14a reads a program stored in a non-volatile storage device such as ROM 14b and performs various arithmetic operations based on the program. For example, CPU 14a performs image processing for image data to be displayed on display device 8, controls the movement of vehicle 1 along a target path such as automatic driving or a target location such as a parking position, and processes related to the calibration of imaging unit 15.

[0043] ROM14b stores various programs and the parameters required to execute those programs.

[0044] RAM14c temporarily stores various data used in the operations of CPU14a.

[0045] In the computational processing of ECU14, the display control unit 14d mainly performs image processing on the image data acquired from the imaging unit 15 and output to the CPU 14a, and conversion of the image data acquired from the CPU 14a into display image data for display on the display device 8.

[0046] In the computational processing of ECU14, the sound control unit 14e mainly performs the processing of the sound obtained from CPU14a and output by the sound output device 9.

[0047] SSD14f is a rewritable, non-volatile storage unit that continues to store data acquired from CPU14a even when the power to ECU14 is disconnected.

[0048] Next, using Figure 4 An example of the functional structure of the ECU 14 in the vehicle 1 according to this embodiment will be described. The ECU 14 operates as a vehicle control device 200. Hereinafter, the function of the ECU 14 will be described as a vehicle control device 200.

[0049] Figure 4 This is a block diagram illustrating an example of the functional structure of the ECU in the vehicle according to the first embodiment.

[0050] like Figure 4 As shown, the vehicle control device 200 mainly includes an environment recognition device 100, a driving control unit 210, and an environment map 230. The environment recognition device 100 mainly includes a determination unit 106, an image acquisition unit 105, an image selection unit 104, a feature point detection unit 103, and an environment recognition unit 110. The environment recognition unit 110 includes a self-position estimation unit 102 and an environment map generation unit 101.

[0051] For example, a processor such as CPU 14a mounted on the circuit board executes an environment recognition program and a vehicle control program stored in a storage medium such as ROM 14b or SSD 14f. Thus, ECU 14 implements the functions of the determination unit 106, image acquisition unit 105, image selection unit 104, feature point detection unit 103, self-position estimation unit 102, environment map generation unit 101, and driving control unit 210. Alternatively, some or all of the determination unit 106, image acquisition unit 105, image selection unit 104, feature point detection unit 103, self-position estimation unit 102, environment map generation unit 101, and driving control unit 210 may be constructed using hardware such as circuitry.

[0052] The determination unit 106 determines the direction of travel of the vehicle 1 based on the position of the gear shift lever and other gear shifting operation parts 7 of the vehicle 1, i.e., the shift position. Specifically, when the shift position is D (forward position), the determination unit 106 determines that the direction of travel of the vehicle 1 is forward (i.e., the vehicle 1 is moving forward), and when the shift position is R (reverse position), the determination unit 106 determines that the direction of travel of the vehicle 1 is backward (i.e., the vehicle 1 is moving backward).

[0053] The image acquisition unit 105 acquires images (frames) obtained by capturing images of the area around the vehicle 1 through a plurality of capturing units 15. That is, the image acquisition unit 105 acquires images captured by the capturing unit 15c at the front of the vehicle 1, the capturing unit 15a at the rear, the capturing unit 15d at the left side, and the capturing unit 15b at the right side. Images may sometimes be referred to as frames.

[0054] The image selection unit 104 selects from the images captured by the plurality of shooting units 15 acquired by the image acquisition unit 105, the image captured by the front shooting unit 15c or the rear shooting unit 15a which is the direction of travel of the vehicle 1, the image captured by the left shooting unit 15d, and the image captured by the right shooting unit 15b.

[0055] For example, when the gear shift position is D (forward position), vehicle 1 moves forward, so the part opposite to the direction of forward movement is the rear. Therefore, the image selection unit 104 selects images from the front camera 15c, the left camera 15d, and the right camera 15b, which are in the direction of travel, respectively, for environment map generation. On the other hand, the image selection unit 104 does not select images from the rear camera 15a, which is in the opposite direction of travel, and does not use them for environment map generation.

[0056] Furthermore, when the gear shift position is R (reverse position), the vehicle 1 reverses, and the part opposite to the direction of reversing is the front. Therefore, the image selection unit 104 selects images from the rear camera 15a, the left camera 15d, and the right camera 15b, which are in the direction of travel, respectively, for environment map generation. On the other hand, the image selection unit 104 does not select images from the front camera 15c, which is in the direction opposite to the direction of travel, and these images are not used for environment map generation.

[0057] Furthermore, while vehicle 1 moves within a predetermined distance from the shift point where it switches from forward to reverse, the image selection unit 104 selects images captured by all the capturing units 15. The shift point is, for example, the position where the shift position changes from D (forward) to R (reverse) or vice versa. The predetermined distance is a pre-determined distance, for example, equivalent to several meters. However, the present invention is not limited to this.

[0058] In addition, the image selection unit 104 determines the position of the vehicle 1 from the forward / backward shift point based on the vehicle 1's moving speed and the time from the moment the shift position was changed.

[0059] Figure 5 This is a diagram illustrating an example of the relationship between the forward and backward switching of the vehicle 1 in the first embodiment and the use of the camera unit 15.

[0060] exist Figure 5 In the example, when vehicle 1 is in the forward position (D) of the transmission operation unit 7, it moves forward. At the location indicated by reference numeral 502, the transmission operation unit 7 is switched to the reverse position (R), thus performing a forward / reverse switching. The location indicated by reference numeral 502 is the forward / reverse switching location. Furthermore, the circle indicated by reference numeral 501 represents a range at a predetermined distance from the forward / reverse switching location 502.

[0061] Therefore, in Figure 5In the example, when vehicle 1 begins to move forward, image selection unit 104 acquires images from the front camera 15c, the left camera 15d, and the right camera 15b, respectively, for environment map generation. On the other hand, image selection unit 104 does not acquire images from the rear camera 15a, which is located in the opposite direction of movement, and these images are not used for environment map generation.

[0062] Furthermore, when the gear shift position is changed from D (forward position) to R (reverse position) at the forward / reverse shift point 502, the image selection unit 104 acquires images captured by all the imaging units 15. Also, while the vehicle 1 continues to move backward and is within a range 501 of a predetermined distance from the forward / reverse shift point 502, the image selection unit 104 acquires images captured by all the imaging units 15 and uses them for environmental map generation.

[0063] Furthermore, while vehicle 1 continues to move backward and reaches a distance 501 beyond the predetermined distance from the forward / backward switching point 502, the image selection unit 104 acquires images from the rear camera unit 15a, the left camera unit 15d, and the right camera unit 15b, respectively, for environment map generation. On the other hand, the image selection unit 104 does not acquire images from the front camera unit 15c, which is located in the opposite direction of the backward movement, and these images are not used for environment map generation.

[0064] Furthermore, the image acquisition unit 105 sequentially caches images from all the capturing units 15 into the RAM 14c according to the movement of the vehicle 1. Therefore, as... Figure 5 As shown, from the moment when the gear shift position is D (forward position) and the vehicle 1 moves forward and enters from outside the range 501 of the above-mentioned specified distance until the forward / backward switching point 502, the image selection unit 104 retrieves all the images captured by the imaging units 15 that have been cached in RAM 14c after the vehicle 1 has performed a forward / backward switching, and uses them for environmental map generation.

[0065] Therefore, even before the vehicle 1 reaches the forward / backward switching point, at the moment when it enters the inner side of the specified distance range 501, an environmental map is generated using images captured by all the camera units 15.

[0066] return Figure 4 The feature point detection unit 103 detects feature points in the image selected by the image selection unit 104. The feature point detection unit 103 is, for example, a feature point detector composed of a learned model. This learned model takes the image selected by the image selection unit 104 as input and outputs a plurality of feature points and feature quantities existing in the image. The feature quantities are descriptors used to represent the feature points and are represented by high-dimensional vectors.

[0067] The learned model constituting the feature point detection unit 103 can be any model that utilizes artificial intelligence methods such as deep learning, for example, the feature point detector disclosed in Patent Document 1 (Chinese Patent Application Publication No. 111344716) can be used.

[0068] The environment recognition unit 110 generates an environment map 230 by performing self-position estimation and other methods on the image from which feature points are detected by the feature point detection unit 103.

[0069] The self-position estimation unit 102 of the environment recognition unit 110 uses, for example, the VSLAM method to detect corresponding points for each feature point in the image (i.e., to perform matching), and performs self-position estimation, pose estimation and bundle adjustment of the vehicle 1 based on the corresponding points.

[0070] The environment map generation unit 101 of the environment recognition unit 110 generates an environment map 230 based on, for example, the VSLAM method, according to its estimated self-position, pose, feature points, and corresponding points. The environment map 230 is data representing map points with three-dimensional coordinates defined for an object. The environment map 230 is generated and stored on a storage device such as an SSD 14f.

[0071] The driving control unit 210 performs driving control of the vehicle 1. Specifically, the driving control unit 210 refers to the environmental map 230 and performs automatic driving and automatic parking assistance processing.

[0072] Next, the environmental identification process of the environmental identification device 100 according to this embodiment, configured as described above, will be explained.

[0073] Figure 6 This is a flowchart illustrating an example of the environmental identification processing steps involved in the first embodiment.

[0074] First, the determination unit 206 obtains the shift position from the shift position sensor 21 and determines the direction of travel of the vehicle 1 (S101).

[0075] Next, the image acquisition unit 105 acquires images captured by the front camera unit 15c, the rear camera unit 15a, the left camera unit 15d, and the right camera unit 15b of the vehicle 1. Then, the image selection unit 104 determines that the images captured by the parts of the vehicle 1 in the direction of travel, namely the front camera unit 15c or the rear camera unit 15a, the left camera unit 15d, and the right camera unit 15b, will be used for environment map generation (S103). Then, the image selection unit 104 determines whether the gear shift position has been switched from forward (D) to reverse (R), for example, from D (forward position) to R (reverse position) (S105).

[0076] If the gear shift position is changed from forward to reverse (S105: Yes), the image selection unit 104 determines to use the images from all the shooting units 15 for environment map generation (S107). If the gear shift position is not changed from forward to reverse (S105: No), the determination in S107 is not performed.

[0077] Next, the image selection unit 104 determines whether the vehicle 1 is within a specified distance from the forward / backward switching location (S109). If it is determined that the vehicle 1 is within a specified distance from the forward / backward switching location (S109: Yes), the image selection unit 104 determines to use the images from all the shooting units 15 for environment map generation (S111).

[0078] On the other hand, in S109, if it is determined that the vehicle 1 is outside the range of the specified distance from the forward / backward switching location (S109: No), the image selection unit 104 determines that the images captured by the front shooting unit 15c or the rear shooting unit 15a, the left shooting unit 15d and the right shooting unit 15b of the vehicle 1 in the direction of travel will be used for environmental map generation (S113).

[0079] Then, the image selection unit 104 selects the image from the shooting unit 15 determined above (S115).

[0080] Next, the feature point detection unit 103 detects feature points in the image selected by the image selection unit 104 in S115 (S117).

[0081] Next, the self-position estimation unit 102 of the environment recognition unit 110 performs feature point matching on the image using, for example, the VSLAM method (i.e., detecting corresponding points for each feature point) (S119). Then, the self-position estimation unit 102 performs a process to exclude corresponding points whose matching intensity is below a predetermined threshold for each corresponding point, excluding corresponding points with low thresholds (S121). Next, the self-position estimation unit 102 performs self-position estimation, pose estimation, and bundle adjustment of the vehicle 1 based on the corresponding points (S123).

[0082] Next, the environment map generation unit 101 uses, for example, the VSLAM method to generate an environment map 230 based on its estimated position, pose, feature points, and corresponding points (S125). Then, the environment map generation unit 101 determines whether the environment map 230 is complete (S127). If the environment map 230 is not complete (S127: No), the process returns to S101, and the environment recognition device 100 repeats the process from S101 to S125.

[0083] Once environment map 230 is completed (S127: Yes), environment map generation unit 101 saves environment map 230 to SSD14f (S129). Then, the process ends.

[0084] As described above, the environment recognition device 100 according to this embodiment includes: a determination unit 106, which determines the direction of travel of the vehicle 1 based on the position of the transmission operation unit 7 of the vehicle 1, i.e., the shift position; an image acquisition unit 105, which acquires images captured by a plurality of imaging units 15 disposed at the front, rear, and side of the vehicle 1; an image selection unit 104, which selects images captured by the imaging unit 15c at the front or the imaging unit 15a at the rear, as well as the imaging unit 15d at the left and the imaging unit 15b at the right, which are the directions of travel of the vehicle, from the acquired images; a feature point detection unit 103, which detects feature points in the selected images; and an environment recognition unit 110, which detects corresponding points for each feature point, estimates its own position based on the corresponding points, and generates an environment map 230.

[0085] Therefore, in this embodiment, the environment map 230 is generated without using images of low importance while the vehicle 1 is moving, thus reducing storage capacity when generating the environment map 230.

[0086] Furthermore, in the environmental recognition device 100 according to this embodiment, during the period when the vehicle 1 moves within a predetermined distance from the shift position where it switches from forward to reverse, the image selection unit 104 selects images captured by all the imaging units 15.

[0087] Therefore, according to this embodiment, when switching between advancing and retreating, an image of the entire surrounding area of ​​vehicle 1 is used at the switching location, thus reducing storage capacity while improving the accuracy of the environment map 230.

[0088] (Second Implementation)

[0089] In the first embodiment, the environment recognition device 100 generates an environment map 230 after performing its own position estimation, etc. However, in this second embodiment, after generating the environment map 230, the environment recognition device 100 completes the environment map 230 based on the accuracy of establishing the association and after excluding corresponding points with low accuracy.

[0090] The structure of the vehicle 1, the vehicle control device 200, and the environmental identification device 100 involved in this embodiment are the same as those in the first embodiment.

[0091] The environment map generation unit 101 of the environment recognition device 100 according to this embodiment performs threshold processing based on the accuracy of the corresponding points (matches) associated with the feature points, excludes corresponding points, and generates an environment map 230 based on the excluded corresponding points. Specifically, there are two methods for threshold processing based on the accuracy of the corresponding points (matches).

[0092] As a first method, the environment map generation unit 101 uses the input image and outputs a learned model that establishes an association (matching) strength between the corresponding points and the feature points with varying accuracy. It then excludes images containing corresponding points whose intensity is below a predetermined threshold and generates an environment map 230. For example, a learned model such as lightGlue can be used as such a learned model.

[0093] As a second method, the environmental map generation unit 101 determines that the accuracy of the corresponding points is low when there are fewer than a predetermined number of images with corresponding points among the multiple images input based on the movement of the vehicle 1, and excludes the corresponding points contained in the fewer than predetermined number of images.

[0094] Figure 7 This is a diagram used to illustrate the second method of the second embodiment.

[0095] like Figure 7 As shown, as vehicle 1 moves, images are acquired and the corresponding points of feature points are calculated. Figure 7 In the example, the specified quantity is set to 3.

[0096] exist Figure 7In the example, the three images captured by the imaging unit 15d during the passage of vehicle 1 contain a corresponding point 701. Therefore, the environment map generation unit 101 determines that the corresponding point 701 has high accuracy and does not exclude the corresponding point 701, and uses it for the generation of environment map 230.

[0097] On the other hand, the corresponding point 702 is only present in the two images captured by the imaging unit 15d as the vehicle 1 travels. Therefore, the environment map generation unit 101 determines that the accuracy of the corresponding point 702 is low, excludes (deletes) the corresponding point 702, and generates the environment map 230.

[0098] Next, the environmental identification process of the environmental identification device 100 according to this embodiment, configured as described above, will be explained.

[0099] Figure 8 This is a flowchart illustrating an example of the environmental identification processing steps involved in the second embodiment. The processing (S101 to S127) from the acquisition of the shift position to the determination of the completion of the environmental map 230 is performed in the same manner as in the first embodiment.

[0100] In S127, if it is determined that the environment map 230 is completed (S127: Yes), the environment map generation unit 101 performs threshold processing based on the accuracy of matching (corresponding points associated with feature points) using either the first or second method (S201). Furthermore, after excluding corresponding points through threshold processing based on the accuracy of matching, the environment map generation unit 101 generates an environment map and saves it in SSD14f (S129).

[0101] As described above, in the environmental recognition device 100 of this embodiment, the environmental recognition unit 110 further excludes corresponding points based on the accuracy of the corresponding points, and generates an environmental map 230 based on the excluded corresponding points. Therefore, by excluding corresponding points with low accuracy, in addition to achieving the same effect as in the first embodiment, the accuracy of the environmental map 230 can also be improved.

[0102] Furthermore, in the environment recognition device 100 according to this embodiment, the environment recognition unit 110 uses the input image and outputs a learned model of the intensity of corresponding points as accuracy, excluding corresponding points whose intensity is below a predetermined threshold. Therefore, according to this embodiment, corresponding points with low intensity can be easily determined using an appropriate learned model, thereby further improving the accuracy of the low-accuracy environment map 230.

[0103] Furthermore, in the environment recognition device 100 according to this embodiment, if there are fewer than a predetermined threshold images among the plurality of images input based on the movement of the vehicle 1, the environment recognition unit 110 determines that the accuracy of the corresponding points is low and excludes the corresponding points included in the images with less than the predetermined threshold. Therefore, according to this embodiment, it is possible to easily determine the corresponding points with low intensity based on the condition of the corresponding points in the images, thereby further improving the accuracy of the low-accuracy environment map 230.

[0104] Furthermore, the environment identification program executed in the environment identification device 100 of this embodiment is provided by pre-installing an environment identification program in a ROM or the like.

[0105] The environment identification program executed by the environment identification device 100 in this embodiment can be configured to be provided as an installable or executable file recorded on a computer-readable recording medium such as a CD-ROM, floppy disk (FD), CD-R, or DVD (Digital Versatile Disk).

[0106] Furthermore, the environment identification program executed by the environment identification device 100 of this embodiment may be stored on a computer connected to a network such as the Internet, and provided by downloading it via the network. Alternatively, the environment identification program executed by the environment identification device 100 of this embodiment may be provided or distributed via a network such as the Internet.

[0107] Several embodiments of the present invention have been described, but these embodiments are given by way of example and are not intended to limit the scope of the invention. These new embodiments can be implemented in various other ways, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, and are included in the scope of the invention as set forth in the claims and its equivalents.

[0108] [Summary of this implementation method]

[0109] The environmental identification device (100) of this embodiment has at least the following structure.

[0110] That is, the environmental identification device (100) according to the embodiment has: The determination unit (106) determines the direction of travel of the vehicle (1) based on the position of the transmission operation unit (7) of the vehicle (1), i.e. the shift position. The image acquisition unit (105) acquires images captured by a plurality of imaging units (15) disposed at the front, rear and side of the vehicle (1); The image selection unit (104) selects, from the images captured by the multiple shooting units (15) respectively, the image captured by the shooting units (15) at the front and side of the vehicle (1) in the direction of travel, or the image captured by the shooting units (15) at the rear and side of the vehicle (1) in the direction of travel. The feature point detection unit (103) detects feature points in the selected image respectively; and The environment recognition unit (110) detects corresponding points for each of the aforementioned feature points, estimates its own position based on the corresponding points, and generates an environment map (230).

[0111] According to this structure, as an example, an environment map (230) is generated without using images of low importance when the vehicle (1) is moving, thus reducing storage capacity when generating the environment map (230).

[0112] Furthermore, in the environmental identification device (100) according to the embodiment, During the period when the vehicle is moving within a specified distance from the forward / backward switching location, the image selection unit (104) selects images captured by all the shooting units (15), and at the forward / backward switching location, the shift position is switched from one forward and one backward to the other.

[0113] According to this structure, as an example, when switching between forward and backward, an image of the entire surrounding area of ​​the vehicle (1) is used at the switching location, thus improving the accuracy of the environment map (230) while reducing storage capacity.

[0114] Furthermore, in the environmental identification device (100) involved in the embodiment...

[0115] The environment identification unit (110) further excludes a portion of the corresponding points based on the accuracy of the corresponding points, and generates the environment map (230) based on the excluded corresponding points.

[0116] According to this structure, as an example, the accuracy of the environment map (230) can be improved by excluding corresponding points with low accuracy.

[0117] Furthermore, in the environmental identification device (100) according to the embodiment, The environment recognition unit (110) uses a learned model to exclude corresponding points whose intensity is below a predetermined threshold. The learned model inputs the image and outputs the intensity of the corresponding points as the accuracy.

[0118] Based on this structure, as an example, it is possible to easily determine the corresponding points of low intensity using an appropriate learned model, thereby further improving the accuracy of the low-precision environment map (230).

[0119] Explanation of reference numerals in the attached figures

[0120] 1 Vehicle, 15 Imaging Unit, 14 ECU, 100 Environmental Recognition Device, 101 Environmental Map Generation Unit, 102 Self-Position Estimation Unit, 103 Feature Point Detection Unit, 104 Image Selection Unit, 105 Image Acquisition Unit, 106 Judgment Unit, 110 Environmental Recognition Unit, 200 Vehicle Control Device, 210 Driving Control Unit, 230 Environmental Map.

Claims

1. An environmental identification device, wherein, have: The determination unit determines the direction of travel of the vehicle based on the position of the vehicle's transmission operation unit, i.e., the shift position. The image acquisition unit acquires images captured by a plurality of shooting units disposed at the front, rear, and sides of the vehicle. The image selection unit selects, from the images captured by the imaging units at the front and side of the vehicle's direction of travel, or the images captured by the imaging units at the rear and side of the vehicle's direction of travel, from the images captured by the plurality of imaging units respectively. The feature point detection unit detects feature points in the selected image. as well as The environment recognition unit detects corresponding points for each of the aforementioned feature points, estimates its own position based on the corresponding points, and generates an environment map.

2. The environmental identification device according to claim 1, wherein, During the period when the vehicle is moving within a specified distance from the forward / reverse switching point, the image selection unit selects images captured by all the shooting units, and at the forward / reverse switching point, the shift position is switched from forward to reverse.

3. The environmental identification device according to claim 1, wherein, The environment identification unit further excludes a portion of the corresponding points based on the accuracy of the corresponding points, and generates the environment map based on the excluded corresponding points.

4. The environmental identification device according to claim 3, wherein, The environment recognition unit uses a learned model to exclude corresponding points whose intensity is below a specified threshold. The learned model inputs the image and outputs the intensity of the corresponding points as the accuracy.

Images