Environment recognition device

Abstract

To reduce storage capacity when generating an environment map.SOLUTION: An environment recognition device according to the present invention comprises: a determination unit for determining the travel direction of a vehicle on the basis of a shift position which is the position of a gear shift operation unit of the vehicle; an image acquisition unit for acquiring images captured by a plurality of imaging units installed at the front, rear and side of the vehicle; an image selection unit for selecting the images captured by the imaging units at the front or rear in the travel direction and at the side of the vehicle among the images captured by each of the plurality of imaging units; a feature point detection unit for detecting a feature point for each of the selected images; and an environment recognition unit for detecting a corresponding point for each feature point, estimating the self position on the basis of the corresponding point, and generating an environment map.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an environment recognition device. [Background technology]

[0002] Visual-SLAM (Simultaneous Localization and Mapping: hereinafter referred to as "VSLAM") technology has been known as a technology for generating an environmental map by estimating a current self-position in three dimensions from images such as video data obtained from an imaging device. For example, Patent Document 1 discloses that an efficient VSLAM technology is realized by selecting effective feature points through edge detection and combination, thereby eliminating unnecessary feature points while securing a certain amount of feature points. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2023-148235 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in such a conventional technique, an environmental map of the vehicle's surroundings is created using all of the imaging devices mounted on the vehicle, which requires a huge amount of storage capacity.

[0005] The present invention has been made in view of the above, and has as its main object to provide an environment recognition device that can reduce the storage capacity required when generating an environmental map. [Means for solving the problem]

[0006] The environmental recognition device of the present invention includes a determination unit that determines the traveling direction of the vehicle based on a shift position, which is the position of the vehicle's gear shift operating unit; an image acquisition unit that acquires images captured by each of a plurality of imaging units provided at the front, rear, and sides of the vehicle; an image selection unit that selects, from the images captured by each of the plurality of imaging units, images captured by the imaging units at the front, rear, or sides, which are in the traveling direction of the vehicle; a feature point detection unit that detects feature points in each of the selected images; and an environmental recognition unit that detects corresponding points for each of the feature points, estimates the vehicle's own position based on the corresponding points, and generates an environmental map. [Effects of the Invention]

[0007] According to the environment recognition device of the present invention, it is possible to reduce the storage capacity required when generating an environment map. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a perspective view showing an example of a state in which a part of a vehicle interior of a vehicle according to a first embodiment is seen through. [Figure 2] FIG. 2 is a plan view of an example of a vehicle according to the first embodiment. [Figure 3] FIG. 3 is a block diagram illustrating an example of the configuration of a vehicle according to the first embodiment. [Figure 4] FIG. 4 is a block diagram illustrating an example of a functional configuration of an ECU included in the vehicle according to the first embodiment. [Figure 5] FIG. 5 is a diagram for explaining an example of the relationship between the case where the vehicle 1 turns and the use of the imaging unit in the first embodiment. [Figure 6] FIG. 6 is a flowchart illustrating an example of a procedure for the environment recognition process according to the first embodiment. [Figure 7] FIG. 7 is a diagram for explaining the second technique of the second embodiment. [Figure 8] FIG. 8 is a flowchart illustrating an example of a procedure for environment recognition processing according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Exemplary embodiments of the present invention are disclosed below. The configurations of the embodiments described below, as well as the actions, results, and advantages brought about by the configurations, are merely examples. The present invention can be realized by configurations other than those disclosed in the following embodiments, and it is possible to obtain at least one of the various advantages based on the basic configurations and derivative advantages.

[0010] (First embodiment) The vehicle according to this embodiment may be an automobile (internal combustion engine automobile) using an internal combustion engine (engine) as a drive source, an automobile (electric automobile, fuel cell automobile, etc.) using an electric motor (motor) as a drive source, or an automobile (hybrid automobile) using both of these as drive sources. The vehicle may be equipped with various transmissions and various devices (systems, parts, etc.) required for driving the internal combustion engine or electric motor. The type, number, layout, etc. of the devices related to the drive of the wheels of the vehicle may be variously set.

[0011] FIG. 1 is a perspective view showing an example of a state in which a portion of a cabin of a vehicle according to a first embodiment is seen through. As shown in FIG. 1, the vehicle 1 includes a vehicle body 2, a steering unit 4, an acceleration operation unit 5, a braking operation unit 6, a gear change operation unit 7, and a monitor device 11. The vehicle body 2 has a cabin 2a in which an occupant rides. Inside the cabin 2a, the steering unit 4, the acceleration operation unit 5, the braking operation unit 6, the gear change operation unit 7, etc. are provided with a driver as an occupant sitting in a seat 2b. The steering unit 4 is, for example, a steering wheel protruding from a dashboard 24. The acceleration operation unit 5 is, for example, an accelerator pedal located under the driver's feet. The braking operation unit 6 is, for example, a brake pedal located under the driver's feet. The gear change operation unit 7 is, for example, a shift lever protruding from a center console.

[0012] The monitor device 11 is provided, for example, at the center of the dashboard 24 in the vehicle width direction (i.e., left-right direction). The monitor device 11 may have functions such as a navigation system or an audio system. The monitor device 11 has a display device 8, an audio output device 9, and an operation input unit 10. The monitor device 11 may also have various operation input units such as switches, dials, joysticks, and push buttons.

[0013] The display device 8 is configured with an LCD (Liquid Crystal Display), an OLED (Organic Electroluminescent Display), or the like, and is capable of displaying various images based on image data. The audio output device 9 is configured with a speaker, or the like, and outputs various sounds based on audio data. The audio output device 9 may be provided in a different position other than the monitor device 11 within the vehicle interior 2a.

[0014] The operation input unit 10 is configured with a touch panel or the like, and enables the occupant to input various types of information. The operation input unit 10 is also provided on the display screen of the display device 8, and is transparent to images displayed on the display device 8. This allows the occupant to view the images displayed on the display screen of the display device 8. The operation input unit 10 detects the occupant's touch operation on the display screen of the display device 8, thereby accepting the occupant's input of various types of information.

[0015] Fig. 2 is a plan view of an example of a vehicle according to the first embodiment. As shown in Fig. 1 and Fig. 2, the vehicle 1 is a four-wheeled vehicle or the like, and has two front wheels 3F (left and right) and two rear wheels 3R ​​(left and right). All or some of the four wheels 3 are steerable.

[0016] The vehicle 1 is equipped with a plurality of imaging units 15 (on-board cameras). In this embodiment, the vehicle 1 is equipped with, for example, four imaging units 15a to 15d. The imaging units 15 are digital cameras having imaging elements such as a CCD (Charge Coupled Device) or a CIS (CMOS Image Sensor). The imaging units 15 are capable of capturing images of the surroundings of the vehicle 1 at a predetermined frame rate. The imaging units 15 output the captured images obtained by capturing images of the surroundings of the vehicle 1. Each imaging unit 15 has a wide-angle lens or a fisheye lens, and is capable of capturing images in a horizontal range of, for example, 140° to 220°. The optical axis of the imaging unit 15 may be set to face diagonally downward.

[0017] Specifically, the imaging unit 15a is located, for example, at the rear end 2e of the vehicle body 2 and is provided on a wall below the rear window of the rear hatch door 2h. The imaging unit 15a is capable of capturing an image of the area behind the vehicle 1 within the surroundings of the vehicle 1. The imaging unit 15b is located, for example, at the right end 2f of the vehicle body 2 and is provided on the right door mirror 2g. The imaging unit 15b is capable of capturing an image of the area to the side of the vehicle 1 within the surroundings of the vehicle 1. The imaging unit 15c is located, for example, on the front side of the vehicle body 2, i.e., at the front end 2c in the longitudinal direction of the vehicle 1, and is provided on the front bumper, front grille, or the like. The imaging unit 15c is capable of capturing an image of the area in front of the vehicle 1 within the surroundings of the vehicle 1. The imaging unit 15d is located, for example, on the left side of the vehicle body 2, i.e., at the left end 2d in the vehicle width direction, and is provided on the left door mirror 2g. The imaging unit 15d is capable of capturing an image of the area around the vehicle 1, on the side of the vehicle 1.

[0018] 3 is a block diagram showing an example of the configuration of a vehicle according to the first embodiment. Next, an example of the configuration of the vehicle 1 according to the present embodiment will be described with reference to FIG.

[0019] As shown in FIG. 3, the vehicle 1 includes a steering system 13, a brake system 18, a steering angle sensor 19, an accelerator sensor 20, a shift sensor 21, a wheel speed sensor 22, an in-vehicle network 23, and an ECU (Electronic Control Unit) 14.

[0020] The monitor device 11, steering system 13, brake system 18, steering angle sensor 19, accelerator sensor 20, shift sensor 21, wheel speed sensor 22, and ECU 14 are electrically connected via an in-vehicle network 23, which is an electrical communication line. The in-vehicle network 23 is configured by a CAN (Controller Area Network) or the like.

[0021] The steering system 13 is an electric power steering system, a steer-by-wire (SBW) system, or the like. The steering system 13 has an actuator 13a and a torque sensor 13b. The steering system 13 is electrically controlled by the ECU 14, etc., and operates the actuator 13a to apply torque to the steering unit 4 to supplement the steering force, thereby steering the wheels 3. The torque sensor 13b detects the torque applied by the driver to the steering unit 4 and transmits the detection result to the ECU 14.

[0022] The brake system 18 includes an ABS (Anti-lock Brake System) that controls the locking of the brakes of the vehicle 1, an ESC (Electronic Stability Control) that prevents the vehicle 1 from skidding when cornering, an electric brake system that increases braking force to assist braking, and a BBW (Brake By Wire).

[0023] The brake system 18 has an actuator 18a and a brake sensor 18b. The brake system 18 is electrically controlled by the ECU 14 and the like, and applies braking force to the wheels 3 via the actuator 18a. The brake system 18 detects signs of brake lock, wheel spinning, and skidding from the difference in rotation between the left and right wheels 3, and executes control to suppress brake lock, wheel spinning, and skidding of the wheels 3. The brake sensor 18b is a displacement sensor that detects the position of the brake pedal, which is a movable part of the brake operating unit 6, and transmits the detection result of the brake pedal position to the ECU 14.

[0024] The steering angle sensor 19 is a sensor that detects the steering amount of the steering unit 4 such as a steering wheel. In this embodiment, the steering angle sensor 19 is configured with a Hall element or the like, detects the rotation angle of the rotating part of the steering unit 4 as the steering amount, and transmits the detection result to the ECU 14.

[0025] The accelerator sensor 20 is a displacement sensor that detects the position of an accelerator pedal, which is a movable part of the acceleration operation unit 5, and transmits the detection result to the ECU .

[0026] The shift sensor 21 is a sensor that detects the position (hereinafter referred to as "shift position") of a movable part (such as a bar, arm, or button) of the gear shift operation unit 7, and transmits the detection result to the ECU 14. In this embodiment, the shift sensor 21 detects the shift position, such as D (drive position) or R (reverse position).

[0027] The wheel speed sensor 22 is a sensor that has a Hall element or the like and detects the amount of rotation of the wheel 3 and the number of rotations of the wheel 3 per unit time, and transmits the detection results to the ECU 14.

[0028] The ECU 14 is configured with a computer or the like, and the hardware and software work together to perform overall control of the vehicle 1. Specifically, the ECU 14 includes a central processing unit (CPU) 14a, a read-only memory (ROM) 14b, a random access memory (RAM) 14c, a display control unit 14d, an audio control unit 14e, and a solid-state drive (SSD) 14f. The CPU 14a, the ROM 14b, and the RAM 14c may be provided on the same circuit board.

[0029] The CPU 14a reads out programs stored in a nonvolatile storage device such as the ROM 14b and executes various types of arithmetic processing in accordance with the programs. For example, the CPU 14a executes image processing for image data to be displayed on the display device 8, control of the vehicle 1 traveling along a target route to a target position such as an automatic driving position or a parking position, processing related to the calibration of the imaging unit 15, and the like.

[0030] The ROM 14b stores various programs and parameters required for executing the programs.

[0031] The RAM 14c temporarily stores various data used in the calculations performed by the CPU 14a.

[0032] Among the arithmetic processing in the ECU 14, the display control unit 14d mainly performs image processing on image data acquired from the imaging unit 15 and output to the CPU 14a, and converts the image data acquired from the CPU 14a into display image data to be displayed on the display device 8.

[0033] The audio control unit 14e mainly executes the audio processing acquired from the CPU 14a and output to the audio output device 9, among the arithmetic processing in the ECU 14.

[0034] The SSD 14f is a rewritable nonvolatile storage unit, and continues to store data acquired from the CPU 14a even when the power supply to the ECU 14 is turned off.

[0035] Next, an example of the functional configuration of the ECU 14 included in the vehicle 1 according to this embodiment will be described with reference to Fig. 4. The ECU 14 operates as a vehicle control device 200. Hereinafter, the functions of the ECU 14 will be described as the vehicle control device 200. FIG. 4 is a block diagram illustrating an example of a functional configuration of an ECU included in the vehicle according to the first embodiment.

[0036] 4, the vehicle control device 200 mainly includes an environment recognition device 100, a driving control unit 210, and an environment map 230. The environment recognition device 100 mainly includes a determination unit 106, an image acquisition unit 105, an image selection unit 104, a feature point detection unit 103, and an environment recognition unit 110. The environment recognition unit 110 includes a self-position estimation unit 102 and an environment map generation unit 101.

[0037] For example, a processor such as CPU 14a mounted on a circuit board executes an environment recognition program and a vehicle control program stored in a storage medium such as ROM 14b or SSD 14f, causing ECU 14 to realize the functions of determination unit 106, image acquisition unit 105, image selection unit 104, feature point detection unit 103, self-position estimation unit 102, environmental map generation unit 101, and driving control unit 210. Some or all of the determination unit 106, image acquisition unit 105, image selection unit 104, feature point detection unit 103, self-position estimation unit 102, environmental map generation unit 101, and driving control unit 210 may be configured by hardware such as circuits.

[0038] The determination unit 106 determines the traveling direction of the vehicle 1 based on the shift position, which is the position of the gear shift operation unit 7 such as a shift lever of the vehicle 1. Specifically, when the shift position is D (drive position), the determination unit 106 determines that the traveling direction of the vehicle 1 is forward (i.e., the vehicle 1 is moving forward), and when the shift position is R (reverse position), the determination unit 106 determines that the traveling direction of the vehicle 1 is backward (i.e., the vehicle 1 is moving backward).

[0039] The image acquisition unit 105 acquires images (frames) obtained by capturing images of the surroundings of the vehicle 1 using the multiple imaging units 15. That is, the image acquisition unit 105 acquires images captured by the front imaging unit 15c, the rear imaging unit 15a, the left imaging unit 15d, and the right imaging unit 15b of the vehicle 1. An image may also be referred to as a frame.

[0040] The image selection unit 104 selects, from the images captured by the multiple imaging units 15 acquired by the image acquisition unit 105, images captured by the front imaging unit 15c or the rear imaging unit 15a, which are in the direction of travel of the vehicle 1, images captured by the left imaging unit 15d, and images captured by the right imaging unit 15b.

[0041] For example, when the shift position is D (drive position), the vehicle 1 is moving forward, so the area opposite to the direction of forward movement is the rear. Therefore, the image selection unit 104 selects images from the front imaging unit 15c, the left imaging unit 15d, and the right imaging unit 15b, which are in the direction of movement, and uses these images for generating an environmental map. On the other hand, the image selection unit 104 does not select images from the rear imaging unit 15a, which is in the direction opposite to the direction of movement, and does not use these images for generating an environmental map.

[0042] Furthermore, when the shift position is in R (reverse position), the vehicle 1 is moving backward, and the area opposite the direction of backward movement is the front. Therefore, the image selection unit 104 selects images from the rear imaging unit 15a, the left imaging unit 15d, and the right imaging unit 15b, which are in the direction of travel, and uses these images for generating an environmental map. On the other hand, the image selection unit 104 does not select images from the front imaging unit 15c, which is in the direction opposite the direction of travel, and does not use these images for generating an environmental map.

[0043] Furthermore, the image selection unit 104 selects images captured by all of the imaging units 15 while the vehicle 1 is moving within a predetermined distance from a switching point where the shift position switches from one of forward and reverse to the other. The switching point is, for example, a position where the shift position switches from D (drive position) to R (reverse position) or vice versa. The predetermined distance is a distance that is set in advance, such as a few meters, but is not limited to this.

[0044] The image selection unit 104 also determines the position of the vehicle 1 from the switching point based on the moving speed of the vehicle 1, the time from the time the shift position was switched, and the like.

[0045] FIG. 5 is a diagram for explaining an example of the relationship between the case where the vehicle 1 makes a turn and the use of the imaging unit 15 in the first embodiment. In the example of Figure 5, the vehicle 1 moves forward with the shift position of the gearshift operating unit 7 in D (drive position), and then switches the shift position of the gearshift operating unit 7 to R (reverse position) at point 502 to perform a change of direction. The position of reference numeral 502 is the change of direction point. The circular range of reference numeral 501 is a range a predetermined distance from the change of direction point 502.

[0046] 5, when the vehicle 1 starts moving forward, the image selection unit 104 acquires images from the front imaging unit 15c, the left imaging unit 15d, and the right imaging unit 15b, and uses these images for generating an environmental map. On the other hand, the image selection unit 104 does not acquire images from the rear imaging unit 15a, which is the part opposite to the direction of forward movement, and does not use these images for generating an environmental map.

[0047] Then, when the shift position is switched from D (drive position) to R (reverse position) at the turning point 502, the image selection unit 104 acquires images captured by all of the imaging units 15. Then, while the vehicle 1 continues to move backward and within a predetermined distance range 501 from the turning point 502, the image selection unit 104 acquires images captured by all of the imaging units 15 and uses them to generate an environmental map.

[0048] Furthermore, when the vehicle 1 continues to reverse and moves out of the range 501 of the predetermined distance from the turning point 502, the image selection unit 104 acquires images from the rear imaging unit 15a, the left imaging unit 15d, and the right imaging unit 15b, and uses these images for generating the environmental map. On the other hand, the image selection unit 104 does not acquire images from the front imaging unit 15c, which is in the opposite direction to the direction of reverse, and does not use these images for generating the environmental map.

[0049] Furthermore, the image acquisition unit 105 sequentially buffers images from all of the imaging units 15 in the RAM 14c as the vehicle 1 travels. For this reason, as shown in Fig. 5, from the point at which the vehicle 1 moves forward with the shift position in D (drive position) and enters the range 501 of the predetermined distance from outside to the turning point 502, the image selection unit 104 acquires images from all of the imaging units 15 that were buffered in the RAM 14c going back to the point at which the vehicle 1 turned, and uses these images for generating an environmental map.

[0050] Therefore, even before the vehicle 1 reaches the turning point, an environmental map is generated using images captured by all of the imaging units 15 as soon as the vehicle 1 enters the range 501 of the predetermined distance.

[0051] Returning to FIG. 4, the feature point detection unit 103 detects feature points for each of the images selected by the image selection unit 104. The feature point detection unit 103 is, for example, a feature point detector configured with a trained model. This trained model inputs the images selected by the image selection unit 104 and outputs a plurality of feature points and feature amounts present in the images. The feature amounts are descriptors for expressing the feature points and are expressed as high-dimensional vectors.

[0052] The trained model that constitutes the feature point detection unit 103 may be one that utilizes artificial intelligence such as a deep learning method, and for example, the feature point detector disclosed in Patent Document 1 (China Patent Application Publication No. 111344716) may be used.

[0053] The environment recognition unit 110 performs self-position estimation and the like using the VSLAM method on the image in which the feature points have been detected by the feature point detection unit 103, and generates an environment map 230.

[0054] The self-position estimation unit 102 of the environment recognition unit 110 uses, for example, the VSLAM method to detect corresponding points for each feature point in the image (i.e., perform matching), and performs self-position estimation, attitude estimation, and bundle adjustment of the vehicle 1 based on the corresponding points.

[0055] The environment map generation unit 101 of the environment recognition unit 110 generates an environment map 230 based on the estimated self-position, orientation, feature points, and corresponding points, for example, using a VSLAM technique. The environment map 230 is data indicating map points of three-dimensional coordinates defined for an object. The environment map 230 is generated and saved in a storage device such as the SSD 14f.

[0056] The driving control unit 210 controls the driving of the vehicle 1. Specifically, the driving control unit 210 refers to the environmental map 230 and executes processes for automatic driving and automatic parking assistance.

[0057] Next, the environment recognition process performed by the environment recognition device 100 according to this embodiment configured as above will be described.

[0058] FIG. 6 is a flowchart illustrating an example of a procedure for the environment recognition process according to the first embodiment. First, the determination unit 206 acquires the shift position from the shift sensor 21 and determines the traveling direction of the vehicle 1 (S101).

[0059] Next, the image acquisition unit 105 acquires images captured by the front imaging unit 15c, the rear imaging unit 15a, the left imaging unit 15d, and the right imaging unit 15b of the vehicle 1. Then, the image selection unit 104 determines that the images captured by the front imaging unit 15c or the rear imaging unit 15a, and the left imaging unit 15d and the right imaging unit 15b, which are parts in the traveling direction of the vehicle 1, will be used to generate an environmental map (S103). Then, the image selection unit 104 determines whether the shift position has been switched from one of forward (D) and reverse (R) to the other, for example, from D (drive position) to R (reverse position) (S105).

[0060] If the shift position has switched from one of forward and reverse to the other (S105: Yes), the image selection unit 104 determines that all images from the imaging unit 15 are to be used for generating the environmental map (S107). If the shift position has not switched from one of forward and reverse to the other (S105: No), the determination in S107 is not made.

[0061] Next, the image selection unit 104 determines whether the vehicle 1 is within a predetermined distance from the turning point (S109). If it is determined that the vehicle 1 is within the predetermined distance from the turning point (S109: Yes), the image selection unit 104 determines that images from all of the imaging units 15 are to be used to generate an environmental map (S111).

[0062] On the other hand, if it is determined in S109 that the vehicle 1 is outside the predetermined distance from the turning point (S109: No), the image selection unit 104 determines that the images captured by the front imaging unit 15c or the rear imaging unit 15a, which are in the direction of travel of the vehicle 1, and the left imaging unit 15d and the right imaging unit 15b, will be used to generate the environmental map (S113). Then, the image selection unit 104 selects the image of the imaging unit 15 determined above (S115).

[0063] Next, the self-location estimation unit 102 of the environment recognition unit 110 performs feature point matching on the image (i.e., detection of corresponding points for each feature point) using, for example, a VSLAM method (S119). Then, the self-location estimation unit 102 performs a process of excluding corresponding points whose matching strength is equal to or less than a predetermined threshold for each corresponding point, and excludes corresponding points with a low threshold (S121). Next, the self-location estimation unit 102 performs self-location estimation, attitude estimation, and bundle adjustment of the vehicle 1 based on the corresponding points (S123).

[0064] Next, the environment map generation unit 101 generates the environment map 230 based on the estimated self-position, orientation, feature points, and corresponding points using, for example, the VSLAM method (S125).The environment map generation unit 101 then determines whether the environment map 230 is complete (S127).If the environment map 230 is incomplete (S127: No), the process returns to S101, and the environment recognition device 100 repeatedly executes the processes from S101 to S125.

[0065] If the environment map 230 is completed (S127: Yes), the environment map generating unit 101 stores the environment map 230 in the SSD 14f (S129), and the process then ends.

[0066] As described above, the environment recognition device 100 according to this embodiment includes a determination unit 106 that determines the traveling direction of the vehicle 1 based on the shift position, which is the position of the gear change operating unit 7 of the vehicle 1; an image acquisition unit 105 that acquires images captured by each of the multiple image capture units 15 provided at the front, rear, and sides of the vehicle 1; an image selection unit 104 that selects, from the acquired images, images captured by the front image capture unit 15c or the rear image capture unit 15a, which are in the traveling direction of the vehicle, and the left image capture unit 15d and the right image capture unit 15b; a feature point detection unit 103 that detects feature points for each of the selected images; and an environment recognition unit 110 that detects corresponding points for each feature point, estimates the vehicle's own position based on the corresponding points, and generates an environmental map 230.

[0067] Therefore, in this embodiment, the environmental map 230 is generated without using images with low importance when the vehicle 1 is traveling, so that the storage capacity required when generating the environmental map 230 can be reduced.

[0068] Furthermore, in the environment recognition device 100 according to this embodiment, the image selection unit 104 selects images captured by all of the imaging units 15 while the vehicle 1 is moving within a predetermined distance from the changeover point where the shift position switches from one of forward and reverse to the other.

[0069] Therefore, according to this embodiment, when turning, an image of the entire periphery of the vehicle 1 at the turning point is used, so the accuracy of the environmental map 230 can be improved while reducing the storage capacity.

[0070] (Second embodiment) In the first embodiment, the environment recognition device 100 generates the environment map 230 after performing self-position estimation, etc., but in this second embodiment, the environment recognition device 100 generates the environment map 230, and then eliminates corresponding points with low accuracy based on the accuracy of the correspondence, and then completes the environment map 230.

[0071] The configuration of the vehicle 1, the configuration of the vehicle control device 200, and the configuration of the environment recognition device 100 according to this embodiment are the same as those in the first embodiment.

[0072] The environment map generation unit 101 of the environment recognition device 100 according to this embodiment performs threshold processing based on the accuracy of corresponding points (matching) in which feature points are associated with each other, excludes the corresponding points, and generates the environment map 230 based on the excluded corresponding points. Specifically, there are the following two methods for threshold processing based on the accuracy of corresponding points (matching).

[0073] As a first technique, the environment map generating unit 101 receives an image as an input and uses a trained model that outputs the strength of the correspondence (matching) of feature points as the accuracy of the corresponding points, to exclude images that include corresponding points whose strength is equal to or less than a predetermined threshold, thereby generating the environment map 230. As such a trained model, for example, a trained model such as lightGlue can be used.

[0074] As a second technique, when there are fewer than a predetermined number of images with corresponding points among the multiple images input as the vehicle 1 moves, the environmental map generation unit 101 determines that the accuracy of the corresponding points is low and excludes the corresponding points contained in the images with fewer than the predetermined number.

[0075] FIG. 7 is a diagram for explaining the second technique of the second embodiment. As shown in Fig. 7, images are acquired as the vehicle 1 travels, and points corresponding to the feature points are found. In the example of Fig. 7, the predetermined number is set to three.

[0076] 7, the corresponding point 701 is included in three images captured by the imaging unit 15d as the vehicle 1 travels. Therefore, the environmental map generating unit 101 determines that the corresponding point 701 has high accuracy, and uses the corresponding point 701 without excluding it to generate the environmental map 230.

[0077] On the other hand, as the vehicle 1 travels, the corresponding point 702 is included in only two images captured by the imaging unit 15d. Therefore, the environment map generating unit 101 determines that the accuracy of the corresponding point 702 is low, and generates the environment map 230 by excluding (deleting) the corresponding point 702.

[0078] Next, the environment recognition process performed by the environment recognition device 100 according to this embodiment configured as above will be described.

[0079] 8 is a flowchart showing an example of the procedure of the environment recognition process according to the second embodiment. The processes (S101 to S127) from obtaining the shift position to determining whether the environmental map 230 is complete are performed in the same manner as in the first embodiment.

[0080] If it is determined in S127 that the environment map 230 is complete (S127: Yes), the environment map generation unit 101 performs threshold processing based on the accuracy of matching (corresponding points obtained by associating feature points) using the first or second method (S201).The environment map generation unit 101 then removes corresponding points using threshold processing based on the matching accuracy, generates an environment map, and stores it in the SSD 14f (S129).

[0081] As described above, in the environment recognition device 100 according to this embodiment, the environment recognition unit 110 further excludes corresponding points based on the accuracy of the corresponding points, and generates the environment map 230 based on the excluded corresponding points. Therefore, by excluding corresponding points with low accuracy, the same effects as those of the first embodiment can be achieved, and the accuracy of the environment map 230 can be improved.

[0082] Furthermore, in the environment recognition device 100 according to this embodiment, the environment recognition unit 110 uses a trained model that inputs an image and outputs the intensity of the corresponding points as accuracy, and excludes corresponding points whose intensity is equal to or less than a predetermined threshold. Therefore, according to this embodiment, it is possible to easily identify corresponding points with low intensity using an appropriate trained model, thereby further improving the accuracy of the low-accuracy environment map 230.

[0083] Furthermore, in the environment recognition device 100 according to this embodiment, if the number of images having corresponding points is less than a predetermined threshold among the multiple images input in accordance with the progress of the vehicle 1, the environment recognition unit 110 determines that the accuracy of the corresponding points is low and excludes the corresponding points included in the images with the accuracy less than the predetermined threshold. Therefore, according to this embodiment, it is possible to easily identify corresponding points with low intensity based on the state of the corresponding points in the images, thereby further improving the accuracy of the environment map 230 with low accuracy.

[0084] The environment recognition program executed by the environment recognition device 100 of this embodiment is provided in a state that it is pre-installed in a ROM or the like.

[0085] The environment recognition program executed by the environment recognition device 100 of this embodiment may be configured to be provided by being recorded in an installable or executable file format on a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, or a DVD (Digital Versatile Disk).

[0086] Furthermore, the environment recognition program executed by the environment recognition device 100 of this embodiment may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Also, the environment recognition program executed by the environment recognition device 100 of this embodiment may be provided or distributed via a network such as the Internet.

[0087] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, and are also included in the scope of the invention and its equivalents as defined in the claims. [Explanation of symbols]

[0088] 1...vehicle, 15...imaging unit, 14...ECU, 100...environment recognition device, 101...environmental map generation unit, 102...self-position estimation unit, 103...feature point detection unit, 104...image selection unit, 105...image acquisition unit, 106...determination unit, 110...environment recognition unit, 200...vehicle control device, 210...driving control unit, 230...environmental map

Claims

1. a determination unit that determines a traveling direction of the vehicle based on a shift position that is a position of a gear shift operation unit of the vehicle; an image acquisition unit that acquires images captured by a plurality of imaging units provided at the front, rear, and sides of the vehicle; an image selection unit that selects, from the images captured by the plurality of imaging units, images captured by the imaging units of the front or rear and side portions that are in the traveling direction of the vehicle; a feature point detection unit that detects feature points for each of the selected images; an environment recognition unit that detects corresponding points for each of the feature points, estimates a self-position based on the corresponding points, and generates an environment map; An environment recognition device comprising:

2. the image selection unit selects images captured by all of the imaging units while the vehicle is moving within a predetermined distance from a switching point where the shift position is switched from one of forward and reverse to the other. The environment recognition device according to claim 1 .

3. the environment recognition unit further excludes some of the corresponding points based on accuracy of the corresponding points, and generates the environment map based on the excluded corresponding points. The environment recognition device according to claim 1 .

4. the environment recognition unit uses a trained model that inputs the image and outputs the intensities of the corresponding points as the accuracy, and excludes the corresponding points whose intensities are equal to or less than a predetermined threshold. The environment recognition device according to claim 3 .

Images