Vehicle control system and program

Abstract

To provide a vehicle control device and a program capable of three-dimensional measurement of an environment surrounding a vehicle.SOLUTION: A vehicle control device 10 comprises an operation section 12 which calculates dimensions of an object based on an image of a space surrounding a vehicle captured with a first measuring section 2 and a measurement of a second measuring section 4 measuring the object existing around the vehicle using electromagnetic waves. The operation section calculates a first measurement of the dimensions of the object based on the image of the object included in the captured image, sets a second measurement as a dimension value of the object when an error between the first measurement and the second measurement of the dimensions of the object measured with the second measuring section is equal to or lower than a predetermined value; and generates three-dimensional data of the space based on the dimension value.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a vehicle control device and a program capable of three-dimensional measurement of the surroundings of a vehicle.

Background Art

[0002] In recent years, for vehicle driving support, it has been required to acquire three-dimensional data of the environment around the vehicle. Patent Document 1 describes a distance calculation device that can calculate the distance to an object by imaging the object such as a vehicle for which pattern matching is possible in a monocular state and a stereo camera state using a stereo camera. This distance calculation device is configured to calculate the distance to an object based on the amount of blur between a monocular camera image captured by the camera with a single eye and a stereo camera image captured by a pair of cameras as a stereo camera. According to the technique described in Patent Document 1, it is possible to calculate the distance to an object based on either one of the monocular camera image and the stereo camera image.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The technique described in Patent Document 1 recognizes the area of an object included in an image by imaging an object such as a vehicle for which pattern matching is possible in advance, and estimates the dimensions of the object. The environment around the vehicle includes terrains that cannot be dimensionally set based on pattern matching, such as terrains formed by rocks, uneven ground having irregularities formed by flat surfaces or inclined surfaces, etc. In such terrains, it may be difficult to generate three-dimensional data using pattern matching based on a captured image captured by a camera.

[0005] The present invention aims to provide a vehicle control device and program capable of three-dimensional measurement of the environment surrounding a vehicle. [Means for solving the problem]

[0006] One aspect of the present invention is a vehicle control device comprising: an image capture unit that captures the space around a vehicle by a first measurement unit and a calculation unit that calculates the dimensions of an object based on measurements from a second measurement unit that measures an object present around the vehicle based on electromagnetic waves, wherein the calculation unit calculates a first measurement value of the dimensions of the object based on the image of the object included in the image capture unit, and if the error between the first measurement value and a second measurement value of the dimensions of the object measured by the second measurement unit is less than or equal to a predetermined value, the calculation unit sets the second measurement value to the dimensions of the object and generates three-dimensional data of the space based on the dimensions. [Effects of the Invention]

[0007] According to the present invention, the environment surrounding a vehicle can be measured in three dimensions. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram showing the configuration of a vehicle control device according to the embodiment. [Figure 2] This diagram shows the relationship between the dimensions of objects included in the captured image and the pixels. [Figure 3] This flowchart shows the processing flow of the vehicle control method executed in the vehicle control device. [Figure 4] This flowchart shows the process flow for generating three-dimensional data performed in a vehicle control system. [Modes for carrying out the invention]

[0009] As shown in Figure 1, vehicle 1 is connected to an external device 20 via a network W so as to be able to communicate with it. The external device 20 is, for example, another vehicle or a server device installed in equipment capable of remotely controlling vehicle 1. Vehicle 1 may be an automated driving vehicle controlled by an onboard vehicle control device 10, or it may be a manually driven vehicle with driving assistance provided by the vehicle control device 10.

[0010] Vehicle 1 includes a first measurement unit 2 and a second measurement unit 4 for measuring the surrounding environment, a position detection unit 6 for detecting the current position, a drive unit 8 which is the power source for driving, and a vehicle control device 10 for controlling vehicle 1. The first measurement unit 2 is composed of a stereo camera capable of measuring the distance to an object. The first measurement unit 2 includes, for example, a pair of camera elements. The first measurement unit 2 captures an image of the object. The first measurement unit 2 may be provided not only to capture an image in the forward direction of vehicle 1, but also to capture an image in the lateral direction and the rearward direction of vehicle 1.

[0011] The first measurement unit 2 is controlled by the vehicle control device 10. The image captured by the first measurement unit 2 is output to the vehicle control device 10. The vehicle control device 10 calculates the distance to the object and the dimensions of the object based on the captured image, as described later.

[0012] The second measurement unit 4 is comprised of a detection device capable of measuring objects present around the vehicle based on electromagnetic waves. The second measurement unit 4 is, for example, a radar device using millimeter waves or a lidar device using laser light. The second measurement unit 4 emits electromagnetic waves and receives reflected waves that strike and reflect off objects. The second measurement unit 4 measures the distance to the object based on the phase difference between the emitted electromagnetic waves and the received reflected waves. The second measurement unit 4 may be provided not only to measure the range in the forward direction of the vehicle 1, but also to measure the range to the sides and the range to the rear of the vehicle 1.

[0013] The second measurement unit 4, for example, scans electromagnetic waves in three dimensions with respect to the direction of the radio wave emission surface and receives reflected waves. The second measurement unit 4 measures the distance and dimensions to the object based on the electromagnetic waves emitted while scanning and the reflected electromagnetic waves. The measured values ​​measured by the second measurement unit 4 are output to the vehicle control device 10. The position detection unit 6 is composed of sensors such as a position sensor capable of detecting the current position, such as a GPS (Global Positioning System), an accelerometer, and an angular velocity meter. The position detection unit 6 detects the current position data of the vehicle 1, as well as acceleration and angular acceleration, and outputs them to the vehicle control device 10.

[0014] The vehicle control device 10 calculates the current position of the vehicle 1 based on the detection results of the position detection unit 6. The vehicle control device 10 calculates the distance to an object and the dimensions of the object based on the measured values, as described later, and generates three-dimensional data of the area around the vehicle 1 at its current position. The vehicle control device 10 performs driving assistance and driving control of the vehicle 1 based on the generated three-dimensional data.

[0015] The drive unit 8 includes a prime mover, which is the power source necessary to move the vehicle 1, and a steering device for controlling the direction of the vehicle 1. The prime mover and steering device may be configured separately or as an integrated unit. The drive unit 8 is controlled by the vehicle control device 10.

[0016] The vehicle control device 10 is comprised of, for example, a computer device mounted on the vehicle 1. The vehicle control device 10 includes a calculation unit 12 that performs calculation processing. The vehicle control device 10 calculates the current position using a position detection unit 6. Based on the measurement results of the first measurement unit 2 and the second measurement unit 4, the vehicle control device 10 generates three-dimensional data of the space around the vehicle 1. Based on the three-dimensional data, the vehicle control device 10 calculates the target vehicle speed at the current position, controls the drive unit 8, and drives the vehicle 1.

[0017] The vehicle control device 10 includes a storage unit 14 that stores programs and data necessary for processing by the calculation unit 12. The storage unit 14 is composed of a non-temporary storage medium such as a hard disk drive (HDD) or solid-state disk (SSD). The vehicle control device 10 includes a communication unit 16 that can be connected to a network W. The communication unit 16 is composed of a wireless communication device. The vehicle control device 10 includes a display unit 18 that can display various information. The display unit 18 is composed of a display device such as a liquid crystal display or an organic EL (Electro-Luminescence) display.

[0018] The calculation unit 12 is composed of at least one hardware processor such as a CPU (Central Processing Unit). The calculation unit 12 generates three-dimensional data of the space around the vehicle 1 based on the captured image taken by the first measurement unit 2 at the current position and the measured values ​​measured by the second measurement unit 4. The calculation unit 12 acquires the captured image of the space around the vehicle 1 taken by the first measurement unit 2. The calculation unit 12 analyzes the captured image acquired from the first measurement unit 2. The calculation unit 12 calculates the distance to objects included in the captured image based on the captured image. The calculation unit 12 extracts objects included in the captured image taken at a predetermined distance apart based on the captured image.

[0019] The calculation unit 12 calculates a first measured value of the object's dimensions based on the image of the object included in the captured image. For example, the calculation unit 12 calculates the dimensions of the object based on the relationship between the number of pixels of the object included in the captured image and the distance to the object.

[0020] As shown in FIG. 2, the arithmetic unit 12 extracts the number of pixels of the object T1 included in the captured image M1 captured at a predetermined distance from the current position. In the figure, a part of the captured image M1 composed of unit pixels Mm arranged in a grid of 6×4 is extracted and illustrated. The object T1 is captured in the captured image M1. The dimensions corresponding to the unit pixels Mm of the captured image M1 are stored in the storage unit 14 in advance as parameters based on the design values of the camera and the lens.

[0021] The arithmetic unit 12 extracts the number of pixels of the object T1 at a predetermined distance from the captured image M1 and calculates the dimensions of the object. The first measurement values (x1, y1, z1) of the dimensions of the object T1 are calculated, for example, by the distance to the object T1: x1, the horizontal dimension of the object T1: y1, and the vertical dimension of the object T1: z1. For example, the arithmetic unit 12 calculates the first measurement values (x1, y1, z1) based on the correspondence between a predetermined pixel area (for example, 2×2 pixels) on the image of the object T1 captured at a predetermined distance x1 and the actual dimensions in the captured image M1 including a plurality of unit pixels Mm. The correspondence between the pixel area and the actual dimensions is an example, and is not limited to the illustrated pixel values, and other values may be applied.

[0022] The illustrated object T1 can be measured with a maximum pixel area (for example, 3×3 pixels) depending on its position in the actual captured image M1. The first dimension of the object T1 can be calculated up to the dimension corresponding to the range of the maximum pixel area of 3×3 pixels. Therefore, the first measurement values include a first measurement error e1 based on the change in the position of the object T1 in the captured image M1. The correspondence between the maximum pixel area and the actual dimensions is an example, and is not limited to the illustrated pixel values, and other values may be applied.

[0023] The calculation unit 12 acquires the second measured value of the dimensions of the object T1 measured by the second measurement unit 4. The second measurement unit 4 detects the second measured value (x2, y2, z2), which includes the distance to the object T1: x2, the horizontal dimension of the object T1: y2, and the vertical dimension of the object T1: z2. The second measured value includes a second measurement error e2, which depends on the performance of the second measurement unit 4. The second measurement error e2 is calculated according to the magnitude of the second measured value.

[0024] The calculation unit 12 calculates the error between the first measured value of the first dimension of the object T1 at a distance X to the object T1 and the second measured value of the second dimension of the object measured by the second measurement unit 4. The calculation unit 12 calculates the difference between the first measured value and the second measured value. The calculation unit 12 compares the difference obtained by subtracting the second measured value from the first measured value with the sum of the first measurement error e1 and the second measurement error e2.

[0025] The calculation unit 12 determines whether the difference obtained by subtracting the second measurement value from the first measurement value is less than or equal to the sum of the first measurement error e1 and the second measurement error e2 (less than or equal to a predetermined value). If the difference obtained by subtracting the second measurement value from the first measurement value is less than or equal to the sum of the first measurement error e1 and the second measurement error e2, the calculation unit 12 determines whether the second dimension of the object based on the second measurement value matches the first dimension within a predetermined standard range. For example, the calculation unit 12 determines whether the horizontal dimension y2 of the second measurement value matches the number of pixels corresponding to the horizontal dimension y1 of the object in the captured image M1.

[0026] The calculation unit 12 may determine whether the vertical dimension z2 of the second measurement value matches the number of pixels corresponding to the vertical dimension z1 of the object in the captured image M1. If the second dimension of the object based on the second measurement value measured at a predetermined distance matches the first dimension within a predetermined standard range, the calculation unit 12 sets the second measurement value as the dimension of the object.

[0027] The calculation unit 12 generates three-dimensional data of the space surrounding the vehicle 1 based on the dimensional values ​​of the set object. The calculation unit 12 controls the vehicle 1 based on the calculated three-dimensional data. Based on the calculated three-dimensional data, the calculation unit 12 can drive through a space where multiple objects of different sizes exist. Based on the above process, the calculation unit 12 can recognize objects whose length and size cannot be assumed by pattern matching.

[0028] Based on the above process, Vehicle 1 can travel on uneven terrain where the shape and dimensions cannot be assumed by pattern matching. For example, based on the above process, Vehicle 1 can travel on uneven terrain including rocks, flat ground, sloping ground, and terrain with depressions. Based on the above process, even for objects whose shape and dimensions cannot be assumed by pattern matching, Vehicle 1 can determine that the measurement value of either one of the two measurement means is relatively normal by measuring the dimensions of the object based on different measurement means of the first measurement unit 2 and the second measurement unit 4.

[0029] Figure 3 shows the processing flow of the vehicle control method executed in the vehicle control device 10. The vehicle control method is executed based on a computer program installed in the computer mounted on the vehicle control device 10. The program causes the computer to perform the following processes.

[0030] The calculation unit 12 detects the initial position of the vehicle 1 based on the detection value of the position detection unit 6 (step S100). The calculation unit 12 acquires the detection values ​​of the position detection unit 6, the first measurement unit 2, and the second measurement unit 4 sensors (step S102). The calculation unit 12 calculates the current position of the vehicle 1 based on the detection value of the position detection unit 6 (step S104). The calculation unit 12 calculates the dimensions of objects surrounding the vehicle 1 based on the first measurement value of the first measurement unit 2 and the second measurement value of the second measurement unit 4, and generates three-dimensional data of the space around the vehicle 1 based on the calculated dimensions of the objects (step S106). The calculation unit 12 calculates the target speed of the vehicle 1 based on the calculated three-dimensional data and controls the drive unit 8 to drive the vehicle 1 (step S108).

[0031] Figure 4 shows the flow of the calculation process performed in step S106. The calculation unit 12 determines whether the difference between the first measured value of the first measurement unit 2 and the second measured value of the second measurement unit 4 is less than or equal to the sum of the first measurement error e1 and the second measurement error e2 of the first measured value (step S200).

[0032] If the calculation unit 12 makes a positive determination in step S200, it determines whether the second dimension of the object, based on the second measurement value measured at a predetermined distance relative to the first dimension of the object, matches within a predetermined range of criteria (for example, within the range between a predetermined pixel area and the maximum pixel area) based on the number of pixels of the object included in the captured image taken at a predetermined distance apart (step S202). If the calculation unit 12 makes a positive determination in step S202, it sets the second measurement value as the dimension of the object (step S204).

[0033] The calculation unit 12 generates three-dimensional data of the space around the vehicle 1 based on the dimensions of the object (step S206). If the calculation unit 12 makes a negative determination in step S202, it sets the first measurement value to the dimensions of the object. The calculation unit 12 repeatedly executes the processes from steps S200 to S206 to update the three-dimensional data of the space around the vehicle 1.

[0034] In step S200, if the difference between the first measured value and the second measured value exceeds a predetermined error level, the calculation unit 12 determines that an abnormality has occurred in either the first measured value or the second measured value, and outputs a predetermined notification to the external device 20 via the network W (step S210). The calculation unit 12 may also cause the vehicle 1's display unit 18 to display a notification indicating the abnormality.

[0035] As described above, the vehicle control device 10 can calculate the dimensions of an object whose dimensions cannot be assumed by pattern matching using the first measurement value of the first measurement unit 2 and the second measurement value of the second measurement unit 4. The vehicle control device 10 can determine whether either the first measurement value of the first measurement unit 2 or the second measurement value of the second measurement unit 4 is normal.

[0036] In the embodiments described above, the computer programs executed in each configuration of the vehicle control device 10 may be provided in the form of being recorded on a computer-readable portable recording medium such as a semiconductor memory, a magnetic recording medium, or an optical recording medium. [Explanation of symbols]

[0037] 1 vehicle 2. First Measurement Department 4. Second Measurement Unit 6 Position detection unit 8 Drive unit 10. Vehicle control system 12 Arithmetic section 14 Storage section 16 Communications Department 18 Display 20 External device

Claims

1. The system includes a calculation unit that calculates the dimensions of an object based on an image captured by a first measurement unit of the space surrounding the vehicle and measured values ​​from a second measurement unit that measures the object present around the vehicle based on electromagnetic waves. The aforementioned arithmetic unit, Based on the image of the object included in the captured image, a first measurement value of the object's dimensions is calculated. If the error between the first measurement value and the second measurement value of the object's dimensions measured by the second measurement unit is less than or equal to a predetermined value, the second measurement value is set to the object's dimensions. Based on the aforementioned dimensional values, three-dimensional data of the space is generated. Vehicle control system.

2. The aforementioned arithmetic unit, The first dimension of the object is calculated based on the number of pixels of the object included in the captured image, which is captured at a predetermined distance apart. If the second dimension of the object, based on the second measurement value measured at the predetermined distance, matches the first dimension within a predetermined standard range, the second measurement value is set to the dimension value of the object. The vehicle control device according to claim 1.

3. The aforementioned arithmetic unit, If the second dimension of the object, based on the second measurement value measured at the predetermined distance, exceeds a predetermined standard for the first dimension, the first measurement value is set to the dimension value of the object. The vehicle control device according to claim 2.

4. The aforementioned arithmetic unit, If the difference between the first measured value and the second measured value exceeds a predetermined error level, it is determined that an abnormality has occurred in either the first measured value or the second measured value, and a predetermined notification is output. The vehicle control device according to claim 1.

5. A program installed on a computer mounted in a vehicle control system, The first measurement unit acquires an image of the space around the vehicle. The second measurement unit measures objects present around the vehicle based on electromagnetic waves, Based on the image of the object included in the captured image, a first measurement value of the object's dimensions is calculated. If the error between the first measurement value and the second measurement value of the object's dimensions measured by the second measurement unit is less than or equal to a predetermined value, the second measurement value is set to the object's dimensions. A program that causes a computer to perform a process to generate three-dimensional data of the space based on the aforementioned dimensional values.

Images