Control device

Abstract

To provide a control technique that prevents a negative effect caused by a variation in an installation position of a camera due to an environmental factor.SOLUTION: A control device acquires a first image captured by a real camera, generates a second image by applying a projective transformation to the acquired first image so that the second image corresponds to an image captured by a unified virtual camera, derives a vehicle control command from the generated second image using a control model, and controls vehicle movement based on the derived control command.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to control technology for autonomous vehicles. [Background technology]

[0002] Patent Document 1 proposes a system for autonomous vehicle control configured to determine vehicle commands from a route, GPS data, and sensor data using a trained neural network. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2019-533810 Summary of the Invention [Problem to be solved by the invention]

[0004] One of the objectives of the present disclosure is to provide a control technique for suppressing adverse effects caused by fluctuations in camera placement due to environmental dependence. [Means for solving the problem]

[0005] The control device according to the first aspect of the present disclosure includes a storage unit that stores a control model and a control unit. The control unit is configured to execute the following: acquiring a first image captured by a real camera mounted on a vehicle; generating a second image by projectively transforming the acquired first image so as to become an image captured by a unified virtual camera; providing the generated second image to the control model to derive a control command for the vehicle; and controlling the traveling of the vehicle according to the derived control command. The control model may be configured by a trained machine learning model. The machine learning model may be configured by a neural network. Effect of the Invention

[0006] According to the present disclosure, it is possible to suppress adverse effects caused by fluctuations in camera placement due to environmental dependence. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 illustrates an example of a situation to which the present disclosure is applied. [Diagram 2] FIG. 2 illustrates an example of a hardware configuration of the control device. [Diagram 3] FIG. 3 shows an example of a processing procedure regarding vehicle control by the control device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] [1 Application example] FIG. 1 is a schematic diagram showing an example of a scene to which the present disclosure is applied. In this embodiment, the control device 1 is mounted on a vehicle M, and the vehicle M is equipped with a camera C. The camera C is connected to the control device 1. The camera C is an example of a real camera. The control device 1 according to this embodiment acquires a first image 31 captured by the camera C (real camera) mounted on the vehicle M. The control device 1 generates a second image 35 by performing projective transformation on the acquired first image 31 so that the image is captured by a unified virtual camera. The first image 31 is an original image, and the second image 35 is an image after projective transformation. The control device 1 provides the generated second image 35 to a control model 5 to derive a control command for the vehicle M. Then, the control device 1 controls the traveling of the vehicle M according to the derived control command.

[0009] As another embodiment of the control device 1 according to the above embodiment, one aspect of the present disclosure may be an information processing method for implementing all or part of the above components, a program, or a storage medium in which such a program is stored and readable by a machine such as a computer. The type of vehicle M may be selected arbitrarily. As a typical example, the vehicle M may be an automobile having an autonomous driving capability of level 2 or higher. The control device 1 may directly control the traveling of the vehicle M, or may indirectly control it via another computer. The control device 1 may be mounted inside the vehicle M (FIG. 1) or may be disposed outside the vehicle M.

[0010] (Control Model) As long as a control command can be derived from the environment of the vehicle (vehicle M) including an image of a camera (camera C), the input / output form of the control model 5 is not particularly limited and may be appropriately selected according to the embodiment. In one example, the control model 5 may be configured to derive a control command from observation data of a sensor (including camera C) at one or more points in time. In another example, the control model 5 may be configured to derive a control command from a recognition result of the surrounding environment. In this case, the control device 1 may further include a trained analysis model that infers the recognition result of the surrounding environment from the observation data of a sensor including the camera C. Alternatively, the control model 5 may include a trained analysis model. The control model 5 may be configured to further accept input of any other information other than sensor data, such as a set speed, a speed limit, a position, map information, and navigation information. The control model 5 may be prepared for each scene, such as lane change, lane keeping, and emergency stop.

[0011] In one example, the control model 5 may be configured by at least one of a trained machine learning model and a rule-based model. The rule-based model is configured to match a given input (e.g., information indicating the environment such as observation data and a result of recognition of the surrounding environment) with a rule, and derive a control command depending on the result of the match (according to the matching rule). The rule may be set manually or at least partially automatically.

[0012] The machine learning model is configured to have one or more calculation parameters that can be adjusted by machine learning. The one or more calculation parameters are used for the calculation of the desired inference (in this case, derivation of a control command). Machine learning is to adjust (optimize) the value of the calculation parameter using learning data (for example, backpropagation method, etc.). The configuration and type of the machine learning model may be appropriately selected according to the embodiment. The machine learning model may be configured, for example, by a neural network, etc. When a neural network is used for the control model 5, the weight of the connection between each node included in the control model 5 (neural network) and the threshold value of each node are examples of the calculation parameters. The structure of the neural network may be determined arbitrarily.

[0013] When the control model 5 includes a machine learning model, it is preferable that, in the machine learning of the control model 5, images after projective transformation to the viewpoint of a unified virtual camera are used as training data (input data) in the same manner as in the above-mentioned inference (deriving the control command). This makes it possible to unify (share) the viewpoint of the image used for the explanatory variable during learning and inference, and therefore it is expected that the accuracy of the inference for deriving the control command will be improved. Note that, when a machine learning model is adopted, the control model 5 may be configured with an end-to-end model structure.

[0014] (camera / sensor) The type of the real camera (camera C) is not particularly limited and may be selected arbitrarily. Typically, the real camera may be an image sensor configured to capture a two-dimensional RGB image. Furthermore, the vehicle M may be equipped with other sensors other than the camera C, and the other sensors may be connected to the control device 1. The control device 1 may derive a control command by providing the control model 5 with the image obtained from the camera C and sensor data obtained from the other sensors. That is, in addition to the image of the camera C, sensor data obtained from other sensors may be further used to derive the control command. The other sensors may include, for example, a radar, a LiDAR (Light Detection And Ranging), a sonar (ultrasonic sensor), an infrared sensor, a GNSS (Global Navigation Satellite System) / GPS (Global Positioning Satellite) module, etc. In addition, any information such as the set speed may be further used to derive the control command.

[0015] (Control command) The control command relates to the operation of the vehicle. The configuration of the control command may be appropriately selected depending on the embodiment. In one example, the control command may be configured by acceleration, deceleration, steering, or a combination thereof. Acceleration and deceleration may include gear changes. When the control command includes at least one of acceleration, deceleration, and steering, the control command may be expressed by a path. Accordingly, the control model 5 may be expressed as a path planner. The control command may further include vehicle operations such as turn signals, hazards, horn, and communication processing (e.g., transmitting data to a center, making an emergency call, etc.).

[0016] (Uniform Virtual Camera) Generally, a world coordinate system, a camera coordinate system, and an image (screen) coordinate system are used to represent the relationship between the camera image and real-world objects. The camera's intrinsic and extrinsic parameters are used for the perspective projection transformation of the camera. The extrinsic parameters are used for the transformation between the world coordinate system and the camera coordinate system, and the intrinsic parameters are used for the transformation between the camera coordinate system and the image coordinate system. In one example, the intrinsic parameters may be the focal length and the optical center. The extrinsic parameters may be the position (translation) and orientation (rotation) of the camera in the world coordinate system. The position may be expressed in x, y, and z coordinates. The orientation may be expressed in roll, pitch, and yaw. The world coordinate system uses the 3D coordinate system defined in ISO8855. A vehicle fixed intermediate axis system may be used.

[0017] The internal and external parameters of the virtual camera are predefined uniformly for each vehicle. That is, the internal and external parameters of the virtual camera are set commonly for all target vehicles (given values ​​are commonly used). The internal parameters of the real camera (camera C) are determined by the attribute information (specifications) of the real camera (that is, they are given values ​​determined by the specifications). On the other hand, the external parameters of the real camera are determined by the positioning of the real camera with respect to the vehicle (vehicle M), and the values ​​of the external parameters of the real camera may differ between vehicles.

[0018] In this embodiment, the values ​​of the external parameters of the real camera may be measured (calibrated) by any method. As an example, the values ​​of the external parameters (position and orientation) of the real camera with respect to the target vehicle may be measured from an image captured by the real camera using a checkerboard, a white line, optical flow, etc. (Reference: JP 2011-185753 A). In this way, both the internal and external parameters of the real camera are obtained.

[0019] Using the acquired values ​​of the internal and external parameters of the real camera and the internal and external parameters of the virtual camera, a projection transformation formula from the image of the real camera (first image 31) to the image of the virtual camera (second image 35) is identified for each vehicle. In one example, assuming a plane, points on the image may be projected onto a world coordinate system using the internal and external parameters of the real camera to calculate the projected points. Next, the calculated points may be projected onto the image coordinate system of the virtual camera using the internal and external parameters of the virtual camera to obtain projected points. An arbitrary number of combinations of points (e.g., four) may be acquired, and a projection transformation formula may be calculated from the obtained combinations. For this calculation, a known calculation method such as the cv2.getperspectivetransform function of OpenCV may be used. A projection transformation formula may be obtained for each vehicle by performing the calculation for each vehicle. The projection transformation formula may be expressed as a matrix. Note that at least one of the position and orientation of the real camera (camera C) may be dynamic even for the same vehicle due to time factors such as aging and load. Therefore, it is preferable to measure the values ​​of the external parameters of the real camera periodically (e.g., every few minutes).

[0020] [2 Configuration example] FIG. 2 is a schematic diagram showing an example of a hardware configuration of the control device 1 according to this embodiment. As shown in FIG. 2, the control device 1 according to this embodiment is a computer to which a control unit 11, a storage unit 12, an external interface 13, an input device 14, an output device 15, and a drive 16 are electrically connected. The control unit 11 includes a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), etc., and executes any information processing. The storage unit 12 is configured to store various information such as a control program 81 and model data 125. The control program 81 is a program for causing the control device 1 to execute information processing related to the autonomous driving of the vehicle M. The control program 81 includes a series of commands for the information processing. The model data 125 is configured to indicate information related to the control model 5.

[0021] The external interface 13 may be, for example, a USB (Universal Serial Bus) port, a dedicated port, or the like, and the control device 1 may be connected to the camera C via the external interface 13. The input device 14 is, for example, a device for inputting such as a mouse or a keyboard. The output device 15 is, for example, a device for outputting such as a display or a speaker. The input device 14 and the output device 15 may be integrally configured using, for example, a touch panel display or the like.

[0022] The drive 16 is a device for reading various information such as programs stored in the storage medium 91. At least one of the control program 81 and the model data 125 may be stored in the storage medium 91 instead of or together with the storage unit 12. The storage medium 91 is configured to accumulate various information (stored programs, etc.) by electrical, magnetic, optical, mechanical, or chemical action so that a machine such as a computer can read the information. The type of the storage medium 91 may be selected arbitrarily (it may be a disk type or a non-disk type). The type of the drive 16 may be selected appropriately depending on the type of the storage medium 91.

[0023] In addition, regarding the specific hardware configuration of the control device 1, components can be omitted, replaced, or added as appropriate depending on the embodiment. For example, the control unit 11 may include multiple hardware processors. The hardware processors may be a microprocessor, a field-programmable gate array (FPGA), a digital signal processor (DSP), an ECU, (Electronic Control Unit), GPU (Graphics Processing Unit), etc. The control device 1 may be a computer designed specifically for the services provided, a general-purpose computer, a terminal device, or the like.

[0024] [3 Example of operation] 3 shows an example of a processing procedure related to vehicle control by the control device 1 according to this embodiment. The following processing procedure is an example of a control method executed by a computer.

[0025] In step S101, the control unit 11 acquires a first image 31 from the camera C. In step S102, the control unit 11 generates a second image 35 by performing projective transformation on the first image 31 so that the first image 31 becomes an image captured by a unified virtual camera.

[0026] In one example, the projective transformation formula may be obtained in advance. For example, when the camera C is mounted on the vehicle M, the values ​​of the external parameters of the camera C may be measured, and the projective transformation formula may be obtained based on the measurement results. In another example, the projective transformation formula may be calculated dynamically at regular or irregular intervals. In this case, the control unit 11 may measure each value of the external parameters of the camera C at any timing before executing step S103, and calculate the projective transformation formula using the measurement results by the above-mentioned method.

[0027] The control unit 11 can substitute the first image 31 into the obtained projective transformation formula and execute arithmetic processing to generate the second image 35. After generating the second image 35, the control unit 11 advances the process to the next step S103.

[0028] In step S103, the control unit 11 derives a control command from the second image 35 by using the control model 5 with reference to the model data 125. That is, the control unit 11 provides the second image 35 to the control model 5 to derive a control command for the vehicle M. In step S104, the control unit 11 controls the traveling of the vehicle M in accordance with the derived control command. Note that the control unit 11 may accept a driver's operation and control the operation of the target vehicle M in accordance with the given operation in place of or by overriding the control command derived in step S103.

[0029] When the control of the target vehicle M is completed, the control unit 11 ends the processing procedure of the control device 1 according to this operation example. The control unit 11 may repeatedly execute the series of information processing from step S101 to step S104. In one example, the control unit 11 may repeatedly execute the series of information processing from step S101 to step S104 for a predetermined period (for example, until the power source of the vehicle M is started and stopped). In this way, the control device 1 may continuously control the vehicle M.

[0030] [Features] Conventionally, a rule-based automatic driving system is known. According to the method of Patent Document 1 and the like, an automatic driving system can be constructed by using a trained machine learning model. However, the present inventors have found that these conventional methods have the following problems. That is, in both the rule-based model and the machine learning model, an image obtained by a camera is used to determine a control command (path) for the vehicle. However, the mounting position of the camera may differ between vehicles due to environmental factors such as differences in vehicle models and mounting errors. In other words, the placement of the camera may differ depending on the environment. As a result, the image obtained by the camera may differ from the expected image. In one example, an image with a different angle of view from the environment (e.g., the experimental environment) for model generation may be obtained. Using such an image for inference may adversely affect the accuracy of inference (determination of the control command).

[0031] In contrast to this, in this embodiment, the first image 31 obtained by the camera C is projectively transformed by the process of step S102 so as to become an image captured by a unified virtual camera. This projective transformation makes it possible to obtain an image (second image 35) in which fluctuations due to environmental dependency are suppressed. Therefore, by using the second image 35 (image after projective transformation) for inference in the control model 5 in step S103, it is possible to suppress adverse effects caused by fluctuations in the position of the camera C due to environmental dependency. This can be expected to improve the accuracy of deriving control commands and the probability of continuing autonomous driving in step S104.

[0032] [4 Variations] Although the embodiment of the present disclosure has been described in detail above, the above description is merely an example of the present disclosure in every respect. It goes without saying that various improvements and modifications can be made without departing from the scope of the present disclosure. [Explanation of symbols]

[0033] 1...control device, 11...control unit, 81...control program, 31...first image, 35...second image, 5...control model, M...vehicle, C...camera,

Claims

[Claim 1] A storage unit that stores the control model; and Control unit, A control device comprising: The control unit is acquiring a first image captured by a real camera mounted on a vehicle; generating a second image by projectively transforming the acquired first image so that the second image is an image captured by a uniform virtual camera; providing the generated second image to the control model to derive a control command for the vehicle; and Controlling vehicle travel in accordance with the derived control command; configured to execute Control device.

Images