Position estimation device, position estimation system, position estimation method, and computer program

The position estimation device enhances vehicle position accuracy by identifying vehicle type and applying type-specific parameters, correcting and transforming captured images to improve estimation precision.

JP2024157649AActive Publication Date: 2024-11-08TOYOTA JIDOSHA KK
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
JP2023072118
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-04-26
Publication Date
2024-11-08
Estimated Expiration
2043-04-26

AI Technical Summary

Technical Problem

Existing methods for estimating the position of a vehicle in automatic remote control systems suffer from accuracy decreases due to variations in vehicle type, which affect the estimation based on captured images.

Method used

A position estimation device that identifies the vehicle type and uses vehicle-specific parameters to convert image coordinates into global coordinates, incorporating image correction, rotation, and perspective transformation to enhance accuracy.

Benefits of technology

The solution improves the accuracy of vehicle position estimation by considering vehicle type, reducing errors caused by class variations and enabling precise tracking of vehicle positions even in diverse environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024157649000001_ABST
    Figure 2024157649000001_ABST
Patent Text Reader

Abstract

To provide a position estimation device, a position estimation system, a position estimation method and a program which accurately estimate a position of a vehicle, by using a captured image including the vehicle.SOLUTION: In a position estimation system, one or more vehicles having vehicle controllers, a plurality of imaging devices, and a position estimation device which estimates a position of the vehicle are connected to a network and communicate with each other. A device CPU 72 with which a position estimation device 7 which estimates the position of the vehicle is provided functions as: a device acquisition unit 721 which acquires a captured image including the vehicle from the imaging device; a position calculation unit 722 which calculates an image coordinate point indicating the position of the vehicle in an image coordinate system, by using the captured image; a vehicle type identification unit 723 which identifies a type of the vehicle included in the captured image; and a position conversion unit 724 which converts the image coordinate point into a vehicle coordinate point indicating the position of the vehicle in a global coordinate system, by using an imaging parameter calculated based on the position of the imaging device in the global coordinate system, and a vehicle parameter determined according to the type of the vehicle.SELECTED DRAWING: Figure 5
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to a position estimation device, a position estimation system, a position estimation method, and a computer program. [Background technology]

[0002] 2. Description of the Related Art Conventionally, vehicles that run autonomously under remote control are known (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2017-538619 Summary of the Invention [Problem to be solved by the invention]

[0004] When a vehicle is made to travel automatically by remote control, it is necessary to estimate the position of the vehicle. The inventors of the present application have found that if the position of the vehicle is estimated using an image captured by capturing an image of an imaging area including the vehicle from outside the vehicle, the accuracy of estimating the position of the vehicle may be reduced because the class of the vehicle varies depending on the type of vehicle. [Means for solving the problem]

[0005] The present disclosure can be realized in the following forms.

[0006] (1) According to a first embodiment of the present disclosure, a position estimation device is provided. The position estimation device for estimating the position of a vehicle includes an acquisition unit that acquires an image including the vehicle from an imaging device, a position calculation unit that calculates an image coordinate point indicating the position of the vehicle in an image coordinate system using the captured image, a vehicle type identification unit that identifies the type of the vehicle included in the captured image, and a position conversion unit that converts the image coordinate point into a vehicle coordinate point indicating the position of the vehicle in the global coordinate system using imaging parameters calculated based on the position of the imaging device in a global coordinate system and vehicle parameters determined according to the type of the vehicle. According to this embodiment, the type of vehicle included in the captured image can be identified and vehicle parameters according to the type of vehicle can be acquired. Then, the position of the vehicle can be estimated from the captured image using the vehicle parameters according to the type of vehicle. In this way, the position of the vehicle can be estimated taking into account the type of vehicle. This makes it possible to suppress a decrease in the estimation accuracy of the position of the vehicle due to differences in vehicle class, etc. depending on the type of vehicle. (2) In the above aspect, the vehicle control system may further include a direction information generating unit that generates direction information indicating a moving direction of the vehicle using the vehicle coordinate points. According to this aspect, the direction information can be generated using the position information of the vehicle. In this way, it is possible to estimate an actual traveling trajectory and traveling direction of the vehicle while it is traveling. (3) In the above aspect, when the position conversion unit converts the image coordinate points into the vehicle coordinate points, the position conversion unit may substitute the values ​​of the vehicle parameters acquired by the acquisition unit into a relational expression that uses the vehicle coordinate points as an objective variable and includes the image coordinate points, the imaging parameters, and the vehicle parameters as explanatory variables. According to this aspect, the image coordinate points can be converted into vehicle coordinate points by substituting the values ​​of the vehicle parameters acquired by the acquisition unit into a relational expression that uses the vehicle coordinate points as an objective variable and includes the image coordinate points, the imaging parameters, and the vehicle parameters as explanatory variables. (4) The above aspect further includes a correction value setting unit that calculates a difference between the vehicle parameter acquired by the acquisition unit and a reference parameter indicating a reference value of the vehicle parameter, and sets a correction parameter indicating a correction value for the reference parameter, the correction parameter being determined according to the type of the vehicle identified by the vehicle type identification unit, and when converting the image coordinate point to the vehicle coordinate point, the position conversion unit may substitute the correction parameter set by the correction value setting unit into a relational expression that uses the vehicle coordinate point as an objective variable and includes the image coordinate point, the imaging parameter, the reference parameter, and the correction parameter as explanatory variables. According to this aspect, the type of the vehicle included in the captured image can be identified and a correction parameter corresponding to the type of the vehicle can be calculated. Then, the image coordinate point can be converted to the vehicle coordinate point by substituting the correction parameter set by the correction value setting unit into a relational expression that uses the vehicle coordinate point as an objective variable and includes the image coordinate point, the imaging parameter, the reference parameter, and the correction parameter as explanatory variables. (5) In the above aspect, the correction value setting unit may set the correction parameter to zero without calculating the difference when the type of the vehicle identified by the vehicle type identification unit is a predetermined type. According to this aspect, when the type of the vehicle identified by the vehicle type identification unit is a predetermined type, the correction parameter can be set to zero without calculating the difference between the reference parameter and the vehicle parameter. (6) In the above aspect, the imaging parameter may be a height of the imaging device from a road surface calculated based on the position of the imaging device in the global coordinate system, and the vehicle parameter may be a height of a predetermined positioning point of the vehicle from the road surface. According to this aspect, the image coordinate points can be converted into vehicle coordinate points using a ratio between the height of the imaging device from the road surface and the height of the vehicle's positioning point from the road surface. (7) In the above embodiment, the position calculation unit may include a detection unit that generates a first mask image to which a mask area that masks the vehicle in the captured image is added, a perspective transformation unit that performs perspective transformation on the first mask image, and a coordinate point calculation unit that sets a specified vertex of a first circumscribing rectangle set in the mask area in the first mask image as a first coordinate point, sets a vertex indicating the same position as the first coordinate point among the vertices of a second circumscribing rectangle set in the mask area in a second mask image obtained by perspective transformation of the first mask image as a second coordinate point, and corrects the first coordinate point using the second coordinate point. According to this embodiment, it is possible to generate a first mask image to which a mask area that masks the vehicle in the captured image is added, and a second mask image obtained by perspective transformation of the first mask image. In this way, it is possible to extract the first coordinate point from the first mask image and extract the second coordinate point from the second mask image. Then, it is possible to calculate the image coordinate point by correcting the first coordinate point using the second coordinate point. This makes it possible to more accurately calculate the image coordinate point. (8) In the above embodiment, the position calculation unit may further include a rotation processing unit that rotates the captured image so that the moving direction of the vehicle faces a predetermined direction. According to this embodiment, the captured image can be rotated so that the direction of a vector indicating the moving direction of the vehicle faces a predetermined direction. In this way, it is possible to detect a vehicle included in the captured image with the direction of the vector indicating the moving direction of the vehicle being unified. This can improve the estimation accuracy of the vehicle position. (9) In the above aspect, the position calculation unit may further include a distortion correction unit that corrects distortion of the captured image. According to this aspect, it is possible to correct distortion of the captured image. This makes it possible to further improve the estimation accuracy of the vehicle position. (10) According to a second aspect of the present disclosure, a position estimation system is provided. The position estimation system for estimating a position of a vehicle includes a vehicle, an imaging device for acquiring a captured image including the vehicle as a target for position estimation, and the position estimation device according to the above aspect. According to this aspect, the type of vehicle included in the captured image can be identified and vehicle parameters corresponding to the type of vehicle can be acquired. Then, the position of the vehicle can be estimated from the captured image using the vehicle parameters corresponding to the type of vehicle. In this way, the position of the vehicle can be estimated taking the type of vehicle into consideration. This makes it possible to suppress a decrease in the estimation accuracy of the vehicle position due to differences in vehicle class, etc. depending on the type of vehicle. (11) According to a third embodiment of the present disclosure, a position estimation method is provided. The position estimation method for estimating the position of a vehicle includes: an acquisition step of acquiring an image including the vehicle from an imaging device; a position calculation step of calculating an image coordinate point indicating the position of the vehicle in an image coordinate system using the captured image; a vehicle type identification step of identifying a type of the vehicle included in the captured image; and a position conversion step of converting the image coordinate point into a vehicle coordinate point indicating the position of the vehicle in the global coordinate system using imaging parameters calculated based on the position of the imaging device in a global coordinate system and vehicle parameters determined according to the type of the vehicle. According to this embodiment, the type of vehicle included in the captured image can be identified and vehicle parameters according to the type of vehicle can be acquired. Then, the position of the vehicle can be estimated from the captured image using the vehicle parameters according to the type of vehicle. In this way, the position of the vehicle can be estimated taking into consideration the type of vehicle. This makes it possible to suppress a decrease in the estimation accuracy of the position of the vehicle due to differences in vehicle class, etc. depending on the type of vehicle. (12) According to a fourth aspect of the present disclosure, a computer program is provided. The computer program used to estimate the position of a vehicle includes an acquisition function for acquiring an image including the vehicle from an imaging device, a position calculation function for calculating an image coordinate point indicating the position of the vehicle in an image coordinate system using the captured image, a vehicle type identification function for identifying the type of the vehicle included in the captured image, and a position conversion function for converting the image coordinate point into a vehicle coordinate point indicating the position of the vehicle in the global coordinate system using imaging parameters calculated based on the position of the imaging device in a global coordinate system and vehicle parameters determined according to the type of the vehicle. According to this aspect, the type of vehicle included in the captured image can be identified and vehicle parameters according to the type of vehicle can be acquired. Then, the position of the vehicle can be estimated from the captured image using the vehicle parameters according to the type of vehicle. In this way, the position of the vehicle can be estimated taking into consideration the type of vehicle. This makes it possible to suppress a decrease in the estimation accuracy of the position of the vehicle due to differences in vehicle class, etc. depending on the type of vehicle. The present disclosure can be realized in various forms other than the above-mentioned position estimation device, position estimation system, position estimation method, and computer program, for example, in the form of a manufacturing method of a position estimation device, a control method of a position estimation device and a position estimation system, etc. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram showing a schematic configuration of a position estimation system. [Diagram 2] FIG. 1 is a diagram showing a schematic configuration of a vehicle. [Diagram 3] FIG. 2 is a diagram showing a schematic configuration of a remote control device. [Figure 4] FIG. 1 is a diagram showing a schematic configuration of a position estimation device. [Diagram 5] FIG. 2 is a diagram showing details of a device CPU according to the first embodiment. [Figure 6] 4 is a flowchart showing a first operation control method. [Figure 7] 6 is a flowchart showing a second operation control method. [Figure 8] 4 is a flowchart showing a vehicle position estimation method in the first embodiment. [Figure 9] Schematic diagrams showing examples of various images. [Figure 10] FIG. 11 is a diagram for explaining details of a coordinate point calculation step. [Figure 11] FIG. 4 is a diagram for explaining a method of acquiring a base coordinate point. [Figure 12] FIG. 1 is a diagram for explaining details of a position conversion step in the first embodiment. [Figure 13] FIG. 2 is a diagram for explaining details of a position conversion process in the first embodiment. [Figure 14] FIG. 11 is a diagram showing details of a device CPU according to a second embodiment. [Figure 15] 10 is a flowchart showing a vehicle position estimation method in a second embodiment. [Figure 16] FIG. 1 is a diagram for explaining details of a position conversion step in the second embodiment. [Figure 17] FIG. 2 is a diagram for explaining details of a position conversion process in the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] A. First embodiment: A-1. Location estimation system configuration: 1 is a diagram showing a schematic configuration of a position estimation system 1. The position estimation system 1 is a system that estimates the position of a vehicle 10. The position estimation system 1 includes one or more vehicles 10, an imaging device 9, and a position estimation device 7 that estimates the position of the vehicle 10.

[0009] The imaging device 9 captures an image of an imaging region RG including the vehicle 10 to be subjected to position estimation from outside the vehicle 10 to obtain a captured image. The imaging device 9 transmits the captured image to the position estimation device 7 together with camera identification information (camera ID) for identifying the multiple imaging devices 9 and the acquisition timing of the captured image. The captured image transmitted to the position estimation device 7 is used to estimate the position of the vehicle 10. The captured image is a two-dimensional image formed by a collection of pixels arranged on the XcYc plane of the camera coordinate system. The camera coordinate system is a coordinate system having the focal point of the imaging device 9 as the origin and coordinate axes indicated by the Xc axis and the Yc axis perpendicular to the Xc axis. The captured image includes at least two-dimensional data of the vehicle 10 to be subjected to position estimation. The captured image is preferably a color image, but may be a gray image. The imaging device 9 is, for example, an infrastructure camera having an imaging element such as a CCD image sensor or a CMOS image sensor and an optical system.

[0010] In this embodiment, the imaging device 9 acquires an image of the road surface 60 and the vehicle 10 traveling on the road surface 60 from above. The installation positions and the number of the imaging devices 9 are determined in consideration of the imaging area RG (angle of view) of the imaging device 9, etc., in order to image the entire road surface 60 by one or more imaging devices 9. Specifically, each imaging device 9 is installed so that the first imaging area RG1, which is the imaging area RG of the adjacent first imaging device 901, and the second imaging area RG2, which is the imaging area RG of the second imaging device 902, overlap with each other. Furthermore, each imaging device 9 is installed at a position where it can image a positioning point 10e that is set in advance for a specific part of the vehicle 10 traveling on the road surface 60. In this embodiment, the positioning point 10e is the rear end on the left side of the vehicle 10 (hereinafter, the left rear end). The positioning point 10e may be other than the left rear end of the vehicle 10. The imaging device 9 may obtain information not only from above the vehicle 10, but also from the front, rear, sides, etc. of the vehicle 10. The imaging device 9 may transmit the captured image to the remote control device 5.

[0011] The vehicle 10 has a manned driving mode and a remote automatic driving mode. The manned driving mode is a driving mode in which a driver in the vehicle 10 determines the driving conditions of the vehicle 10 by operating a driver input device (not shown) such as a steering wheel or an accelerator provided on the vehicle 10, and the vehicle 10 drives. The driving conditions here are conditions that specify the driving operation of the vehicle 10. The driving conditions include, for example, the driving route, position, driving speed, acceleration, and steering angle of the wheels of the vehicle 10. The remote automatic driving mode is a driving mode in which the vehicle 10 receives the driving conditions of the vehicle 10 from outside the vehicle 10, and the vehicle 10 automatically drives without a driver in the vehicle 10. When driving in the remote automatic driving mode, the vehicle 10 automatically drives by controlling the operation of a drive device and the like mounted on the vehicle 10, which will be described later, according to a control value received from a remote control device 5 such as a server provided at a location different from the vehicle 10. In this embodiment, the vehicle 10 drives in the remote automatic driving mode in a factory where the vehicle 10 is produced by executing a plurality of production processes. The factory is not limited to being located in one building, or at one site or address, but may be located across multiple buildings, multiple sites, multiple addresses, etc. In this case, the vehicle 10 may travel on public roads, not limited to private roads. The position estimation system 1 may also estimate the position of the vehicle 10 traveling outside the factory.

[0012] The vehicle 10 is, for example, an electric vehicle, a hybrid vehicle, a fuel cell vehicle, a gasoline vehicle, or a diesel vehicle. The vehicle 10 may be a private vehicle such as a passenger car, or a commercial vehicle such as a truck, a bus, or a construction vehicle. The vehicle 10 includes both a completed vehicle 10 as a product and a semi-finished or work-in-progress vehicle 10.

[0013] Fig. 2 is a diagram showing a schematic configuration of a vehicle 10. Fig. 2 shows a representative portion of the configuration of the vehicle 10. The vehicle 10 includes a drive device 11, a steering device 12, a braking device 13, an external sensor group 16 having an external sensor 160, an internal sensor group 17 having an internal sensor 170, and a vehicle control device 2.

[0014] The drive device 11 accelerates the vehicle 10. For example, when the vehicle 10 is an electric vehicle, the drive device 11 includes a battery (not shown), a motor (not shown) driven by the battery power, and wheels (not shown) rotated by the motor.

[0015] The steering device 12 changes the traveling direction of the vehicle 10. The steering device 12 rotates a steering shaft (not shown) by torque from a steering motor (not shown) according to a control value received from the remote control device 5 so that the steering angle of a steering wheel (not shown) coincides with a target steering angle of the wheels (not shown). In this way, in the remote automatic driving mode, the steering device 12 steers the wheels without steering operation.

[0016] The braking device 13 decelerates the vehicle 10. The braking device 13 is, for example, a disc brake device.

[0017] The external sensor group 16 includes a plurality of types of external sensors 160. The external sensors 160 are sensors that acquire peripheral information indicating the state of the area surrounding the vehicle 10. The external sensors 160 transmit the acquired data to the vehicle control device 2. In this embodiment, the external sensor group 16 includes, as the external sensors 160, an in-vehicle camera 161, a radar 162, and a lidar 163.

[0018] The in-vehicle camera 161 captures an image of an area including at least a part of the surrounding area of ​​the vehicle 10 to obtain internal imaging information as imaging data. The internal imaging information may be still image data or video data. The internal imaging information may be color data or monochrome data.

[0019] The radar 162 emits search waves (radio waves) into a predetermined search range and receives the reflected waves reflected by objects in the surrounding area of ​​the vehicle 10, thereby detecting the distance, angle, relative speed, etc. to objects in the surrounding area of ​​the vehicle 10.

[0020] The lidar 163 irradiates a laser light within a predetermined measurement range and detects the light reflected by an object present in the peripheral area of ​​the vehicle 10, thereby detecting the distance, angle, shape, etc. of the object present in the peripheral area of ​​the vehicle 10. Note that the configuration of the external sensor group 16 is not limited to this.

[0021] The internal sensor group 17 includes multiple types of internal sensors 170. The internal sensors 170 are sensors that acquire various physical quantities necessary to control the traveling operation of the vehicle 10. The internal sensors 170 transmit the acquired data to the vehicle control device 2. In this embodiment, the internal sensor group 17 includes a wheel speed sensor 171 and a steering angle sensor 172 as the internal sensors 170. The wheel speed sensor 171 measures the rotation speed of each wheel (hereinafter, wheel speed). The steering angle sensor 172 measures the actual steering angle of each wheel. Note that the configuration of the internal sensor group 17 is not limited to this.

[0022] The vehicle control device 2 includes a vehicle communication unit 21 as a communication unit of the vehicle control device 2, a vehicle memory unit 23 as a memory unit of the vehicle control device 2, and a vehicle CPU 22 as a central processing unit of the vehicle control device 2. The vehicle communication unit 21, the vehicle memory unit 23, and the vehicle CPU 22 are connected to each other via an internal bus and an interface circuit.

[0023] The vehicle communication unit 21 connects the internal devices and external devices to the vehicle control device 2 so that they can communicate with each other. The internal devices are devices mounted on the vehicle 10, such as the internal sensor 170 and the external sensor 160, and are capable of communicating with the vehicle control device 2. The external devices are devices provided in a location different from the vehicle 10, such as the remote control device 5 and the imaging device 9. The vehicle communication unit 21 is, for example, a wireless communication device. The vehicle communication unit 21 communicates with the internal devices, for example, by CAN (Controller Area Network) communication. The CAN communication is a communication standard that allows multi-directional transmission or reception. The vehicle communication unit 21 also communicates with external devices, such as the remote control device 5 and the imaging device 9, connected to the network N, via an access point (not shown) in a factory, for example. The vehicle communication unit 21 may communicate with the vehicle control device 2 of another vehicle 10. The communication method by the vehicle communication unit 21 is not limited to this.

[0024] The vehicle storage unit 23 stores various information including various programs that control the traveling operation of the vehicle 10. The vehicle storage unit 23 includes, for example, a RAM, a ROM, and a hard disk drive (HDD).

[0025] The vehicle CPU 22 deploys various programs stored in the vehicle storage unit 23, and thereby functions as a vehicle speed calculation unit 221, a vehicle acquisition unit 222, a vehicle transmission unit 223, and an operation control unit 224.

[0026] The vehicle speed calculation unit 221 calculates the traveling speed of the vehicle 10 (hereinafter, vehicle speed) using the output value of the wheel speed sensor 171. The vehicle speed calculation unit 221 performs a calculation process such as averaging the wheel speeds of the individual wheels, and then calculates the vehicle speed based on the wheel speed per unit time. Note that the method of calculating the vehicle speed is not limited to this. Also, at least a part of the functions of the vehicle speed calculation unit 221 may be executed by the wheel speed sensor 171 or the remote control device 5.

[0027] The vehicle acquisition unit 222 acquires information including at least a part of the information acquired by the internal sensor 170 and the external sensor 160, and the vehicle speed calculated by the vehicle speed calculation unit 221 (hereinafter, vehicle sensor information).

[0028] The vehicle transmission unit 223 transmits various information to an external device. For example, the vehicle transmission unit 223 transmits vehicle sensor information to the remote control device 5 together with vehicle identification information for identifying the multiple vehicles 10.

[0029] The operation control unit 224 receives control values ​​transmitted from the remote control device 5, and drives the drive device 11, the steering device 12, and the braking device 13 according to the received control values. In this way, the operation control unit 224 controls the traveling operation of the vehicle 10 according to instructions from the remote control device 5. Note that at least a part of the functions of the vehicle CPU 22 may be realized as one function of the remote control device 5 or the imaging device 9.

[0030] 3 is a diagram showing a schematic configuration of the remote control device 5. The remote control device 5 includes a remote communication unit 51 as a communication unit of the remote control device 5, a remote storage unit 53 as a storage unit of the remote control device 5, and a remote CPU 52 as a central processing unit of the remote control device 5. The remote communication unit 51, the remote storage unit 53, and the remote CPU 52 are connected to each other via an internal bus and an interface circuit.

[0031] The remote communication unit 51 communicably connects the vehicle control device 2, the position estimation device 7, and the imaging device 9 to the remote control device 5. The remote communication unit 51 is, for example, a wireless communication device. Note that the communication method by the remote communication unit 51 is not limited to this.

[0032] The remote storage unit 53 stores various information including various programs for controlling the operation of the remote control device 5. The remote storage unit 53 includes, for example, a RAM, a ROM, and a hard disk drive (HDD).

[0033] The remote CPU 52 deploys various programs stored in the remote storage unit 53 to function as a remote acquisition unit 521 , a control value creation unit 522 , and a remote transmission unit 523 .

[0034] The remote acquisition unit 521 acquires information related to the driving conditions of the vehicle 10 (hereinafter, driving information). The driving information includes, for example, captured images transmitted from the imaging device 9, vehicle coordinate points as information indicating the position of the vehicle 10 transmitted from the position estimation device 7, vehicle sensor information transmitted from the vehicle control device 2, and driving route information stored in advance in the remote storage unit 53. The driving route information is information indicating a target driving route of the vehicle 10 traveling in the remote automatic driving mode. Note that the type of information included in the driving information is not limited to this.

[0035] The control value creation unit 522 creates a control value that specifies the driving operation of the vehicle 10 by using the driving information acquired by the remote acquisition unit 521. Specifically, the control value creation unit 522 creates a control value (hereinafter, reference control value) for driving the vehicle 10 along the target driving route. In this embodiment, the reference control value includes a control value that specifies the acceleration in the forward direction of the vehicle 10 (hereinafter, acceleration control value) and a control value that specifies the steering angle of the vehicle 10 (hereinafter, steering angle control value). The reference control value may include a control value (hereinafter, direction control value) for switching the traveling direction of the vehicle 10 between the forward direction and the backward direction opposite to the forward direction. Note that, in other embodiments, the reference control value may include only the target driving route. In this case, the operation control unit 224 of the vehicle control device 2 determines the acceleration, steering angle, traveling direction, etc. of the vehicle 10 based on the information on the target driving route included in the reference control value, and drives the drive device 11, the steering device 12, and the brake device 13.

[0036] Furthermore, the control value creation unit 522 creates a control value (hereinafter, a corrected control value) for correcting the relative position of the vehicle 10 to the target travel route. Specifically, the control value creation unit 522 calculates the actual travel trajectory of the vehicle 10 by, for example, arranging a plurality of captured images captured by different imaging regions RG in a chronological order. Then, the control value creation unit 522 compares the target travel route with the actual travel trajectory of the vehicle 10 to calculate the difference of the travel trajectory with respect to the target travel route. Furthermore, the control value creation unit 522 creates a corrected control value for realizing the target travel route while analyzing the captured images so as to reduce the difference between the target travel route and the actual travel trajectory of the vehicle 10. In this embodiment, the corrected control value includes at least a steering angle control value. The corrected control value may include a direction control value. Note that, in other embodiments, the corrected control value may include only a corrected travel route. The corrected travel route is information indicating a travel route along which the vehicle 10 should travel in order to correct the relative position of the vehicle 10 onto the target travel route. In this case, the operation control unit 224 of the vehicle control device 2 determines the acceleration, steering angle, direction of travel, etc. of the vehicle 10 based on information regarding the corrected driving route contained in the modified control value, and drives the drive device 11, steering device 12, and braking device 13.

[0037] The remote transmission unit 523 transmits the control value created by the control value creation unit 522 to the vehicle 10 to be controlled as a traveling operation target. The remote transmission unit 523 may transmit the control value to the vehicle 10 via the position estimation device 7 or the imaging device 9. When the remote transmission unit 523 transmits the control value to the vehicle 10 via the imaging device 9, the remote transmission unit 523 transmits the control value to the imaging device 9. Then, the imaging device 9 transmits the control value received from the remote control device 5 to the vehicle 10. In this way, the vehicle 10 can receive the control value from the imaging device 9 that is closer to the vehicle 10. This makes it possible to reduce the influence of communication failure. Therefore, it is possible to reduce the possibility that the vehicle 10 traveling in the remote automatic driving mode will stop due to communication failure. At least a part of the functions of the remote CPU 52 may be realized as one function of the vehicle control device 2 or the imaging device 9.

[0038] FIG. 4 is a diagram showing a schematic configuration of the position estimation device 7. The position estimation device 7 adopts the position of a predetermined positioning point 10e (FIG. 1) of the vehicle 10 as the position of the vehicle 10. The position estimation device 7 includes a device communication unit 71 as a communication unit of the position estimation device 7, a device storage unit 73 as a storage unit of the position estimation device 7, and a device CPU 72 as a central processing unit of the position estimation device 7. The device communication unit 71, the device storage unit 73, and the device CPU 72 are connected to each other via an internal bus or an interface circuit. The position estimation device 7 may include a display device (not shown) that displays various information to the user. The display device is, for example, a liquid crystal display or an organic EL display.

[0039] The device communication unit 71 communicably connects the vehicle control device 2, the remote control device 5, and the imaging device 9 with the position estimation device 7. The device communication unit 71 is, for example, a wireless communication device. Note that the communication method by the device communication unit 71 is not limited to this.

[0040] The device storage unit 73 stores various information including various programs for controlling the operation of the position estimation device 7, a position detection model Md1, a vehicle type identification model Md2, a distortion correction parameter Pa1, and a perspective transformation parameter Pa2. Furthermore, the device storage unit 73 stores a vehicle database D1 and a camera database D2. The device storage unit 73 includes, for example, a RAM, a ROM, and a hard disk drive (HDD).

[0041] The position detection model Md1 is a trained machine learning model used to identify the position of the vehicle 10 included in the captured image. In this embodiment, the position detection model Md1 is a machine learning model that has been trained in advance to mask the vehicle 10 in the input image by inputting a captured image or various images obtained by performing various processes on the captured image. As an algorithm of the position detection model Md1, for example, a deep neural network (hereinafter, DNN) having a structure of a convolutional neural network (hereinafter, CNN) that realizes semantic segmentation and instance segmentation is used. As an example of a DNN used as the algorithm of the position detection model Md1, there is a DNN that performs instance segmentation such as YOLACT++. Note that the configuration of the position detection model Md1 is not limited to this. The position detection model Md1 may be a trained machine learning model that uses an algorithm other than a neural network, for example.

[0042] The vehicle type identification model Md2 is a trained machine learning model used to identify the type of the vehicle 10 included in the captured image. The type of the vehicle 10 referred to here means, for example, the type of the vehicle 10 classified by the vehicle name or model. The vehicle type identification model Md2 is a machine learning model that has been trained in advance to output information indicating the type of the vehicle 10 in the input image by inputting a captured image or various images obtained by performing various processes on the captured image. The vehicle type identification model Md2 learns feature amounts according to the type of the vehicle 10 in order to identify the type of the vehicle 10. The feature amounts according to the type of the vehicle 10 are, for example, the shape of the vehicle 10 and the vehicle size determined by the overall length, width, and height of the vehicle 10. In this embodiment, the vehicle type identification model Md2 outputs vehicle type identification information as information indicating the type of the vehicle 10 in the input image. The vehicle type identification information is a unique ID (identifier) ​​assigned to each type so as not to overlap between types in order to identify a plurality of types of the vehicle 10. For example, CNN is used as the algorithm of the vehicle type identification model Md2. The configuration of the vehicle type identification model Md2 is not limited to this. For example, the vehicle type identification model Md2 may be a trained machine learning model that uses an algorithm other than a neural network.

[0043] The distortion correction parameter Pa1 is a parameter used when correcting the distortion of the captured image. The perspective transformation parameter Pa2 is a parameter used when performing perspective transformation on the first mask image. The correction parameter Pa1 and the perspective transformation parameter Pa2 will be described in detail later.

[0044] The vehicle database D1 is information indicating vehicle parameters for each type of vehicle 10. The vehicle parameters are parameters determined according to the type of vehicle 10. The vehicle parameters are, for example, parameters related to the distance from a reference point (for example, the road surface 60 or a road boundary) on the road on which the vehicle 10 is traveling to the positioning point 10e of the vehicle 10. In this embodiment, the vehicle database D1 is a database that links vehicle type identification information with vehicle parameters for the vehicle 10 of the type identified by the vehicle type identification information.

[0045] The camera database D2 is information indicating, for each imaging device 9, imaging parameters calculated based on the position of the imaging device 9 in a global coordinate system. In this embodiment, the camera database D2 is a database that links camera identification information with imaging parameters for the imaging device 9 identified by the camera identification information. The imaging parameters are calculated based on the position of the imaging device 9 in the global coordinate system. At least a part of the various information stored in the device storage unit 73 may be stored in a storage unit of another device (for example, the vehicle storage unit 23, the remote storage unit 53, or the storage unit of the imaging device 9).

[0046] 5 is a diagram showing details of the device CPU 72 in the first embodiment. The device CPU 72 deploys various programs stored in the device storage unit 73, thereby functioning as a device acquisition unit 721 and a position calculation unit 722. Furthermore, the device CPU 72 deploys various programs stored in the device storage unit 73, thereby functioning as a vehicle type identification unit 723, a position conversion unit 724, and a device transmission unit 725.

[0047] The device acquisition unit 721 acquires various information. For example, the device acquisition unit 721 acquires a captured image Im1 including a vehicle 10 from the imaging device 9. The device acquisition unit 721 acquires vehicle parameters according to the type of the vehicle 10 identified by the vehicle type identification unit 723 by referring to a vehicle database D1 pre-stored in the device storage unit 73. The device acquisition unit 721 acquires imaging parameters for the imaging device 9 that is the source of the captured image to be analyzed by referring to a camera database D2 pre-stored in the device storage unit 73. Note that the type of information acquired by the device acquisition unit 721 is not limited to this.

[0048] The position calculation unit 722 calculates image coordinate points using the captured image. The image coordinate points are coordinate points that indicate the position of the vehicle 10 in an image coordinate system described below. The position calculation unit 722 includes a distortion correction unit 722a, a rotation processing unit 722b, a crop processing unit 722c, a detection unit 722d, a perspective transformation unit 722e, and a coordinate point calculation unit 722f.

[0049] The distortion correction unit 722a generates a corrected image by correcting the distortion of the captured image. The rotation processing unit 722b generates a rotated image by rotating the corrected image so that the direction of a vector (hereinafter, movement vector) indicating the moving direction of the vehicle 10 faces a predetermined direction. The crop processing unit 722c generates a processed image in which a necessary area including the vehicle area and the surrounding area is cut out by deleting an area (hereinafter, unnecessary area) other than the vehicle area corresponding to the vehicle 10 and a predetermined surrounding area of ​​the vehicle 10 (hereinafter, necessary area) from the rotated image. In this embodiment, when the vehicle 10 moves a distance exceeding a predetermined threshold, the crop processing unit 722c deletes the moved area as an unnecessary area, which is a moved area according to the distance moved by the vehicle 10, from the rotated image. As a result, the crop processing unit 722c generates a processed image in which an unmoved area including the vehicle 10 is cut out as a necessary area from the rotated image. The detection unit 722d detects the vehicle 10 included in the processed image using the position detection model Md1, and generates a first mask image to which a mask area that masks the vehicle 10 in the processed image is added. The perspective transformation unit 722e generates a second mask image by performing perspective transformation on the first mask image. The coordinate point calculation unit 722f corrects the first coordinate point using the second coordinate point to calculate an image coordinate point. The first coordinate point is a coordinate point in the image coordinate system of a specified vertex of a first circumscribed rectangle set in the mask area in the first mask image. The second coordinate point is a coordinate point in the image coordinate system of a vertex that indicates the same position as the first coordinate point, among the vertices of a second circumscribed rectangle set in the mask area in the second mask image. Note that the configuration of the position calculation unit 722 is not limited to this.

[0050] The vehicle type identification unit 723 identifies the type of the vehicle 10 included in the captured image. In this embodiment, the vehicle type identification unit 723 identifies the type of the vehicle 10 by using a vehicle type identification model Md2.

[0051] The position conversion unit 724 converts the image coordinate points into vehicle coordinate points using the imaging parameters and vehicle parameters acquired by the device acquisition unit 721. The vehicle coordinate points are coordinate points that indicate the position of the vehicle 10 in the global coordinate system. In other words, the position conversion unit 724 calculates the vehicle coordinate points using the imaging parameters for the imaging device 9 that acquired the captured image acquired by the device acquisition unit 721 and the vehicle parameters for the vehicle 10 of the type identified by the vehicle type identification unit 723.

[0052] The device transmission unit 725 transmits the vehicle coordinate points to the remote control device 5 as information indicating the position of the vehicle 10. The device transmission unit 725 may transmit various information other than the vehicle coordinate points to the remote control device 5. The device transmission unit 725 may also transmit various information to the vehicle control device 2. At least a part of the functions of the device CPU 72 may be realized as one function of the remote control device 5, the vehicle control device 2, or the imaging device 9.

[0053] A-2. Vehicle operation control method: 6 is a flowchart showing a first operation control method. The first operation control method is a method for controlling the traveling operation of the vehicle 10 in order to start traveling of the vehicle 10 in the remote automatic driving mode.

[0054] When preparations to start traveling in the remote automatic driving mode are completed (step S101: Yes), the vehicle transmitter 223 of the vehicle 10 transmits a preparation completion signal to the remote control device 5 (step S102).

[0055] When the remote acquisition unit 521 of the remote control device 5 receives the preparation completion signal (step S103: Yes), the remote acquisition unit 521 acquires the driving information (step S104). At this time, the driving information includes at least driving route information indicating a target driving route. After step S104, the control value creation unit 522 creates a reference control value using the driving information (step S105). After step S105, the remote transmission unit 523 transmits the reference control value created by the control value creation unit 522 to the vehicle 10 (step S106).

[0056] When the operation control unit 224 of the vehicle 10 receives the reference control value (step S107: Yes), the operation control unit 224 sets the driving route of the vehicle 10 as a target driving route and starts driving in the remote automatic driving mode (step S108). In other words, the operation control unit 224 drives the drive device 11, steering device 12, and braking device 13 mounted on the vehicle 10 so that the vehicle 10 drives along the target driving route.

[0057] Fig. 7 is a flowchart showing a second motion control method. The second motion control method is a method for correcting the driving route of the vehicle 10 that is driving in the remote automatic driving mode. The second motion control method is executed after the first motion control method shown in Fig. 6 is completed.

[0058] As shown in FIG. 7, when a predetermined correction timing is reached (step S201: Yes), the remote acquisition unit 521 of the remote control device 5 acquires the driving information (step S202). The correction timing is, for example, a timing when a predetermined time has elapsed since the timing when driving in the remote automatic driving mode was started, or a timing when a predetermined time has elapsed since the previous correction timing. At this time, the driving information includes at least driving route information indicating a target driving route and vehicle coordinate points. After step S202, the control value creation unit 522 creates a corrected control value using the driving information (step S203). After step S203, the remote transmission unit 523 transmits the corrected control value created by the control value creation unit 522 to the vehicle 10 (step S204).

[0059] When the operation control unit 224 of the vehicle 10 receives the corrected control value (step S205: Yes), the operation control unit 224 corrects the driving route of the vehicle 10 to the corrected driving route in accordance with the corrected control value, and continues driving in the remote automatic driving mode (step S206). In other words, the operation control unit 224 drives the drive device 11, steering device 12, and braking device 13 mounted on the vehicle 10 so that the vehicle 10 drives along the corrected driving route. When step S206 is completed, the process returns to step S201, and each step from step S201 to step S206 is repeatedly executed.

[0060] A-3. Vehicle position estimation method: Fig. 8 is a flowchart showing a position estimation method for the vehicle 10 in the first embodiment. The position of the vehicle 10 estimated by the position estimation method is transmitted to the remote control device 5 as one piece of travel information, and is used to create a control value for controlling the travel operation of the vehicle 10. Therefore, the position estimation method is executed, for example, before starting the second operation control method shown in Fig. 7. That is, in step S202 of the second operation control method, the position estimation method is executed at a timing when a vehicle coordinate point indicating the position of the vehicle 10 can be provided from the position estimation device 7 to the remote control device 5 as one piece of travel information.

[0061] FIG. 9 is a schematic diagram showing examples of various images when the position estimation method shown in FIG. 8 is executed. In FIG. 9, step numbers corresponding to the steps in FIG. 8 are assigned. In this embodiment, a case will be described in which the vehicle 10 moves (runs) along the traveling direction on a road surface 60 on which mesh-like lattice lines 61 are drawn along the Xg axis parallel to the traveling direction of the vehicle 10 and the Yg axis perpendicular to the Xg axis. The Xg axis and the Yg axis are each a coordinate axis of a global coordinate system. In other embodiments, the lattice lines 61 may be omitted.

[0062] In the position estimation method, first, an image acquisition step (step S31) is executed. The image acquisition step is a step of acquiring a captured image Im1 including a vehicle 10 that is a target of position estimation. In the image acquisition step, the device acquisition unit 721 acquires the captured image Im1 acquired by the imaging device 9.

[0063] After the image acquiring step, a position calculating step (step S32) is executed. The position calculating step is a step of calculating image coordinate points using the captured image Im1 acquired in the image acquiring step.

[0064] In the position calculation process, first, a distortion correction process (step S321) is executed. The distortion correction process is a process for correcting distortion of the captured image Im1. In the distortion correction process, the distortion correction unit 722a corrects the distortion of the captured image Im1 to generate a corrected image Im2. Specifically, the distortion correction unit 722a corrects the distortion of the captured image Im1, for example, by using a distortion correction parameter Pa1 stored in advance in the device storage unit 73. The distortion correction parameter Pa1 is, for example, a parameter related to position information of the grid lines 61 obtained by calibration. Note that the distortion correction method is not limited to this. The distortion correction parameter Pa1 may be any parameter other than the grid lines 61.

[0065] After the distortion correction step, a rotation processing step (step S322) is executed. The rotation processing step is a step of rotating the corrected image Im2 so that the direction of the movement vector for the vehicle 10 included in the corrected image Im2 faces a predetermined direction. In the rotation processing step, the rotation processing unit 722b rotates the corrected image Im2 so that the direction of the movement vector for the vehicle 10 included in the corrected image Im2 faces a predetermined direction. In this way, the rotation processing unit 722b generates a rotated image Im3. Specifically, the rotation processing unit 722b rotates the corrected image Im2 with the center of gravity of the vehicle 10 in the corrected image Im2 as the center of rotation so that the direction of the movement vector of the vehicle 10 faces upward on the screen of a display device (not shown) that displays the corrected image Im2. The movement of the feature point (center of gravity) of the vehicle 10 can be expressed as the direction of the movement vector by, for example, an optical flow method. The amount and direction of the movement vector of the vehicle 10 are estimated based on, for example, a change in position between image frames of a feature point of the vehicle 10 that is appropriately set on the corrected image Im2. Note that the rotation processing method is not limited to this.

[0066] After the rotation process, a crop process (step S323) is executed. The crop process is a process of generating a processed image Im4 including a target area from the rotated image Im3. In this embodiment, in the crop process, when the vehicle 10 moves a distance exceeding a predetermined threshold, the crop processing unit 722c deletes the moved area A2 corresponding to the distance moved by the vehicle 10 from the rotated image Im3 as an unnecessary area. As a result, the crop processing unit 722c generates a processed image Im4 by cutting out the unmoved area A1 including the vehicle 10 as a target area from the rotated image Im3. At this time, the crop processing unit 722c estimates the moved area A2 by recognizing the distance moved by the vehicle 10 from, for example, the estimated movement vector amount of the vehicle 10. Note that, in the position calculation process, either the rotation process or the crop process may be executed first. Also, the crop process method is not limited to this.

[0067] As shown in FIG. 8, following the cropping process, a detection process (step S324) is executed. The detection process is a process of detecting the outer shape (outline) of the vehicle 10 included in the processed image Im4 using the position detection model Md1. In the detection process, the detection unit 722d inputs the processed image Im4 to the position detection model Md1. As a result, as shown in FIG. 9, the detection unit 722d detects the vehicle 10 included in the processed image Im4, and generates a first mask image Im5 to which a mask area Ms that masks the vehicle 10 in the processed image Im4 is added. Note that the method of detecting the outer shape of the vehicle 10 is not limited to this.

[0068] As shown in FIG. 8, after the detection step, a perspective transformation step (step S325) is executed. The perspective transformation step is a step of performing perspective transformation on the first mask image Im5 shown in FIG. 9. In the perspective transformation step, the perspective transformation unit 722e performs perspective transformation on the first mask image Im5 to generate a second mask image Im6. Specifically, the perspective transformation unit 722e performs perspective transformation on the first mask image Im5 into a bird's-eye image viewed from a viewpoint above the vehicle 10 (for example, directly above the vehicle 10) that is substantially perpendicular to the road surface 60, for example, using a perspective transformation parameter Pa2 stored in advance in the device storage unit 73. The perspective transformation parameter Pa2 is, for example, a parameter related to position information and internal parameters of the imaging device 9 obtained by calibration. As a result, the perspective transformation unit 722e generates a second mask image Im6 expressed in an image coordinate system from the first mask image Im5 expressed in a camera coordinate system. The image coordinate system is a coordinate system having a point on the image plane projected by the perspective transformation as the origin and coordinate axes indicated by the Xi axis and the Yi axis perpendicular to the Xi axis. Note that the perspective transformation method is not limited to this. The perspective transformation parameter Pa2 may be any parameter other than the above.

[0069] As shown in Fig. 8, after the perspective transformation step, a coordinate point calculation step (step S326) is executed. Fig. 10 is a diagram for explaining the details of the coordinate point calculation step. The coordinate point calculation step is a step for calculating an image coordinate point P3.

[0070] In the coordinate point calculation step, the coordinate point calculation unit 722f first acquires a base coordinate point P0 from a first circumscribing rectangle R1 set in a mask area Ms in a first mask image Im5, which is an image before perspective transformation. FIG. 11 is a diagram for explaining a method for acquiring the base coordinate point P0. In order to acquire the base coordinate point P0, the coordinate point calculation unit 722f first sets a base circumscribing rectangle R0 for the mask area Ms in the first mask image Im5. Next, the coordinate point calculation unit 722f rotates the first mask image Im5 by a required rotation amount with the center of gravity C of the mask area Ms as the rotation center so that the direction of the movement vector V of the vehicle 10 corresponding to the mask area Ms in the first mask image Im5 faces a predetermined direction. The predetermined direction is, for example, the upward direction on the screen of a display device (not shown). Next, the coordinate point calculation unit 722f sets a first circumscribing rectangle R1 for the mask area Ms of the rotated first mask image Im5 so that the long side is parallel to the direction of the movement vector V. Next, the coordinate point calculation unit 722f rotates the first mask image Im5 to which the first circumscribing rectangle R1 has been added by the above-mentioned rotation amount with the center of gravity C of the mask area Ms as the rotation center. As a result, the coordinate point calculation unit 722f acquires, as a base coordinate point P0, the coordinate point of the vertex that is one of the four vertices of the first circumscribing rectangle R1 and has the coordinates that are closest to the positioning point 10e of the vehicle 10. Then, as shown in FIG. 10, the coordinate point calculation unit 722f performs perspective transformation on the first mask image Im5 after the reverse rotation, that is, the first mask image Im5 after the base coordinate point P0 is acquired. As a result, the coordinate point calculation section 722f obtains the first coordinate point P1 as the coordinate point corresponding to the base coordinate point P0 in the first circumscribed rectangle R1 transformed by the perspective transformation.

[0071] Furthermore, the coordinate point calculation unit 722f sets a second circumscribing rectangle R2 for the mask region Ms in the second mask image Im6 obtained by performing perspective transformation on the first mask image Im5. Then, the coordinate point calculation unit 722f acquires, as the second coordinate point P2, a vertex indicating the same position as the first coordinate point P1 among the vertices of the second circumscribing rectangle R2. Specifically, the coordinate point calculation unit 722f acquires, as the second coordinate point P2, one of the four vertices of the second circumscribing rectangle R2, a vertex having coordinates that are closest to the positioning point 10e of the vehicle 10. That is, the first coordinate point P1 and the second coordinate point P2 are coordinate points indicating the same position, and therefore have a correlation with each other.

[0072] Furthermore, the coordinate point calculation unit 722f performs correction to replace the coordinates (Xi1, Yi1) of the first coordinate point P1 with the coordinates (Xi2, Yi2) of the second coordinate point P2 depending on the magnitude relationship between the coordinate values ​​of the first coordinate point P1 and the second coordinate point P2. If the coordinate value Xi1 of the first coordinate point P1 in the Xi axis direction is greater than the coordinate value Xi2 of the second coordinate point P2 in the Xi axis direction (Xi1>Xi2), the coordinate point calculation unit 722f replaces the coordinate value Xi1 of the first coordinate point P1 in the Xi axis direction with the coordinate value Xi2 of the second coordinate point P2 in the Xi axis direction. When the coordinate value Yi1 of the first coordinate point P1 in the Yi-axis direction is larger than the coordinate value Yi2 of the second coordinate point P2 in the Yi-axis direction (Yi1>Yi2), the coordinate point calculation unit 722f replaces the coordinate value Yi1 of the first coordinate point P1 in the Yi-axis direction with the coordinate value Yi2 of the second coordinate point P2 in the Yi-axis direction. In this embodiment, the coordinate value Xi1 of the first coordinate point P1 in the Xi-axis direction is larger than the coordinate value Xi2 of the second coordinate point P2 in the Xi-axis direction. In addition, the coordinate value Yi1 of the first coordinate point P1 in the Yi-axis direction is smaller than the coordinate value Yi2 of the second coordinate point P2 in the Yi-axis direction. Therefore, the image coordinate point P3 has coordinates (Xi2, Yi1). In this way, the coordinate point calculation unit 722f calculates the image coordinate point P3 indicating the position (estimated position) of the vehicle 10 in the image coordinate system by correcting the first coordinate point P1 using the second coordinate point P2. It should be noted that the method of calculating the image coordinate point P3 is not limited to this.

[0073] As shown in FIG. 8, following the position calculation step, a vehicle type identification step (step S33) is executed. The vehicle type identification step is a step of identifying the type of vehicle 10 included in the captured image Im1. In the vehicle type identification step, the vehicle type identification unit 723 inputs the captured image Im1 to the vehicle type identification model Md2. As a result, the vehicle type identification unit 723 acquires vehicle type identification information indicating the type of the vehicle 10 included in the captured image Im1, thereby identifying the type of the vehicle 10. Note that the method of identifying the type of the vehicle 10 is not limited to this.

[0074] After the vehicle type identification step, a parameter acquisition step (step S34) is executed. The parameter acquisition step is a step of acquiring imaging parameters and vehicle parameters to be substituted into the relational expression. In the parameter acquisition step, the device acquisition unit 721 refers to the vehicle database D1 to acquire values ​​of vehicle parameters linked to vehicle type identification information indicating the type identified by the vehicle type identification unit 723. In this embodiment, the height h (FIG. 12 described later) of the positioning point 10e of the vehicle 10 from the road surface 60 is different for each type of vehicle 10. Therefore, in this embodiment, the vehicle parameter acquired in the parameter acquisition step is the height h of the positioning point 10e of the vehicle 10 of the type identified by the vehicle type identification unit 723 from the road surface 60. Furthermore, the device acquisition unit 721 refers to the camera database D2 to acquire values ​​of imaging parameters linked to the camera identification information received together with the captured image Im1 including the vehicle 10 to be the target of position estimation. In this embodiment, in order to convert image coordinate points into vehicle coordinate points using the similarity relationship between the vehicle parameters and the imaging parameters, the imaging parameters acquired in the parameter acquisition step are as follows. In this case, the imaging parameters are the height H (FIG. 12 described later) from the road surface 60 of the imaging device 9 that acquired the captured image Im1 including the vehicle 10 whose position is to be estimated, that is, the height H relative to the road surface 60 on which the vehicle 10 is located. Note that the method of acquiring the vehicle parameters and the imaging parameters is not limited to this.

[0075] As shown in FIG. 8, after the parameter acquisition step, a position conversion step (step S37) is executed. The position conversion step is a step of converting the image coordinate point P3 into a vehicle coordinate point to calculate a vehicle coordinate point indicating the position of the vehicle 10 to be estimated in the global coordinate system. In the position conversion step, the position conversion unit 724 converts the image coordinate point into the vehicle coordinate point using the vehicle parameters and the imaging parameters acquired in the parameter acquisition step. In this embodiment, the position conversion unit 724 converts the image coordinate point P3 into the vehicle coordinate point using a relational expression expressed by Equations (1) to (3) described later, which uses the vehicle coordinate point as a response variable and includes the image coordinate point P3, the imaging parameters, and the vehicle parameters as explanatory variables. In this case, the position conversion unit 724 substitutes the coordinate value of the image coordinate point P3 calculated by the position calculation unit 722 into the relational expression expressed by Equations (1) to (3). The position conversion unit 724 substitutes the values ​​of the imaging parameters acquired by the device acquisition unit 721, i.e., the values ​​of the imaging parameters corresponding to the imaging device 9 that acquired the captured image Im1, into the relational expressions expressed by equations (1) to (3). Furthermore, the position conversion unit 724 substitutes the values ​​of the vehicle parameters acquired by the device acquisition unit 721, i.e., the values ​​of the vehicle parameters corresponding to the type of the vehicle 10 whose position is to be estimated, into the relational expressions expressed by equations (1) to (3).

[0076] FIG. 12 is a first diagram for explaining the details of the position conversion process in the first embodiment. FIG. 12 illustrates a state in which the vehicle 10 is viewed from the left side. FIG. 13 is a second diagram for explaining the details of the position conversion process in the first embodiment. FIG. 13 illustrates a state in which the vehicle 10 is viewed from the roof side. The global coordinate system shown in FIG. 12 and FIG. 13 is a coordinate system having a fixed coordinate point Pf indicating an arbitrary reference position of the road surface 60 as the origin, and coordinate axes indicated by the Xg axis and the Yg axis perpendicular to the Xg axis. The imaging coordinate point Pc is the position of the imaging device 9 that acquired the captured image Im1 used in calculating the image coordinate point P3, and is a coordinate point indicating the position of the imaging device 9 in the global coordinate system. The fixed coordinate point Pf and the imaging coordinate point Pc are stored in advance in the device storage unit 73.

[0077] 12, the observation distance on the XgYg plane between the position of the imaging device 9 and the position of the vehicle 10 (image coordinate point P3) is set to Do. The observation error is set to ΔD. The height [m] of the imaging device 9 from the road surface 60 as an acquired vehicle parameter is set to H. The height [m] of the positioning point 10e of the vehicle 10 from the road surface 60 as an acquired vehicle parameter is set to h. In this case, the observation error ΔD can be expressed by the following equation (1). ΔD=h / H×Do Equation (1) That is, the larger the observation distance Do, the larger the observation error ΔD.

[0078] Next, if the actual distance between the position of the imaging device 9 and the position of the positioning point 10e of the vehicle 10 (hereinafter, referred to as a first actual distance) is D, the first actual distance D can be expressed by the following formula (2). D=Do×(1-h / H) Equation (2) That is, the first actual distance D is determined by the observation distance Do, the height H of the imaging device 9 as an imaging parameter, and the height h of the positioning point 10e of the vehicle 10 as a vehicle parameter.

[0079] Then, as shown in FIG. 13, if the estimated distance between the reference position and the estimated position of the vehicle 10 is Dp, and the actual distance between the reference position and the position of the vehicle 10 (hereinafter, the second actual distance) is Dt, the second actual distance Dt can be expressed by the following equation (3). Dt = Dp × (1-h / H) Equation (3)

[0080] Here, the estimated distance Dp can be calculated using an actual distance (hereinafter, the third actual distance Dc) calculated from the fixed coordinate point Pf and the image coordinate point Pc, the image coordinate point P3, and the fixed coordinate point Pf. Therefore, the position conversion unit 724 can calculate the vehicle coordinate point Pv using the second actual distance Dt and the fixed coordinate point Pf calculated by correcting the estimated distance Dp using the above formula (3). At this time, the calculated vehicle coordinate point Pv is a coordinate point indicating the position of the vehicle 10 in the global coordinate system, and therefore corresponds to the position of the vehicle 10 in the real space. In this way, the position conversion unit 724 can convert the image coordinate point P3 to the vehicle coordinate point Pv by substituting the acquired image capture parameter value and vehicle parameter value into the relational expressions expressed by formulas (1) to (3). Note that, in order for the remote control device 5 to generate an appropriate control value according to the current location of the vehicle 10, the above-described method for estimating the position of the vehicle 10 is repeatedly executed, for example, at predetermined time intervals.

[0081] According to the first embodiment, when the vehicle 10 is made to travel automatically by remote control, the position of the vehicle 10 can be estimated using the captured image Im1 acquired by capturing an image of the imaging area RG including the vehicle 10 from outside the vehicle 10. In this way, the position of the vehicle 10 can be estimated without installing any installation object such as a sensor, a marker, a transceiver, etc., used for estimating the position of the vehicle 10 on the vehicle 10. In addition, the position of the vehicle 10 can be estimated without mounting the position estimation device 7 on the vehicle 10. This can increase the versatility of the position estimation device 7.

[0082] Furthermore, according to the first embodiment, when the position of the vehicle 10 is estimated using the captured image Im1, the distortion of the captured image Im1 can be corrected. This makes it possible to further improve the accuracy of estimating the position of the vehicle 10.

[0083] Furthermore, according to the first embodiment, when the position of the vehicle 10 is estimated using the captured image Im1, the corrected image Im2 generated by processing the captured image Im1 can be rotated so that the direction of the movement vector V of the vehicle 10 faces a predetermined direction. In this way, the vehicle 10 included in the captured image Im1 can be detected with the direction of the movement vector V being unified. This can further improve the accuracy of estimating the position of the vehicle 10.

[0084] Furthermore, according to the first embodiment, a crop process is performed to cut out a necessary area including the vehicle 10 from the rotated image Im3 generated by processing the captured image Im1, thereby generating a processed image Im4 from which unnecessary areas have been removed. In this way, the vehicle 10 can be detected in a state in which elements other than the vehicle 10 to be detected have been removed from the image. This allows the vehicle 10 included in the captured image Im1 to be detected with high accuracy, thereby further improving the accuracy of estimating the position of the vehicle 10.

[0085] Furthermore, according to the first embodiment, a processed image Im4 can be generated by performing a cropping process to cut out an unmoved area A1 including the vehicle 10 from a rotated image Im3 generated by processing the captured image Im1. In this way, the area of ​​the vehicle 10 in the image can be made larger than when the cropping process is not performed. This makes it easier to detect a vehicle 10 that is farther away from the imaging device 9. Therefore, the accuracy of image processing for a vehicle 10 that is farther away from the imaging device 9 can be further improved.

[0086] According to the first embodiment, the contour of the vehicle 10 can be detected by inputting the processed image Im4 generated by processing the captured image Im1 to the position detection model Md1, which is a trained machine learning model. As a result, a first mask image Im5 can be generated in which a mask area Ms is added by masking an area corresponding to the vehicle 10 in the processed image Im4. At this time, according to the first embodiment, a DNN having a CNN structure that realizes semantic segmentation and instance segmentation can be used as an algorithm for the position detection model Md1. This makes it possible to suppress a decrease in the estimation accuracy of the position of the vehicle 10 due to the diversity of the background area in the captured image Im1. The background area is an area in the captured image Im1 other than the area corresponding to the vehicle 10.

[0087] Furthermore, according to the first embodiment, the second mask image Im6 can be generated by performing perspective transformation on the first mask image Im5, thereby making it possible to transform the camera coordinate system into the image coordinate system.

[0088] According to the first embodiment, the first circumscribing rectangle R1 is set for the mask area Ms before the perspective transformation of the first mask image Im5, so that the base coordinate point P0, which is the vertex of the first circumscribing rectangle R1 having the coordinates closest to the positioning point 10e of the vehicle 10, can be calculated. Then, the first mask image Im5 after the calculation of the base coordinate point P0 is perspective transformed, so that the first coordinate point P1, which is the coordinate point corresponding to the base coordinate point P0, can be calculated. Furthermore, the second circumscribing rectangle R2 is set for the mask area Ms after the perspective transformation of the first mask image Im5, so that the second coordinate point P2, which is the vertex of the second circumscribing rectangle R2 having the coordinates closest to the positioning point 10e of the vehicle 10, can be calculated. Then, the first coordinate point P1 is corrected using the second coordinate point P2, so that the image coordinate point P3 can be calculated. In this way, the image coordinate point P3 can be calculated more accurately by comparing and correcting the coordinate points before and after the perspective transformation. This can further improve the estimation accuracy of the position of the vehicle 10.

[0089] Furthermore, according to the first embodiment, the type of the vehicle 10 included in the captured image Im1 can be identified by inputting the captured image Im1 or various images Im2 to Im4 generated by processing the captured image Im1 to the vehicle type identification model Md2, which is a trained machine learning model. In other words, the type of the vehicle 10 can be identified using machine learning.

[0090] Moreover, according to the first embodiment, by referring to the vehicle database D1 pre-stored in the device storage unit 73, in which each type of vehicle 10 is associated with a vehicle parameter, it is possible to acquire the value of the vehicle parameter corresponding to the type of vehicle 10. Furthermore, by referring to the camera database D2 pre-stored in the device storage unit 73, in which each imaging device 9 is associated with an imaging parameter, it is possible to acquire the value of the imaging parameter related to the imaging device 9 corresponding to the captured image Im1.

[0091] Furthermore, according to the first embodiment, when estimating the position of the vehicle 10 using the captured image Im1, the position of the vehicle 10 can be estimated using vehicle parameters determined according to the type of the vehicle 10. That is, the position of the vehicle 10 can be estimated taking into consideration the type of the vehicle 10. This makes it possible to prevent a decrease in the accuracy of estimating the position of the vehicle 10 even when the vehicle rank, etc., differs depending on the type of the vehicle 10. In other words, when estimating the position of the vehicle 10 using the captured image Im1, the position of the vehicle 10 can be estimated with high accuracy by using the vehicle parameters determined according to the type of the vehicle 10.

[0092] Moreover, according to the first embodiment, the position of the vehicle 10 can be estimated taking into consideration the type of the vehicle 10. As a result, even when a plurality of types of vehicles 10 are traveling in the remote automated driving mode, the position of the vehicle 10 can be appropriately estimated according to the type of the vehicle 10.

[0093] According to the first embodiment, when converting the image coordinate point P3 into the vehicle coordinate point Pv, the vehicle coordinate point Pv is used as a response variable, and the vehicle parameter value according to the type of the vehicle 10 and the imaging parameter value corresponding to the captured image Im1 are substituted into a relational expression including the image coordinate point P3, the imaging parameters, and the vehicle parameters as explanatory variables. This allows the image coordinate point P3 to be converted into the vehicle coordinate point Pv.

[0094] Moreover, according to the first embodiment, the imaging parameter is the height H of the imaging device 9 from the road surface 60, which is calculated based on the position of the imaging device 9 in the global coordinate system. Moreover, the vehicle parameter is the height h of the predetermined positioning point 10e of the vehicle 10 from the road surface 60. In this way, the observation error ΔD can be calculated based on the similarity relationship between the imaging parameter and the vehicle parameter. Then, the image coordinate point P3 can be converted into the vehicle coordinate point Pv using the calculated observation error ΔD.

[0095] Moreover, according to the first embodiment, the position of the vehicle 10 can be estimated without using data from GNSS satellites, etc. As a result, even when the vehicle 10 travels on a road surface 60 that is installed indoors, etc., where data from GNSS satellites is difficult to reach, the position of the vehicle 10 can be estimated.

[0096] B. Second embodiment: FIG. 14 is a diagram showing details of the device CPU 72a in the second embodiment. FIG. 15 is a flowchart showing a position estimation method for the vehicle 10 in the second embodiment. In this embodiment, a conversion method for converting the image coordinate point P3 into the vehicle coordinate point Pv is different from that in the first embodiment. Specifically, in this embodiment, the device CPU 72 further includes a correction value setting unit 726. The correction value setting unit 726 sets the correction parameter by calculating the difference between the vehicle parameter acquired by the device acquisition unit 721 and the reference parameter. Then, the position conversion unit 724 converts the image coordinate point P3 into the vehicle coordinate point Pv using a relational expression expressed by Equation (4) to Equation (6) described later, which uses the vehicle coordinate point Pv as an objective variable and includes the image coordinate point P3, the imaging parameter, the reference parameter, and the correction parameter as explanatory variables. The reference parameter is a parameter indicating a reference value of the vehicle parameter. The reference parameter is, for example, a vehicle parameter of a vehicle 10 of a predetermined type. The reference parameter is, for example, stored in advance in the device storage unit 73. The correction parameters are parameters determined according to the type of vehicle 10, and indicate correction values ​​for the reference parameters. Thus, in this embodiment, part of the functions of the device CPU 72 and part of the method for estimating the position of the vehicle 10 are different from those in the first embodiment. The other configurations are the same as those in the first embodiment. The same reference numerals are used for the steps that are the same as those in the first embodiment, and the description thereof will be omitted.

[0097] As shown in FIG. 15, following the parameter acquisition step, a correction value setting step (step S36) is executed. In the correction value setting step, the correction value setting unit 726 calculates the difference between the vehicle parameters acquired by the device acquisition unit 721 and the reference parameters, thereby setting a correction parameter determined according to the type of vehicle 10 identified by the vehicle type identification unit 723. If the correction parameter is larger than the reference parameters, the correction parameter is a positive value. If the correction parameter is smaller than the reference parameters, the correction parameter is a negative value. If the correction parameter is the same as the reference parameters, the correction parameter is zero.

[0098] After the correction value setting step, a position conversion step (step S38) is executed. In the position conversion step, the position conversion unit 724 converts the image coordinate point P3 into the vehicle coordinate point Pv using the relational expressions expressed by the formulas (4) to (6). Specifically, the position conversion unit 724 substitutes the coordinate value of the image coordinate point P3 calculated by the position calculation unit 722 into the relational expressions expressed by the formulas (4) to (6). The position conversion unit 724 substitutes the value of the imaging parameter acquired by the device acquisition unit 721 into the relational expressions expressed by the formulas (4) to (6). The position conversion unit 724 substitutes the reference parameter stored in advance in the device storage unit 73 into the relational expressions expressed by the formulas (4) to (6). Furthermore, the position conversion unit 724 substitutes the correction parameter set by the correction value setting unit 726, i.e., the correction parameter according to the type of the vehicle 10 to be the target of position estimation, into the relational expressions expressed by the formulas (4) to (6). The position conversion process in this embodiment will be described in detail below.

[0099] Fig. 16 is a first diagram for explaining the details of the position conversion process in the second embodiment. Fig. 16 illustrates a state in which the vehicle 10 is viewed from the left side. Fig. 17 is a second diagram for explaining the details of the position conversion process in the second embodiment. Fig. 17 illustrates a state in which the vehicle 10 is viewed from the roof side.

[0100] As shown in FIG. 16, the observation distance on the XgYg plane between the position of the imaging device 9 and the position of the vehicle 10 (image coordinate point P3) is Do. The observation error is ΔD. The height [m] of the imaging device 9 from the road surface 60 as the acquired imaging parameter is H. The height [m] of the positioning point 10e of the vehicle 10 from the road surface 60 as the acquired vehicle parameter is h. The reference parameter is hs. In this embodiment, the reference parameter hs is the height [m] of the positioning point 10e of a vehicle 10 of a predetermined type from the road surface 60. The correction parameter is Δh. In this embodiment, the correction parameter Δh is the difference between the height hs as the reference parameter and the height h of the positioning point 10e of the vehicle 10 to be estimated from the road surface 60. In this case, the observation error ΔD can be expressed by the following formula (4). ΔD=(hs+Δh) / H×Do Equation (4) That is, the larger the observation distance Do, the larger the observation error ΔD.

[0101] Next, if a first actual distance, which is an actual distance between the position of the imaging device 9 and the position of the positioning point 10e of the vehicle 10, is D, the first actual distance D can be expressed by the following equation (5). D=Do×[{1-(hs+Δh)} / H] Equation (5) That is, the first actual distance D is determined by the observation distance Do, the height H of the imaging device 9 as an imaging parameter, and the correction parameter Δh.

[0102] Then, as shown in FIG. 17, if the estimated distance between the reference position and the estimated position of the vehicle 10 is Dp, and the second actual distance, which is the actual distance between the reference position and the position of the vehicle 10, is Dt, the second actual distance Dt can be expressed by the following equation (6). Dt=Dp×[{1-(hs+Δh)} / H] Equation (6)

[0103] Here, the estimated distance Dp can be calculated using the third actual distance Dc, the image coordinate point P3, and the fixed coordinate point Pf, as in the first embodiment. Therefore, the position conversion unit 724 can calculate the vehicle coordinate point Pv using the second actual distance Dt and the fixed coordinate point Pf, which are obtained by correcting the estimated distance Dp using the above formula (6). In this way, the image coordinate point P3 can be converted into the vehicle coordinate point Pv by substituting the acquired imaging parameter value and the set correction parameter Δh value into the relational expressions expressed by formulas (4) to (6).

[0104] According to the second embodiment, when converting the image coordinate point P3 into the vehicle coordinate point Pv, the vehicle coordinate point Pv is used as a response variable, and the imaging parameters corresponding to the captured image Im1 and the correction parameter Δh set according to the type of the vehicle 10 are substituted into a relational expression including the image coordinate point P3, the imaging parameters, the reference parameter hs, and the correction parameter Δh as explanatory variables. This makes it possible to convert the image coordinate point P3 into the vehicle coordinate point Pv.

[0105] C. Other embodiments: C-1. Alternative embodiment 1: In another embodiment, when the type of the vehicle 10 identified by the vehicle type identification unit 723 is a predetermined type, the correction value setting unit 726 may set the correction parameter Δh to zero without calculating the difference between the vehicle parameter and the reference parameter hs. The predetermined type is, for example, a type in which the difference from the reference parameter hs is less than a predetermined threshold. In this case, the vehicle database D1 may further include judgment information for judging whether the type of the vehicle 10 identified by the vehicle type identification unit 723 is a type in which the difference from the reference parameter hs is less than a predetermined threshold. The judgment information is, for example, information in which the vehicle type identification information and the correction necessity information are linked. The correction necessity information is information indicating whether the type is a type in which the difference from the reference parameter hs is less than a predetermined threshold. The correction value setting unit 726 judges whether to set the correction parameter Δh to zero by referring to the correction necessity information corresponding to the vehicle type identification information indicating the type of the vehicle 10 identified by the vehicle type identification unit 723. In this embodiment, when the difference between the vehicle parameter of the vehicle 10 to be the subject of position estimation and the reference parameter hs is less than a predetermined threshold, the position of the vehicle 10 can be estimated without taking into account the correction parameter Δh according to the type of the vehicle 10. In other words, when the value of the vehicle parameter of the vehicle 10 to be the subject of position estimation is equal to or close to the value of the reference parameter hs, the position of the vehicle 10 can be estimated without correcting the vehicle parameter. In this way, the processing content in the step of setting the correction parameter Δh can be simplified.

[0106] C-2. Alternative embodiment 2: In another embodiment, the position estimation device 7 may further include a direction information generating unit as one function of the device CPU 72. The direction information generating unit generates direction information indicating the moving direction of the vehicle 10 using a plurality of vehicle coordinate points Pv obtained at different times for the same vehicle 10. In other words, the direction information generating unit generates direction information using a plurality of vehicle coordinate points Pv obtained at different timings for the same vehicle 10. Specifically, the direction information generating unit generates direction information by, for example, arranging the vehicle coordinate points Pv generated from a plurality of captured images Im1 captured by different imaging devices 9 in a chronological order. In this manner, the actual traveling trajectory of the vehicle 10 can be calculated. In addition, in this manner, the traveling direction of the vehicle 10 can be estimated. As a result, for example, when the position estimation method shown in FIG. 8 and FIG. 15 is repeatedly executed at predetermined times, the moving direction of the traveling vehicle 10 can be estimated using the direction information.

[0107] C-3. Alternative embodiment 3: In another embodiment, the vehicle type identification unit 723 may identify the type of vehicle 10 included in the captured image Im1 by using schedule information stored in advance in the device storage unit 73. The schedule information is information indicating which type of vehicle 10 is scheduled to travel at which position on the target travel route at which time. The schedule information is, for example, information linking vehicle type identification information, imaging position information, and timing information. The imaging position information is information indicating the imaging area RG of the imaging device 9 identified by the camera identification information. Therefore, the imaging position information is information indicating the position range on the target travel route. The timing information is information indicating the scheduled time when the vehicle 10 of the type identified by the vehicle type identification information travels in the imaging area RG identified by the imaging position information. Note that the schedule information is not limited to this. In the case where a plurality of vehicles 10 of one type are continuously produced in a factory, and the captured image Im1 is acquired by the imaging device 9 to estimate the position of the vehicle 10 traveling on the target travel route in the factory during the production process, the schedule information may be the following information. In this case, the schedule information may include process position information instead of the imaging position information. The process position information is information indicating the position of a work site where a production process specified by process identification information (process ID) that identifies a plurality of production processes executed in the production process of the vehicle 10 is executed. Therefore, the process position information is information indicating a position range on the target driving route. In addition, when the schedule information includes the process position information instead of the imaging position information, the timing information is information indicating a scheduled time for executing the production process specified by the process identification information for the vehicle 10 of the type specified by the vehicle type identification information. That is, in this case, the schedule information is production plan information indicating a production plan for each type of vehicle 10. In this form, the type of the vehicle 10 included in the captured image Im1 can be identified using the schedule information. As a result, the type of the vehicle 10 included in the captured image Im1 can be identified without performing image analysis, and therefore the processing load of the vehicle type identification unit 723 in the vehicle type identification process can be reduced.

[0108] C-4. Alternative embodiment 4: In the position calculation process, the distortion correction process is not an essential process. For example, in the position calculation process, if the distortion correction process is not performed and each process after the rotation process is performed, the rotation processing unit 722b performs the following process. In this case, in the rotation process, the rotation processing unit 722b rotates the captured image Im1 instead of the corrected image Im2. Even in this form, the image coordinate point P3 can be calculated.

[0109] C-5. Alternative embodiment 5: In the position calculation process, the rotation process is not an essential process. For example, when the distortion correction process and the rotation process are not performed in the position calculation process, the crop processing unit 722c performs the following process. In other words, when only the processes after the crop process are performed in the position calculation process, the crop processing unit 722c performs the following process. In this case, the crop processing unit 722c performs crop processing on the captured image Im1 instead of the rotated image Im3. Even in this form, the image coordinate point P3 can be calculated.

[0110] C-6. Alternative embodiment 6: In the position calculation process, the cropping process is not an essential process. For example, when the distortion correction process, the rotation process, and the cropping process are not performed in the position calculation process, the detection unit 722d performs the following process. In other words, when only the detection process and the subsequent processes are performed in the position calculation process, the detection unit 722d performs the following process. In this case, the detection unit 722d generates a first mask image Im5 by masking the vehicle 10 included in the captured image Im1 instead of the processed image Im4. Even in this form, the image coordinate point P3 can be calculated.

[0111] C-7. Alternative embodiment 7: In another embodiment, the captured image Im1 may include a plurality of vehicles 10. In this case, the device CPU 72 may further include a deletion unit that deletes, for example, a mask region Ms of the vehicle 10 that is not a target of position estimation from the first mask image Im5. For example, the deletion unit determines that, among the mask regions Ms generated in the detection process, a mask region Ms that exists outside the recognition target region is a mask region Ms of the vehicle 10 that is not a target of position estimation, and deletes it from the first mask image Im5. The recognition target region is, for example, a predetermined region in the first mask image Im5 in which the vehicle 10 moves. The predetermined region in which the vehicle 10 moves is, for example, a region corresponding to the region of the grid line 61. The recognition target region is stored in advance in the device storage unit 73. In this embodiment, when a plurality of vehicles 10 are captured in the captured image Im1, the influence of the vehicle 10 that is not a target of position estimation can be eliminated. This can improve the estimation accuracy of the position of the vehicle 10.

[0112] C-8. Other embodiment 8: In another embodiment, the captured image Im1 may include a plurality of vehicles 10. In this case, a DNN that performs instance segmentation such as YOLACT++ may be used as an algorithm for the position detection model Md1. In this embodiment, the plurality of vehicles 10 included in the captured image Im1 can be classified, and a first mask image Im5 in which each vehicle 10 is masked can be generated. As a result, when the captured image Im1 includes a plurality of vehicles 10, the vehicle 10 to be the target of position estimation can be selected, and the position of the selected vehicle 10 can be estimated.

[0113] C-9. Other embodiment 9: In another embodiment, the captured image Im1 may include a plurality of vehicles 10. In this case, the vehicle type identification unit 723 uses a vehicle type identification model Md2 capable of classifying a plurality of vehicles 10 included in the captured image Im1 to identify the type of the vehicle 10 to be the target of the position estimation without identifying the type of the vehicle 10 that is not the target of the position estimation. In this embodiment, even if the captured image Im1 includes a plurality of vehicles 10, it is possible to identify the type of the vehicle 10 to be the target of the position estimation.

[0114] C-10. Other embodiment 10: In another embodiment, the position estimation device 7 may estimate the position of the vehicle 10 that is stationary. When estimating the position of the vehicle 10 that is stationary, the position estimation device 7 estimates the position of the vehicle 10 by using, for example, an initial vector direction of the vehicle 10 estimated from the captured image Im1 that is first acquired after the position estimation device 7 is started, instead of the direction of the movement vector V of the vehicle 10 that is moving. In this embodiment, even when the vehicle 10 is stationary, the position of the vehicle 10 can be estimated by using the captured image Im1.

[0115] C-11. Other embodiment 11: In another embodiment, the position estimation device 7 may process a three-dimensional image captured by a stereo camera or a TOF (Time Of Flight) camera as the imaging device 9. In this embodiment, the processing target can be expanded, and therefore the versatility of the position estimation device 7 can be improved.

[0116] C-12. Other embodiment 12: In another embodiment, the remote control device 5 and the position estimation device 7 may be configured as one unit. In still another embodiment, each unit of the position estimation device 7 may be realized by, for example, cloud computing configured by one or more computers. In such a form, the configuration of the position estimation device 7 can be appropriately changed.

[0117] C-13. Other embodiment 13: In another embodiment, the position estimation device 7 may estimate the position of the vehicle 10 traveling in a remote manual driving mode. The remote manual driving mode is a driving mode in which the vehicle 10 travels without a driver getting in the vehicle 10 by receiving driving conditions generated by an operator operating an operator input device. In this manner, the position of the vehicle 10 traveling in a remote manual driving mode can be estimated not only in the remote automatic driving mode but also in the remote automatic driving mode. This can increase the versatility of the position estimation device 7.

[0118] C-14. Other embodiment 14: In another embodiment, the device CPU 72 may include two or more functional units that realize the function of the device acquisition unit 721. For example, the device CPU 72 may include an image acquisition unit that acquires the captured image Im1, and a parameter acquisition unit that acquires vehicle parameters and imaging parameters. Even in such a form, the device CPU 72 can acquire various information as necessary.

[0119] C-15. Other embodiment 15: In the above embodiment, the vehicle parameter was the height h of the positioning point 10e of the vehicle 10 from the road surface 60. That is, in the above embodiment, the vehicle parameter was the distance from the road surface 60 along the height direction of the vehicle 10. However, the present disclosure is not limited to this. The vehicle parameter may be, for example, the distance between a road boundary as a reference point on the roadway of the vehicle 10 and the positioning point 10e of the vehicle 10. That is, the vehicle parameter may be the distance from a reference point along the width direction of the vehicle 10. In this form, when the dimension in the width direction of the vehicle 10 (hereinafter, vehicle width) differs depending on the type of the vehicle 10, the position of the vehicle 10 can be estimated using a value related to the vehicle width as the vehicle parameter.

[0120] C-16. Other embodiment 16: In each of the above embodiments, the vehicle 10 may have a configuration capable of moving by remote control, and may be in the form of a platform having the configuration described below, for example. Specifically, the vehicle 10 may have at least the vehicle control device 2, the drive device 11, the steering device 12, the braking device 13, and the vehicle communication unit 21 in order to perform the three functions of "running", "turning", and "stopping" by remote control. That is, the vehicle 10 that can be moved by remote control may not have at least some of the interior parts such as the driver's seat and the dashboard, may not have at least some of the exterior parts such as the bumper and the fenders, and may not have a body shell. In this case, the remaining parts such as the body shell may be attached to the vehicle 10 before the vehicle 10 is shipped from the factory, or the remaining parts such as the body shell may be attached to the vehicle 10 after the vehicle 10 is shipped from the factory in a state in which the remaining parts such as the body shell are not attached to the vehicle 10. Note that the position estimation may be performed for the platform form in the same manner as for the vehicle 10 in each embodiment.

[0121] The present disclosure is not limited to the above-mentioned embodiment, and can be realized in various configurations without departing from the spirit of the present disclosure. For example, the technical features of the embodiments corresponding to the technical features in each aspect described in the Summary of the Invention can be appropriately replaced or combined to solve some or all of the above-mentioned problems or to achieve some or all of the above-mentioned effects. Furthermore, if a technical feature is not described as essential in this specification, it can be appropriately deleted. [Explanation of symbols]

[0122] 1...position estimation system, 2...vehicle control device, 5...remote control device, 7...position estimation device, 9...imaging device, 10...vehicle, 10e...positioning point, 11...driving device, 12...steering device, 13...braking device, 16...external sensor group, 17...internal sensor group, 21...vehicle communication unit, 22...vehicle CPU, 23...vehicle memory unit, 51...remote communication unit, 52...remote CPU, 53...remote memory unit, 60...road surface, 61...grid line, 71...device communication unit, 72...device CPU, 72a...device CPU, 73...device memory unit, 160...external sensor, 161...vehicle-mounted camera, 162... radar, 163... lidar, 170... internal sensor, 171... wheel speed sensor, 172... steering angle sensor, 221... vehicle speed calculation unit, 222... vehicle acquisition unit, 223... vehicle transmission unit, 224... operation control unit, 521... remote acquisition unit, 522... control value creation unit, 523... remote transmission unit, 721... device acquisition unit, 722... position calculation unit, 722a... distortion correction unit, 722b... rotation processing unit, 722c... crop processing unit, 722d... detection unit, 722e... perspective conversion unit, 722f... coordinate point calculation unit, 723... vehicle type identification unit, 724... position conversion unit, 72 5...device transmission unit, 726...correction value setting unit, 901...first imaging device, 902...second imaging device, A1...unmoved area, A2...moved area, C...center of gravity, D...first actual distance, ΔD...observation error, Dc...third actual distance, Do...observed distance, Dp...estimated distance, Dt...second actual distance, D1...vehicle database, D2...camera database, h...height of vehicle positioning point from road surface, Δh...correction parameter, hs...reference parameter, H...height of imaging device from road surface, Im1...captured image, Im2...corrected image, Im3...rotated image, Im4...after processing Image, Im5...first mask image, Im6...second mask image, Md1...position detection model, Md2...vehicle type identification model, Ms...mask area, N...network, P0...base coordinate point, P1...first coordinate point, P2...second coordinate point, P3...image coordinate point, Pa1...correction parameter, Pa2...perspective transformation parameter, Pc...imaging coordinate point, Pf...fixed coordinate point, Pv...vehicle coordinate point, R0...base circumscribing rectangle, R1...first circumscribing rectangle, R2...second circumscribing rectangle, RG...imaging area, RG1...first imaging area, RG2...second imaging area, V...movement vector

Claims

1. A position estimation device that estimates a position of a vehicle, an acquisition unit that acquires an image including the vehicle from an imaging device; a position calculation unit that calculates an image coordinate point indicating the position of the vehicle in an image coordinate system using the captured image; A vehicle type identification unit that identifies a type of the vehicle included in the captured image; a position conversion unit that converts the image coordinate points into vehicle coordinate points that indicate the position of the vehicle in the global coordinate system using imaging parameters calculated based on the position of the imaging device in a global coordinate system and vehicle parameters determined according to the type of the vehicle.

2. The position estimation device according to claim 1 , further comprising: A position estimation device comprising: a direction information generation unit that generates direction information indicating a moving direction of the vehicle using the vehicle coordinate points.

3. The position estimation device according to claim 1 , A position estimation device in which, when converting the image coordinate points into the vehicle coordinate points, the position conversion unit uses the vehicle coordinate points as an objective variable and substitutes the values ​​of the vehicle parameters acquired by the acquisition unit into a relational equation that includes the image coordinate points, the imaging parameters, and the vehicle parameters as explanatory variables.

4. The position estimation device according to claim 1 , further comprising: a correction value setting unit that sets a correction parameter indicating a correction value for the reference parameter by calculating a difference between the vehicle parameter acquired by the acquisition unit and a reference parameter indicating a reference value of the vehicle parameter, the correction parameter being determined according to the type of the vehicle identified by the vehicle type identification unit; a position conversion unit that, when converting the image coordinate points into the vehicle coordinate points, converts the image coordinate points into the vehicle coordinate points as objective variables and substitutes the correction parameters set by the correction value setting unit into a relational equation that includes the image coordinate points, the imaging parameters, the reference parameters, and the correction parameters as explanatory variables.

5. The position estimation device according to claim 4, further comprising: The correction value setting unit sets the correction parameter to zero without calculating the difference when the type of the vehicle identified by the vehicle type identification unit is a predetermined type.

6. The position estimation device according to claim 1 , the imaging parameter is a height of the imaging device from a road surface calculated based on the position of the imaging device in the global coordinate system; A position estimation device, wherein the vehicle parameter is a height of a predetermined positioning point of the vehicle from the road surface.

7. The position estimation device according to claim 1 , The position calculation unit is a detection unit that generates a first mask image to which a mask region that masks the vehicle in the captured image is added; a perspective transformation unit that performs perspective transformation on the first mask image; a coordinate point calculation unit that sets a specified vertex of a first circumscribing rectangle set in the mask area in the first mask image as a first coordinate point, sets a vertex indicating the same position as the first coordinate point among the vertices of a second circumscribing rectangle set in the mask area in a second mask image obtained by perspective transforming the first mask image as a second coordinate point, and corrects the first coordinate point using the second coordinate point.

8. 8. The position estimation device according to claim 7, The position estimation device, wherein the position calculation unit further includes a rotation processing unit that rotates the captured image so that the moving direction of the vehicle faces a predetermined direction.

9. 8. The position estimation device according to claim 7, The position estimation device, wherein the position calculation unit further includes a distortion correction unit that corrects distortion of the captured image.

10. A position estimation system for estimating a position of a vehicle, comprising: Vehicles and an imaging device for acquiring an image including the vehicle whose position is to be estimated; A position estimation system comprising: a position estimation device according to any one of claims 1 to 9.

11. A method for estimating a position of a vehicle, comprising: an acquisition step of acquiring an image including the vehicle from an imaging device; a position calculation step of calculating an image coordinate point indicating the position of the vehicle in an image coordinate system using the captured image; a vehicle type identification step of identifying the type of the vehicle included in the captured image; a position conversion process for converting the image coordinate points into vehicle coordinate points indicating the position of the vehicle in the global coordinate system using imaging parameters calculated based on the position of the imaging device in a global coordinate system and vehicle parameters determined according to the type of the vehicle.

12. 1. A computer program for use in estimating a vehicle's position, comprising: an acquisition function for acquiring an image including the vehicle from an imaging device; a position calculation function that calculates an image coordinate point indicating the position of the vehicle in an image coordinate system using the captured image; A vehicle type identification function that identifies the type of the vehicle included in the captured image; a position conversion function that converts the image coordinate points into vehicle coordinate points that indicate the position of the vehicle in the global coordinate system using imaging parameters calculated based on the position of the imaging device in a global coordinate system and vehicle parameters determined according to the type of the vehicle.

Citation Information

Patent Citations

  • Vehicle traveling locus observation system, vehicle traveling locus observation method and program

    JP2010020729A

  • Outside recognition device for vehicle

    JP2019066333A

  • Method for operating a vehicle and method for operating a manufacturing system

    JP2017538619A