Position estimation device, position estimation system, position estimation method, and computer program

Abstract

To provide a position estimation device, a position estimation system, a position estimation method and a program which accurately estimate a position of a vehicle, by using a captured image including the vehicle.SOLUTION: In a position estimation system, one or more vehicles having vehicle controllers, a plurality of imaging devices, and a position estimation device which estimates a position of the vehicle are connected to a network and communicate with each other. A device CPU 72 with which a position estimation device 7 which estimates the position of the vehicle is provided functions as: a device acquisition unit 721 which acquires a captured image including the vehicle from the imaging device; a position calculation unit 722 which calculates an image coordinate point indicating the position of the vehicle in an image coordinate system, by using the captured image; a vehicle type identification unit 723 which identifies a type of the vehicle included in the captured image; and a position conversion unit 724 which converts the image coordinate point into a vehicle coordinate point indicating the position of the vehicle in a global coordinate system, by using an imaging parameter calculated based on the position of the imaging device in the global coordinate system, and a vehicle parameter determined according to the type of the vehicle.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present disclosure relates to a position estimation device, a position estimation system, a position estimation method, and a computer program.

Background Art

[0002] Conventionally, vehicles that automatically travel by remote control are known (Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When a vehicle is automatically driven by remote control, it is necessary to estimate the position of the vehicle. When estimating the position of the vehicle using a captured image obtained by capturing an imaging area including the vehicle from outside the vehicle, the inventors of the present application have found that the estimation accuracy of the position of the vehicle may decrease because the vehicle body and the like differ depending on the type of the vehicle.

Means for Solving the Problems

[0005] The present disclosure can be realized in the following forms.

[0006] (1) According to a first embodiment of the present disclosure, a position estimation device is provided. The position estimation device for estimating the position of a vehicle includes: an acquisition unit that acquires an image of the vehicle from an imaging device; a position calculation unit that uses the image to calculate an image coordinate point indicating the position of the vehicle in an image coordinate system; a vehicle type identification unit that identifies the type of vehicle included in the image; and a position conversion unit that uses imaging parameters calculated based on the position of the imaging device in a global coordinate system and vehicle parameters determined according to the type of vehicle to convert the image coordinate point into a vehicle coordinate point indicating the position of the vehicle in the global coordinate system. According to this embodiment, the type of vehicle included in the image can be identified and vehicle parameters corresponding to the type of vehicle can be acquired. Then, the position of the vehicle can be estimated from the image using the vehicle parameters corresponding to the type of vehicle. In this way, the position of the vehicle can be estimated taking into account the type of vehicle. This makes it possible to suppress a decrease in the accuracy of estimating the position of the vehicle due to differences in vehicle class, etc., depending on the type of vehicle. (2) The above configuration may further include a direction information generation unit that generates direction information indicating the direction of movement of the vehicle using the vehicle coordinate points. In this configuration, direction information can be generated using the vehicle's position information. In this way, the actual trajectory and direction of movement of the vehicle while it is in motion can be estimated. (3) In the above configuration, when the position conversion unit converts the image coordinate points to the vehicle coordinate points, it may substitute the values ​​of the vehicle parameters acquired by the acquisition unit into a relational expression in which the vehicle coordinate points are the target variable and the image coordinate points, the imaging parameters, and the vehicle parameters are the explanatory variables. According to this configuration, the image coordinate points can be converted to vehicle coordinate points by substituting the values ​​of the vehicle parameters acquired by the acquisition unit into a relational expression in which the vehicle coordinate points are the target variable and the image coordinate points, the imaging parameters, and the vehicle parameters are the explanatory variables. (4) The above configuration further includes a correction value setting unit which calculates the difference between the vehicle parameters acquired by the acquisition unit and a reference parameter indicating a reference value of the vehicle parameters, and sets a correction parameter indicating a correction value for the reference parameter, which is determined according to the type of vehicle identified by the vehicle type identification unit, and the position conversion unit, when converting the image coordinate points to the vehicle coordinate points, may substitute the correction parameter set by the correction value setting unit into a relational expression in which the vehicle coordinate points are the target variable and the image coordinate points, the imaging parameters, the reference parameter, and the correction parameter are the explanatory variables. According to this configuration, the type of vehicle included in the captured image can be identified and a correction parameter corresponding to the type of vehicle can be calculated. Then, the image coordinate points can be converted to vehicle coordinate points by substituting the correction parameter set by the correction value setting unit into a relational expression in which the vehicle coordinate points are the target variable and the image coordinate points, the imaging parameters, the reference parameter, and the correction parameter are the explanatory variables. (5) In the above configuration, the correction value setting unit may further set the correction parameter to zero without calculating the difference if the type of vehicle identified by the vehicle type identification unit is a predetermined type. In this configuration, if the type of vehicle identified by the vehicle type identification unit is a predetermined type, the correction parameter can be set to zero without calculating the difference between the reference parameter and the vehicle parameter. (6) In the above configuration, the imaging parameter may be the height of the imaging device from the road surface calculated based on the position of the imaging device in the global coordinate system, and the vehicle parameter may be the height of the vehicle's predetermined positioning point from the road surface. In this configuration, the image coordinate point can be converted to the vehicle coordinate point using the ratio of the height of the imaging device from the road surface to the height of the vehicle's positioning point from the road surface. (7) In the above embodiment, the position calculation unit may include: a detection unit that generates a first mask image to which a mask region that masks the vehicle in the captured image is added; a perspective transformation unit that performs perspective transformation on the first mask image; and a coordinate point calculation unit that sets a specified vertex of a first bounding rectangle set in the mask region in the first mask image as a first coordinate point, sets a vertex of a second bounding rectangle set in the mask region in the second mask image obtained by perspective transformation of the first mask image that indicates the same position as the first coordinate point as a second coordinate point, and corrects the first coordinate point using the second coordinate point. In this embodiment, a first mask image to which a mask region that masks the vehicle in the captured image is added, and a second mask image obtained by perspective transformation of the first mask image can be generated. In this way, a first coordinate point can be extracted from the first mask image, and a second coordinate point can be extracted from the second mask image. Then, by correcting the first coordinate point using the second coordinate point, the image coordinate point can be calculated. This makes it possible to calculate the image coordinate point more accurately. (8) In the above configuration, the position calculation unit may further include a rotation processing unit that rotates the captured image so that the direction of movement of the vehicle is oriented in a predetermined direction. In this configuration, the captured image can be rotated so that the direction of the vector indicating the direction of movement of the vehicle is oriented in a predetermined direction. In this way, the vehicle included in the captured image can be detected with the direction of the vector indicating the direction of movement of the vehicle unified. This improves the accuracy of estimating the position of the vehicle. (9) In the above configuration, the position calculation unit may further include a distortion correction unit for correcting the distortion of the captured image. This configuration allows for the correction of distortion in the captured image, thereby further improving the accuracy of the vehicle position estimation. (10) A second embodiment of the present disclosure provides a position estimation system. The position estimation system for estimating the position of a vehicle comprises a vehicle, an imaging device for acquiring an image containing the vehicle to be the subject of position estimation, and the position estimation device described in the above embodiment. In this embodiment, the type of vehicle included in the image can be identified and vehicle parameters corresponding to the type of vehicle can be acquired. Then, the position of the vehicle can be estimated from the image using the vehicle parameters corresponding to the type of vehicle. In this way, the position of the vehicle can be estimated taking into account the type of vehicle. This makes it possible to suppress a decrease in the accuracy of estimating the position of the vehicle due to differences in vehicle class, etc., depending on the type of vehicle. (11) A third embodiment of the present disclosure provides a method for estimating the position of a vehicle. The method for estimating the position of a vehicle includes: an acquisition step of acquiring an image of the vehicle from an imaging device; a position calculation step of calculating an image coordinate point indicating the position of the vehicle in an image coordinate system using the image of the vehicle; a vehicle type identification step of identifying the type of vehicle included in the image of the vehicle; and a position transformation step of converting the image coordinate point to a vehicle coordinate point indicating the position of the vehicle in the global coordinate system using imaging parameters calculated based on the position of the imaging device in a global coordinate system and vehicle parameters determined according to the type of vehicle. According to this embodiment, the type of vehicle included in the image of the vehicle can be identified and vehicle parameters corresponding to the type of vehicle can be acquired. Then, the position of the vehicle can be estimated from the image of the vehicle using the vehicle parameters corresponding to the type of vehicle. In this way, the position of the vehicle can be estimated taking into account the type of vehicle. This makes it possible to suppress a decrease in the accuracy of estimating the position of the vehicle due to differences in vehicle class, etc., depending on the type of vehicle. (12) According to a fourth embodiment of the present disclosure, a computer program is provided. The computer program used to estimate the position of a vehicle includes: an acquisition function for acquiring an image of the vehicle from an imaging device; a position calculation function for calculating an image coordinate point indicating the position of the vehicle in an image coordinate system using the image of the vehicle; a vehicle type identification function for identifying the type of vehicle included in the image of the vehicle; and a position transformation function for converting the image coordinate point to a vehicle coordinate point indicating the position of the vehicle in the global coordinate system using imaging parameters calculated based on the position of the imaging device in a global coordinate system and vehicle parameters determined according to the type of vehicle. According to this embodiment, the type of vehicle included in the image of the vehicle can be identified and vehicle parameters corresponding to the type of vehicle can be acquired. Then, the position of the vehicle can be estimated from the image of the vehicle using the vehicle parameters corresponding to the type of vehicle. In this way, the position of the vehicle can be estimated taking into account the type of vehicle. This makes it possible to suppress a decrease in the accuracy of estimating the position of the vehicle due to differences in vehicle class, etc., depending on the type of vehicle. This disclosure can be implemented in various forms other than the position estimation device, position estimation system, position estimation method, and computer program described above. For example, it can be implemented in the form of a method for manufacturing a position estimation device, a method for controlling a position estimation device and a position estimation system, and so on. [Brief explanation of the drawing]

[0007] [Figure 1] A diagram showing the schematic configuration of the position estimation system. [Figure 2] A diagram showing the general configuration of the vehicle. [Figure 3] A diagram showing the schematic configuration of a remote control device. [Figure 4] A diagram showing the schematic configuration of a position estimation device. [Figure 5] A diagram showing the details of the device CPU in the first embodiment. [Figure 6] A flowchart illustrating the first operation control method. [Figure 7] A flowchart illustrating the second operation control method. [Figure 8] Flowchart showing the vehicle position estimation method in the first embodiment. [Figure 9] Schematic diagram showing examples of various images. [Figure 10] Diagram for explaining the details of the coordinate point calculation step. [Figure 11] Diagram for explaining the method of obtaining the base coordinate point. [Figure 12] Figure 1 for explaining the details of the position conversion step in the first embodiment. [Figure 13] Figure 2 for explaining the details of the position conversion step in the first embodiment. [Figure 14] Diagram showing the details of the device CPU in the second embodiment. [Figure 15] Flowchart showing the vehicle position estimation method in the second embodiment. [Figure 16] Figure 1 for explaining the details of the position conversion step in the second embodiment. [Figure 17] Figure 2 for explaining the details of the position conversion step in the second embodiment.

Mode for Carrying Out the Invention

[0008] A. First Embodiment: A-1. Configuration of the Position Estimation System: FIG. 1 is a diagram showing the schematic configuration of a position estimation system 1. The position estimation system 1 is a system for estimating the position of a vehicle 10. The position estimation system 1 includes one or more vehicles 10, an imaging device 9, and a position estimation device 7 for estimating the position of the vehicle 10.

[0009] The imaging device 9 acquires a captured image by imaging an imaging region RG including the vehicle 10 that is the target of position estimation from outside the vehicle 10. The imaging device 9 transmits the captured image to the position estimation device 7 together with camera identification information (camera ID) for identifying the plurality of imaging devices 9 and the acquisition timing of the captured image. The captured image transmitted to the position estimation device 7 is used to estimate the position of the vehicle 10. The captured image is a two-dimensional image formed by a collection of pixels arranged on the XcYc plane of the camera coordinate system. The camera coordinate system is a coordinate system having coordinate axes indicated by an Xc axis with the focal point of the imaging device 9 as the origin and a Yc axis orthogonal to the Xc axis. The captured image includes at least two-dimensional data of the vehicle 10 that is the target of position estimation. The captured image is preferably a color image, but may be a gray-scale image. The imaging device 9 is an infra-camera having an imaging element such as a CCD image sensor or a CMOS image sensor and an optical system, for example.

[0010] In the present embodiment, the imaging device 9 acquires a captured image that looks down on the road surface 60 and the vehicle 10 traveling on the road surface 60 from above. The installation position and the number of installations of the imaging device 9 are determined in consideration of the imaging region RG (viewing angle) and the like of the imaging device 9 in order to image the entire road surface 60 with one or more imaging devices 9. Specifically, each imaging device 9 is installed such that the first imaging region RG1, which is the imaging region RG of the adjacent first imaging device 901, and the second imaging region RG2, which is the imaging region RG of the second imaging device 902, overlap. Further, each imaging device 9 is installed at a position where it can image a preset measurement point 10e on a specific part of the vehicle 10 traveling on the road surface 60. In the present embodiment, the measurement point 10e is the rear end on the left side surface of the vehicle 10 (hereinafter, the left rear end). Note that the measurement point 10e may be other than the left rear end of the vehicle 10. Further, the imaging device 9 may acquire information not only from above the vehicle 10 but also from the front, rear, side, etc. of the vehicle 10. Note that the imaging device 9 may transmit the captured image to the remote control device 5.

[0011] Vehicle 10 has a manned driving mode and a remote automatic driving mode. In manned driving mode, a driver riding in vehicle 10 operates driver input devices (not shown) such as a steering wheel and accelerator installed in vehicle 10 to determine the driving conditions of vehicle 10 and drive the vehicle. The driving conditions here refer to the conditions that define the driving operation of vehicle 10. Driving conditions include, for example, the driving route, position, driving speed, acceleration, and steering angle of the wheels of vehicle 10. In remote automatic driving mode, vehicle 10 automatically drives without a driver riding in vehicle 10 by receiving the driving conditions of vehicle 10 from outside vehicle 10. When driving in remote automatic driving mode, vehicle 10 automatically drives by controlling the operation of the drive device etc. installed in vehicle 10, which will be described later, according to control values ​​received from a remote control device 5 such as a server located in a different location from vehicle 10. In this embodiment, vehicle 10 drives in remote automatic driving mode within a factory that produces vehicle 10 by executing multiple production processes. Furthermore, the factory is not limited to being located in a single building or on a single site or address; it may be located across multiple buildings, multiple sites, multiple addresses, etc. In this case, the vehicle 10 may travel on public roads as well as private roads. Also, the position estimation system 1 may estimate the position of the vehicle 10 traveling outside the factory.

[0012] Vehicle 10 is, for example, an electric vehicle, a hybrid vehicle, a fuel cell vehicle, a gasoline vehicle, or a diesel vehicle. Vehicle 10 may be a private vehicle such as a passenger car, or a commercial vehicle such as a truck, bus, or construction vehicle. Vehicle 10 includes both a finished vehicle 10 as a product and a semi-finished or work-in-progress vehicle 10.

[0013] Figure 2 is a diagram illustrating the schematic configuration of vehicle 10. Figure 2 shows a representative part of the configuration of vehicle 10. Vehicle 10 comprises a drive unit 11, a steering unit 12, a braking unit 13, an external sensor group 16 having an external sensor 160, an internal sensor group 17 having an internal sensor 170, and a vehicle control unit 2.

[0014] The drive unit 11 accelerates the vehicle 10. For example, if the vehicle 10 is an electric vehicle, the drive unit 11 includes a battery (not shown), a motor (not shown) powered by the battery, and wheels (not shown) rotated by the motor.

[0015] The steering device 12 changes the direction of travel of the vehicle 10. The steering device 12 rotates the steering shaft (not shown) by torque from a steering motor (not shown) according to control values ​​received from the remote control device 5, so that the steering angle of the steering wheel (not shown) matches the target steering angle of the wheels (not shown). In this way, the steering device 12 steers the wheels without steering input in remote automatic driving mode.

[0016] The braking device 13 slows down the vehicle 10. The braking device 13 is, for example, a disc brake device.

[0017] The external sensor group 16 comprises multiple types of external sensors 160. The external sensors 160 are sensors that acquire ambient information indicating the state of the area surrounding the vehicle 10. The external sensors 160 transmit the acquired data to the vehicle control device 2. In this embodiment, the external sensor group 16 comprises an on-board camera 161, a radar 162, and a lidar 163 as external sensors 160.

[0018] The in-vehicle camera 161 acquires internal imaging information as imaging data by capturing an area that includes at least a portion of the surrounding area of ​​the vehicle 10. The internal imaging information may be still image data or video data. Furthermore, the internal imaging information may be color data or monochrome data.

[0019] The radar 162 emits a search wave (radio wave) within a predetermined search range and receives the reflected wave reflected by objects in the area surrounding the vehicle 10, thereby detecting the distance, angle, and relative velocity of objects in the area surrounding the vehicle 10.

[0020] The lidar 163 detects the distance, angle, and shape of objects in the area surrounding the vehicle 10 by irradiating a laser beam into a predetermined measurement range and detecting the reflected light from objects in the area surrounding the vehicle 10. The configuration of the external sensor group 16 is not limited to this.

[0021] The internal sensor group 17 comprises multiple types of internal sensors 170. The internal sensors 170 are sensors that acquire various physical quantities necessary to control the driving operation of the vehicle 10. The internal sensors 170 transmit the acquired data to the vehicle control device 2. In this embodiment, the internal sensor group 17 comprises a wheel speed sensor 171 and a steering angle sensor 172 as internal sensors 170. The wheel speed sensor 171 measures the rotational speed of each wheel (hereinafter referred to as wheel speed). The steering angle sensor 172 measures the actual steering angle of each wheel. Note that the configuration of the internal sensor group 17 is not limited to this.

[0022] The vehicle control device 2 comprises a vehicle communication unit 21 as the communication unit of the vehicle control device 2, a vehicle storage unit 23 as the storage unit of the vehicle control device 2, and a vehicle CPU 22 as the central processing unit of the vehicle control device 2. The vehicle communication unit 21, the vehicle storage unit 23, and the vehicle CPU 22 are connected to each other via an internal bus and interface circuits.

[0023] The vehicle communication unit 21 connects internal and external devices to the vehicle control device 2 in a communicative manner. Internal devices are devices mounted on the vehicle 10, such as internal sensors 170 and external sensors 160, that are capable of communicating with the vehicle control device 2. External devices are devices located in a different location from the vehicle 10, such as remote control devices 5 and imaging devices 9. The vehicle communication unit 21 is, for example, a wireless communication device. The vehicle communication unit 21 communicates with internal devices, for example, by CAN (Controller Area Network) communication. CAN communication is a communication standard that can transmit or receive in multiple directions. The vehicle communication unit 21 also communicates with external devices such as remote control devices 5 and imaging devices 9 that are connected to the network N, for example, via an access point (not shown) in a factory. The vehicle communication unit 21 may also communicate with the vehicle control device 2 of another vehicle 10. The communication method by the vehicle communication unit 21 is not limited to this.

[0024] The vehicle storage unit 23 stores various information, including various programs that control the driving operation of the vehicle 10. The vehicle storage unit 23 includes, for example, RAM, ROM, and a hard disk drive (HDD).

[0025] The vehicle CPU 22 functions as a vehicle speed calculation unit 221, a vehicle acquisition unit 222, a vehicle transmission unit 223, and an operation control unit 224 by deploying various programs stored in the vehicle memory unit 23.

[0026] The vehicle speed calculation unit 221 calculates the vehicle speed (hereinafter referred to as vehicle speed) of the vehicle 10 using the output value of the wheel speed sensor 171. The vehicle speed calculation unit 221 calculates the vehicle speed based on the wheel speed per unit time after performing calculation processing such as averaging the wheel speed of each wheel. However, the method of calculating the vehicle speed is not limited to this. In addition, at least some of the functions of the vehicle speed calculation unit 221 may be performed by the wheel speed sensor 171 or the remote control device 5.

[0027] The vehicle information acquisition unit 222 acquires information (hereinafter referred to as "vehicle sensor information") that includes at least a portion of the information acquired by the internal sensor 170 and the external sensor 160, and the vehicle speed calculated by the vehicle speed calculation unit 221.

[0028] The vehicle transmission unit 223 transmits various information to external devices. For example, the vehicle transmission unit 223 transmits vehicle sensor information to the remote control device 5 along with vehicle identification information that identifies multiple vehicles 10.

[0029] The motion control unit 224 receives control values ​​transmitted from the remote control device 5 and drives the drive unit 11, steering unit 12, and braking unit 13 according to the received control values. In this way, the motion control unit 224 controls the driving operation of the vehicle 10 in accordance with instructions from the remote control device 5. At least some of the functions of the vehicle CPU 22 may be implemented as a function of the remote control device 5 or the imaging device 9.

[0030] Figure 3 shows a schematic configuration of the remote control device 5. The remote control device 5 comprises a remote communication unit 51 as the communication unit of the remote control device 5, a remote storage unit 53 as the storage unit of the remote control device 5, and a remote CPU 52 as the central processing unit of the remote control device 5. The remote communication unit 51, the remote storage unit 53, and the remote CPU 52 are connected to each other via an internal bus and interface circuits.

[0031] The remote communication unit 51 connects the vehicle control device 2, the position estimation device 7, and the imaging device 9 to the remote control device 5 in a communication-enabled manner. The remote communication unit 51 is, for example, a wireless communication device. However, the communication method by the remote communication unit 51 is not limited to this.

[0032] The remote storage unit 53 stores various information, including various programs that control the operation of the remote control device 5. The remote storage unit 53 includes, for example, RAM, ROM, and a hard disk drive (HDD).

[0033] The remote CPU 52 functions as a remote acquisition unit 521, a control value creation unit 522, and a remote transmission unit 523 by deploying various programs stored in the remote storage unit 53.

[0034] The remote acquisition unit 521 acquires information regarding the driving conditions of the vehicle 10 (hereinafter referred to as driving information). The driving information includes, for example, captured images transmitted from the imaging device 9, vehicle coordinate points as information indicating the position of the vehicle 10 transmitted from the position estimation device 7, vehicle sensor information transmitted from the vehicle control device 2, and driving route information pre-stored in the remote storage unit 53. The driving route information is information indicating the target driving route of the vehicle 10 when it is driving in remote automatic driving mode. However, the types of information included in the driving information are not limited to these.

[0035] The control value creation unit 522 uses the driving information acquired by the remote acquisition unit 521 to create control values ​​that define the driving operation of the vehicle 10. Specifically, the control value creation unit 522 creates control values ​​(hereinafter referred to as reference control values) for driving the vehicle 10 along a target driving route. In this embodiment, the reference control values ​​include a control value that defines the acceleration of the vehicle 10 in the forward direction (hereinafter referred to as acceleration control value) and a control value that defines the steering angle of the vehicle 10 (hereinafter referred to as steering angle control value). The reference control values ​​may also include a control value (hereinafter referred to as direction control value) for switching the direction of travel of the vehicle 10 to either the forward direction or the reverse direction opposite to the forward direction. In other embodiments, the reference control values ​​may include only the target driving route. In this case, the operation control unit 224 of the vehicle control device 2 determines the acceleration, steering angle, direction of travel, etc. of the vehicle 10 based on the information regarding the target driving route included in the reference control values, and drives the drive unit 11, steering unit 12, and braking unit 13.

[0036] Furthermore, the control value creation unit 522 creates control values ​​(hereinafter referred to as corrected control values) that correct the relative position of the vehicle 10 to the target driving route. Specifically, the control value creation unit 522 calculates the actual driving trajectory of the vehicle 10 by arranging multiple captured images taken by different imaging areas RG in chronological order. Then, the control value creation unit 522 compares the target driving route with the actual driving trajectory of the vehicle 10 and calculates the difference in the driving trajectory relative to the target driving route. Furthermore, the control value creation unit 522 creates corrected control values ​​to realize the target driving route while analyzing the captured images in order to reduce the difference between the target driving route and the actual driving trajectory of the vehicle 10. In this embodiment, the corrected control values ​​include at least steering angle control values. The corrected control values ​​may also include direction control values. In other embodiments, the corrected control values ​​may include only the corrected driving route. The corrected driving route is information indicating the driving route that the vehicle 10 should travel in order to correct the relative position of the vehicle 10 to the target driving route. In this case, the operation control unit 224 of the vehicle control device 2 determines the acceleration, steering angle, and direction of travel of the vehicle 10 based on the information regarding the corrected driving route included in the corrected control value, and drives the drive unit 11, steering unit 12, and braking unit 13.

[0037] The remote transmission unit 523 transmits the control values ​​created by the control value creation unit 522 to the vehicle 10 whose driving operation is to be controlled. Alternatively, the remote transmission unit 523 may transmit the control values ​​to the vehicle 10 via the position estimation device 7 or the imaging device 9. When the remote transmission unit 523 transmits the control values ​​to the vehicle 10 via the imaging device 9, the remote transmission unit 523 transmits the control values ​​to the imaging device 9. The imaging device 9 then transmits the control values ​​received from the remote control device 5 to the vehicle 10. In this way, the vehicle 10 can receive the control values ​​from the imaging device 9, which is closer to the vehicle 10. This makes it less susceptible to communication failures. Therefore, the possibility of the vehicle 10 stopping due to a communication failure while driving in remote automatic driving mode can be reduced. At least some of the functions of the remote CPU 52 may be implemented as a function of the vehicle control device 2 or the imaging device 9.

[0038] Figure 4 shows a schematic configuration of the position estimation device 7. The position estimation device 7 adopts the position of a predetermined positioning point 10e (Figure 1) of the vehicle 10 as the position of the vehicle 10. The position estimation device 7 includes a device communication unit 71 as the communication unit of the position estimation device 7, a device storage unit 73 as the memory unit of the position estimation device 7, and a device CPU 72 as the central processing unit of the position estimation device 7. The device communication unit 71, the device storage unit 73, and the device CPU 72 are connected to each other via an internal bus or interface circuit. The position estimation device 7 may also include a display device (not shown) that displays various information to the user. The display device may be, for example, a liquid crystal display or an organic EL display.

[0039] The device communication unit 71 connects the vehicle control device 2, the remote control device 5, the imaging device 9, and the position estimation device 7 in a communicative manner. The device communication unit 71 is, for example, a wireless communication device. However, the communication method by the device communication unit 71 is not limited to this.

[0040] The device storage unit 73 stores various information, including various programs that control the operation of the position estimation device 7, a position detection model Md1, a vehicle type identification model Md2, a distortion correction parameter Pa1, and a perspective transformation parameter Pa2. Furthermore, the device storage unit 73 stores a vehicle database D1 and a camera database D2. The device storage unit 73 includes, for example, RAM, ROM, and a hard disk drive (HDD).

[0041] The position detection model Md1 is a pre-trained machine learning model used to identify the position of vehicle 10 included in captured images. In this embodiment, the position detection model Md1 is a machine learning model that has been pre-trained to mask vehicle 10 in input images by taking captured images and various images that have undergone various processing as input. As the algorithm for the position detection model Md1, for example, a deep neural network (DNN) having the structure of a convolutional neural network (CNN) that realizes semantic segmentation or instance segmentation is used. An example of a DNN used as the algorithm for the position detection model Md1 is a DNN that performs instance segmentation such as YOLACT++. However, the configuration of the position detection model Md1 is not limited to this. The position detection model Md1 may be a pre-trained machine learning model that uses algorithms other than neural networks, for example.

[0042] The vehicle type identification model Md2 is a pre-trained machine learning model used to identify the type of vehicle 10 contained in an captured image. Here, the type of vehicle 10 refers to the type of vehicle 10 classified by, for example, the vehicle name or model number. The vehicle type identification model Md2 is a machine learning model that has been pre-trained to output information indicating the type of vehicle 10 in an input image, by taking an captured image or various images that have undergone various processing as input. In order to identify the type of vehicle 10, the vehicle type identification model Md2 has learned features corresponding to the type of vehicle 10. Features corresponding to the type of vehicle 10 include, for example, the shape of the vehicle 10 and the vehicle class determined by the total length, width, and height of the vehicle 10. In this embodiment, the vehicle type identification model Md2 outputs vehicle type identification information as information indicating the type of vehicle 10 in the input image. The vehicle type identification information is a unique ID (identifier) ​​that is assigned to each type so as not to overlap between types in order to identify multiple types of vehicle 10. For example, a CNN is used as the algorithm for the vehicle type identification model Md2. Furthermore, the configuration of the vehicle identification model Md2 is not limited to this. The vehicle identification model Md2 may also be a pre-trained machine learning model that uses algorithms other than neural networks, for example.

[0043] The distortion correction parameter Pa1 is used to correct distortion in the captured image. The perspective transformation parameter Pa2 is used to perform perspective transformation on the first mask image. Details of the correction parameter Pa1 and the perspective transformation parameter Pa2 will be described later.

[0044] The vehicle database D1 is information that shows vehicle parameters for each type of vehicle 10. Vehicle parameters are parameters that are determined according to the type of vehicle 10. For example, vehicle parameters are parameters related to the distance from a reference point on the vehicle 10's route (e.g., road surface 60 or road boundary) to the vehicle 10's positioning point 10e. In this embodiment, the vehicle database D1 is a database that links vehicle type identification information with vehicle parameters for the type of vehicle 10 identified by the vehicle type identification information.

[0045] Camera database D2 is information that shows imaging parameters calculated based on the position of the imaging device 9 in the global coordinate system, for each imaging device 9. In this embodiment, camera database D2 is a database that links camera identification information with imaging parameters for the imaging device 9 identified by the camera identification information. The imaging parameters are calculated based on the position of the imaging device 9 in the global coordinate system. At least a portion of the various information stored in the device storage unit 73 may also be stored in the storage units of other devices (for example, the vehicle storage unit 23, the remote storage unit 53, or the storage unit of the imaging device 9).

[0046] Figure 5 shows the details of the device CPU 72 in the first embodiment. The device CPU 72 functions as a device acquisition unit 721 and a position calculation unit 722 by deploying various programs stored in the device storage unit 73. Furthermore, the device CPU 72 functions as a vehicle type identification unit 723, a position conversion unit 724, and a device transmission unit 725 by deploying various programs stored in the device storage unit 73.

[0047] The device acquisition unit 721 acquires various types of information. For example, the device acquisition unit 721 acquires an image Im1 including the vehicle 10 from the imaging device 9. The device acquisition unit 721 acquires vehicle parameters corresponding to the type of vehicle 10 identified by the vehicle type identification unit 723 by referring to the vehicle database D1 pre-stored in the device storage unit 73. The device acquisition unit 721 acquires imaging parameters for the imaging device 9, which is the source of the image to be analyzed, by referring to the camera database D2 pre-stored in the device storage unit 73. Note that the types of information acquired by the device acquisition unit 721 are not limited to these.

[0048] The position calculation unit 722 calculates image coordinate points using the captured image. The image coordinate points are coordinate points that indicate the position of the vehicle 10 in the image coordinate system described later. The position calculation unit 722 includes a distortion correction unit 722a, a rotation processing unit 722b, a cropping processing unit 722c, a detection unit 722d, a perspective transformation unit 722e, and a coordinate point calculation unit 722f.

[0049] The distortion correction unit 722a generates a corrected image by correcting the distortion of the captured image. The rotation processing unit 722b generates a rotated image by rotating the corrected image so that the direction of the vector indicating the direction of movement of the vehicle 10 (hereinafter referred to as the movement vector) points in a predetermined direction. The cropping processing unit 722c generates a processed image in which the necessary region, including the vehicle region and the surrounding region, is cropped from the rotated image by deleting from each region of the rotated image regions areas other than the vehicle region corresponding to the vehicle 10 and the predetermined surrounding region of the vehicle 10 (hereinafter referred to as the necessary region). In this embodiment, when the vehicle 10 moves a distance exceeding a predetermined threshold, the cropping processing unit 722c deletes the moved region, which is the unnecessary region, from the rotated image, corresponding to the distance the vehicle 10 has moved. As a result, the cropping processing unit 722c generates a processed image in which the unmoved region, including the vehicle 10, is cropped from the rotated image. The detection unit 722d uses the position detection model Md1 to detect the vehicle 10 included in the processed image and generates a first mask image with a mask region added that masks the vehicle 10 in the processed image. The perspective transformation unit 722e generates a second mask image by performing a perspective transformation on the first mask image. The coordinate point calculation unit 722f calculates image coordinate points by correcting the first coordinate points using the second coordinate points. The first coordinate points are the coordinate points in the image coordinate system of the specified vertices of the first bounding rectangle set in the mask region of the first mask image. The second coordinate points are the coordinate points in the image coordinate system of the vertices of the second bounding rectangle set in the mask region of the second mask image that indicate the same position as the first coordinate points. Note that the configuration of the position calculation unit 722 is not limited to this.

[0050] The vehicle type identification unit 723 identifies the type of vehicle 10 included in the captured image. In this embodiment, the vehicle type identification unit 723 identifies the type of vehicle 10 using the vehicle type identification model Md2.

[0051] The position conversion unit 724 converts image coordinate points to vehicle coordinate points using the imaging parameters and vehicle parameters acquired by the device acquisition unit 721. The vehicle coordinate points are coordinate points that indicate the position of the vehicle 10 in the global coordinate system. In other words, the position conversion unit 724 calculates the vehicle coordinate points using the imaging parameters for the imaging device 9 that acquired the captured image acquired by the device acquisition unit 721, and the vehicle parameters for the type of vehicle 10 identified by the vehicle type identification unit 723.

[0052] The device transmission unit 725 transmits the vehicle coordinate points to the remote control device 5 as information indicating the position of the vehicle 10. The device transmission unit 725 may also transmit various other information to the remote control device 5. Furthermore, the device transmission unit 725 may transmit various information to the vehicle control device 2. At least some of the functions of the device CPU 72 may be implemented as functions of the remote control device 5, the vehicle control device 2, or the imaging device 9.

[0053] A2. Vehicle operation control method: Figure 6 is a flowchart showing the first operation control method. The first operation control method is a method for controlling the driving operation of the vehicle 10 in order to start the vehicle 10 driving in remote automatic driving mode.

[0054] When preparations for starting to drive in remote automatic driving mode are complete (step S101: Yes), the vehicle transmitter 223 of the vehicle 10 transmits a ready signal to the remote control device 5 (step S102).

[0055] When the remote acquisition unit 521 of the remote control device 5 receives a ready signal (step S103: Yes), the remote acquisition unit 521 acquires driving information (step S104). At this time, the driving information includes at least driving route information indicating the target driving route. After step S104, the control value creation unit 522 creates a reference control value using the driving information (step S105). After step S105, the remote transmission unit 523 transmits the reference control value created by the control value creation unit 522 to the vehicle 10 (step S106).

[0056] When the vehicle 10's operation control unit 224 receives a reference control value (step S107: Yes), the operation control unit 224 sets the vehicle 10's travel route to a target travel route and starts driving in remote automatic driving mode (step S108). In other words, the operation control unit 224 drives the drive unit 11, steering unit 12, and braking unit 13 mounted on the vehicle 10 so that the vehicle 10 travels along the target travel route.

[0057] Figure 7 is a flowchart of the second operation control method. The second operation control method is a method for correcting the driving route of vehicle 10 that is driving in remote automatic driving mode. The second operation control method is executed after the first operation control method shown in Figure 6 is completed.

[0058] As shown in Figure 7, when a predetermined correction timing is reached (Step S201: Yes), the remote acquisition unit 521 of the remote control device 5 acquires driving information (Step S202). The correction timing is, for example, when the elapsed time from the start of driving in remote automatic driving mode exceeds a predetermined time, or when a predetermined time has elapsed since the previous correction timing. At this time, the driving information includes at least driving route information indicating the target driving route and vehicle coordinate points. After Step S202, the control value creation unit 522 creates a corrected control value using the driving information (Step S203). After Step S203, the remote transmission unit 523 transmits the corrected control value created by the control value creation unit 522 to the vehicle 10 (Step S204).

[0059] If the operation control unit 224 of the vehicle 10 receives a correction control value (step S205: Yes), the operation control unit 224 corrects the vehicle 10's travel route to a corrected travel route according to the correction control value and continues driving in remote automatic driving mode (step S206). In other words, the operation control unit 224 drives the drive unit 11, steering unit 12, and braking unit 13 mounted on the vehicle 10 so that the vehicle 10 travels along the corrected travel route. When step S206 is completed, the process returns to step S201, and each step from step S201 to step S206 is repeated.

[0060] A-3. Method for estimating the vehicle's position: Figure 8 is a flowchart illustrating the vehicle 10 position estimation method in the first embodiment. The position of the vehicle 10 estimated by the position estimation method is transmitted to the remote control device 5 as one of the driving information and is used to create control values ​​that control the driving operation of the vehicle 10. Therefore, the position estimation method is performed, for example, before starting the second operation control method shown in Figure 7. In other words, the position estimation method is performed at a timing in step S202 of the second operation control method when the vehicle coordinate points indicating the position of the vehicle 10 can be provided from the position estimation device 7 to the remote control device 5 as one of the driving information.

[0061] Figure 9 is a schematic diagram showing examples of various images obtained when the position estimation method shown in Figure 8 is performed. Figure 9 is labeled with step numbers corresponding to each step in Figure 8. In this embodiment, we will explain using the example of a vehicle 10 moving (traveling) along a road surface 60 on which a mesh-like grid 61 is drawn along the Xg axis parallel to the direction of travel of the vehicle 10 and the Yg axis perpendicular to the Xg axis. The Xg axis and Yg axis are coordinate axes of the global coordinate system, respectively. In other embodiments, the grid 61 may be omitted.

[0062] In the position estimation method, first, an image acquisition step (step S31) is performed. The image acquisition step is a step of acquiring an image Im1 that includes the vehicle 10 that is the target of position estimation. In the image acquisition step, the device acquisition unit 721 acquires the image Im1 acquired by the imaging device 9.

[0063] Following the image acquisition process, the position calculation process (step S32) is performed. The position calculation process is a process of calculating image coordinate points using the captured image Im1 acquired in the image acquisition process.

[0064] In the position calculation process, a distortion correction process (step S321) is performed first. The distortion correction process corrects the distortion of the captured image Im1. In the distortion correction process, the distortion correction unit 722a generates a corrected image Im2 by correcting the distortion of the captured image Im1. Specifically, the distortion correction unit 722a corrects the distortion of the captured image Im1 using, for example, a distortion correction parameter Pa1 that is pre-stored in the device storage unit 73. The distortion correction parameter Pa1 is, for example, a parameter related to the position information of the grid lines 61 obtained by calibration. However, the distortion correction method is not limited to this. The distortion correction parameter Pa1 may be any other parameter other than the grid lines 61.

[0065] Following the distortion correction process, a rotation process (step S322) is performed. The rotation process is a process of rotating the corrected image Im2 so that the direction of the movement vector for the vehicle 10 included in the corrected image Im2 points in a predetermined direction. In the rotation process, the rotation processing unit 722b rotates the corrected image Im2 so that the direction of the movement vector for the vehicle 10 included in the corrected image Im2 points in a predetermined direction. As a result, the rotation processing unit 722b generates a rotated image Im3. Specifically, the rotation processing unit 722b rotates the corrected image Im2 around the center of gravity of the vehicle 10 in the corrected image Im2, for example, so that the direction of the movement vector of the vehicle 10 points upward on the screen of a display device (not shown) that displays the corrected image Im2. The movement of the feature points (center of gravity) of the vehicle 10 can be represented as the direction of the movement vector, for example, by the optical flow method. The amount and direction of the vehicle 10's movement vector are estimated, for example, based on the change in the position of the vehicle 10's feature points, which are appropriately set on the modified image Im2, between image frames. Note that the rotation processing method is not limited to this.

[0066] Following the rotation process, a cropping process (step S323) is executed. The cropping process is a process of generating a processed image Im4 including the target region from the rotated image Im3. In this embodiment, in the cropping process, when the vehicle 10 moves a distance exceeding a predetermined threshold, the cropping processing unit 722c deletes the moved region A2 corresponding to the distance the vehicle 10 has moved from the rotated image Im3 as an unnecessary region. As a result, the cropping processing unit 722c generates a processed image Im4 from the rotated image Im3, with the unmoved region A1 including the vehicle 10 cut out as the target region. At this time, the cropping processing unit 722c estimates the moved region A2 by recognizing the distance the vehicle 10 has moved, for example, by the estimated movement vector amount of the vehicle 10. Note that in the position calculation process, either the rotation process or the cropping process may be executed first. Also, the cropping method is not limited to this.

[0067] As shown in Figure 8, following the cropping process, a detection process (step S324) is performed. The detection process is a process of detecting the outline (contour) of the vehicle 10 contained in the processed image Im4 using a position detection model Md1. In the detection process, the detection unit 722d inputs the processed image Im4 to the position detection model Md1. As a result, as shown in Figure 9, the detection unit 722d detects the vehicle 10 contained in the processed image Im4 and generates a first masked image Im5 with a masked area Ms added that masks the vehicle 10 in the processed image Im4. Note that the method for detecting the outline of the vehicle 10 is not limited to this.

[0068] As shown in Figure 8, following the detection step, a perspective transformation step (step S325) is performed. The perspective transformation step is a step of perspective transformation of the first mask image Im5 shown in Figure 9. In the perspective transformation step, the perspective transformation unit 722e generates a second mask image Im6 by perspective transformation of the first mask image Im5. Specifically, the perspective transformation unit 722e uses, for example, a perspective transformation parameter Pa2 pre-stored in the device storage unit 73 to perspective transform the first mask image Im5 into a bird's-eye view image viewed from above the vehicle 10 which is approximately perpendicular to the road surface 60 (for example, directly above the vehicle 10). The perspective transformation parameter Pa2 is, for example, a parameter related to the position information and internal parameters of the imaging device 9 obtained by calibration. As a result, the perspective transformation unit 722e generates a second mask image Im6, which is represented in image coordinates, from the first mask image Im5, which is represented in camera coordinates. The image coordinate system is one in which the origin is a point on the image plane projected by the perspective transformation, and the coordinate axes are defined by the Xi axis and the Yi axis which is orthogonal to the Xi axis. Note that the perspective transformation method is not limited to this. The perspective transformation parameter Pa2 may be any other parameter besides those described above.

[0069] As shown in Figure 8, the coordinate point calculation step (step S326) is performed after the perspective transformation step. Figure 10 is a diagram illustrating the details of the coordinate point calculation step. The coordinate point calculation step is the process of calculating the image coordinate point P3.

[0070] In the coordinate point calculation process, the coordinate point calculation unit 722f first obtains the base coordinate point P0 from the first bounding rectangle R1 set in the mask region Ms in the first mask image Im5, which is the image before perspective transformation. Figure 11 is a diagram illustrating the method for obtaining the base coordinate point P0. To obtain the base coordinate point P0, the coordinate point calculation unit 722f first sets the base bounding rectangle R0 for the mask region Ms in the first mask image Im5. Next, the coordinate point calculation unit 722f rotates the first mask image Im5 by the required amount, with the centroid C of the mask region Ms as the rotation center, so that the direction of the movement vector V of the vehicle 10 corresponding to the mask region Ms in the first mask image Im5 points in a predetermined direction. The predetermined direction is, for example, the upward direction on the screen of a display device (not shown). Next, the coordinate point calculation unit 722f sets a first circumscribed rectangle R1 on the mask region Ms of the rotated first mask image Im5 such that its longer side is parallel to the direction of the movement vector V. Next, the coordinate point calculation unit 722f rotates the first mask image Im5, to which the first circumscribed rectangle R1 has been added, in the opposite direction by the above amount, using the centroid C of the mask region Ms as the rotation center. As a result, the coordinate point calculation unit 722f obtains the coordinate point of one of the four vertices of the first circumscribed rectangle R1 that has the coordinate closest to the positioning point 10e of the vehicle 10 as the base coordinate point P0. Then, as shown in Figure 10, the coordinate point calculation unit 722f performs a perspective transformation on the first mask image Im5 after the reverse rotation, that is, on the first mask image Im5 after obtaining the base coordinate point P0. As a result, the coordinate point calculation unit 722f obtains the first coordinate point P1 as the coordinate point corresponding to the base coordinate point P0 in the first circumscribing rectangle R1 that has been deformed by perspective transformation.

[0071] Furthermore, the coordinate point calculation unit 722f sets a second bounding rectangle R2 for the mask region Ms in the second mask image Im6, which is obtained by perspective transforming the first mask image Im5. Then, the coordinate point calculation unit 722f acquires the vertex of the second bounding rectangle R2 that indicates the same position as the first coordinate point P1 as the second coordinate point P2. Specifically, the coordinate point calculation unit 722f acquires the vertex of the second bounding rectangle R2 that has the coordinate closest to the positioning point 10e of the vehicle 10 as the second coordinate point P2. In other words, the first coordinate point P1 and the second coordinate point P2 are coordinate points that indicate the same position and therefore have a correlation with each other.

[0072] Furthermore, the coordinate point calculation unit 722f performs a correction by replacing the coordinates (Xi1, Yi1) of the first coordinate point P1 with the coordinates (Xi2, Yi2) of the second coordinate point P2, depending on the relative magnitudes of the coordinate values ​​of the first coordinate point P1 and the second coordinate point P2. If the coordinate value Xi1 of the first coordinate point P1 in the Xi axis direction is greater than the coordinate value Xi2 of the second coordinate point P2 in the Xi axis direction (Xi1 > Xi2), the coordinate point calculation unit 722f replaces the coordinate value Xi1 of the first coordinate point P1 in the Xi axis direction with the coordinate value Xi2 of the second coordinate point P2 in the Xi axis direction. If the coordinate value Yi1 of the first coordinate point P1 in the Yi-axis direction is greater than the coordinate value Yi2 of the second coordinate point P2 in the Yi-axis direction (Yi1 > Yi2), the coordinate point calculation unit 722f replaces the coordinate value Yi1 of the first coordinate point P1 in the Yi-axis direction with the coordinate value Yi2 of the second coordinate point P2 in the Yi-axis direction. In this embodiment, the coordinate value Xi1 of the first coordinate point P1 in the Xi-axis direction is greater than the coordinate value Xi2 of the second coordinate point P2 in the Xi-axis direction. Also, the coordinate value Yi1 of the first coordinate point P1 in the Yi-axis direction is smaller than the coordinate value Yi2 of the second coordinate point P2 in the Yi-axis direction. Therefore, the image coordinate point P3 has the coordinates (Xi2, Yi1). In this way, the coordinate point calculation unit 722f calculates the image coordinate point P3, which indicates the position (estimated position) of the vehicle 10 in the image coordinate system, by correcting the first coordinate point P1 using the second coordinate point P2. Note that this is not the only method for calculating the image coordinate point P3.

[0073] As shown in Figure 8, following the position calculation step, the vehicle type identification step (step S33) is executed. The vehicle type identification step is a step in which the type of vehicle 10 included in the captured image Im1 is identified. In the vehicle type identification step, the vehicle type identification unit 723 inputs the captured image Im1 to the vehicle type identification model Md2. As a result, the vehicle type identification unit 723 identifies the type of vehicle 10 by obtaining vehicle type identification information that indicates the type of vehicle 10 included in the captured image Im1. Note that the method for identifying the type of vehicle 10 is not limited to this.

[0074] Following the vehicle type identification process, the parameter acquisition process (step S34) is executed. The parameter acquisition process is the process of acquiring imaging parameters and vehicle parameters to be substituted into the relational expression. In the parameter acquisition process, the device acquisition unit 721 refers to the vehicle database D1 and acquires the values ​​of vehicle parameters associated with vehicle type identification information indicating the type identified by the vehicle type identification unit 723. In this embodiment, the height h of the positioning point 10e from the road surface 60 in the vehicle 10 differs for each type of vehicle 10 (Figure 12 described later). Therefore, in this embodiment, the vehicle parameter acquired in the parameter acquisition process is the height h of the positioning point 10e from the road surface 60 in the vehicle 10 of the type identified by the vehicle type identification unit 723. Furthermore, the device acquisition unit 721 refers to the camera database D2 and acquires the values ​​of imaging parameters associated with camera identification information received together with the captured image Im1 including the vehicle 10 that is the target of position estimation. In this embodiment, in order to convert image coordinate points to vehicle coordinate points using the similarity relationship between vehicle parameters and imaging parameters, the imaging parameters acquired in the parameter acquisition step are as follows. In this case, the imaging parameter is the height H (Figure 12, described later) of the imaging device 9 from the road surface 60 that acquired the image Im1 including the vehicle 10 that is the subject of position estimation, and is the height H relative to the road surface 60 on which the vehicle 10 is located. Note that the method of acquiring vehicle parameters and imaging parameters is not limited to this.

[0075] As shown in Figure 8, following the parameter acquisition step, a position transformation step (step S37) is performed. The position transformation step is a process of calculating vehicle coordinate points that indicate the position of the vehicle 10 to be targeted for position estimation in the global coordinate system by transforming image coordinate points P3 into vehicle coordinate points. In the position transformation step, the position transformation unit 724 transforms image coordinate points into vehicle coordinate points using the vehicle parameters and imaging parameters acquired in the parameter acquisition step. In this embodiment, the position transformation unit 724 transforms image coordinate points P3 into vehicle coordinate points using relational expressions represented by equations (1) to (3), which will be described later, with vehicle coordinate points as the target variable and image coordinate points P3, imaging parameters, and vehicle parameters as explanatory variables. In this case, the position transformation unit 724 substitutes the coordinate values ​​of image coordinate points P3 calculated by the position calculation unit 722 into the relational expressions represented by equations (1) to (3). The position conversion unit 724 substitutes the values ​​of the imaging parameters acquired by the device acquisition unit 721, that is, the values ​​of the imaging parameters corresponding to the imaging device 9 that acquired the captured image Im1, into the relational expressions represented by equations (1) to (3). Furthermore, the position conversion unit 724 substitutes the values ​​of the vehicle parameters acquired by the device acquisition unit 721, that is, the values ​​of the vehicle parameters corresponding to the type of vehicle 10 to be targeted for position estimation, into the relational expressions represented by equations (1) to (3).

[0076] Figure 12 is the first figure illustrating the details of the position transformation process in the first embodiment. Figure 12 shows the state of the vehicle 10 as viewed from the left side. Figure 13 is the second figure illustrating the details of the position transformation process in the first embodiment. Figure 13 shows the state of the vehicle 10 as viewed from the roof side. The global coordinate system shown in Figures 12 and 13 is a coordinate system having a fixed coordinate point Pf as the origin, which indicates an arbitrary reference position on the road surface 60, and having coordinate axes indicated by the Xg axis and the Yg axis which is orthogonal to the Xg axis. The imaging coordinate point Pc is the position of the imaging device 9 that acquired the captured image Im1 used to calculate the image coordinate point P3, and is a coordinate point that indicates the position of the imaging device 9 in the global coordinate system. The fixed coordinate point Pf and the imaging coordinate point Pc are stored in advance in the device storage unit 73.

[0077] As shown in Figure 12, let Do be the observed distance on the XgYg plane between the position of the imaging device 9 and the position of the vehicle 10 (image coordinate point P3). Let ΔD be the observation error. imaging Let H be the height [m] of the imaging device 9 from the road surface 60 as a parameter. Let h be the height [m] of the positioning point 10e of the vehicle 10 from the road surface 60 as an acquired vehicle parameter. In this case, the observation error ΔD can be expressed by the following equation (1). ΔD=h / H×Do Equation (1) In other words, the larger the observation distance Do, the larger the observation error ΔD becomes.

[0078] Next, if we define D as the actual distance between the position of the imaging device 9 and the position of the vehicle 10's positioning point 10e, then the first actual distance D can be expressed by the following equation (2). D=Do×(1-h / H) Equation (2) In other words, the first actual distance D is determined by the observation distance Do, the height H of the imaging device 9 as an imaging parameter, and the height h of the positioning point 10e of the vehicle 10 as a vehicle parameter.

[0079] Then, as shown in Figure 13, if Dp is the estimated distance between the reference position and the estimated position of the vehicle 10, and Dt is the actual distance between the reference position and the position of the vehicle 10 (hereinafter referred to as the second actual distance), the second actual distance Dt can be expressed by the following equation (3). Dt = Dp × (1 - h / H) Equation (3)

[0080] Here, the estimated distance Dp can be calculated using the actual distance obtained from the fixed coordinate point Pf and the imaging coordinate point Pc (hereinafter referred to as the third actual distance Dc), the image coordinate point P3, and the fixed coordinate point Pf. Therefore, the position transformation unit 724 can calculate the vehicle coordinate point Pv using the second actual distance Dt, which is obtained by correcting the estimated distance Dp using the above equation (3), and the fixed coordinate point Pf. At this time, the calculated vehicle coordinate point Pv is a coordinate point that indicates the position of the vehicle 10 in the global coordinate system, and therefore corresponds to the position of the vehicle 10 in real space. In this way, the position transformation unit 724 can convert the image coordinate point P3 to the vehicle coordinate point Pv by substituting the acquired imaging parameter values ​​and vehicle parameter values ​​into the relational equations expressed by equations (1) to (3). In order for the remote control device 5 to create appropriate control values ​​according to the current location of the vehicle 10, the vehicle 10 position estimation method described above is repeatedly executed, for example, at predetermined intervals.

[0081] According to the first embodiment described above, when the vehicle 10 is driven automatically by remote control, the position of the vehicle 10 can be estimated using the captured image Im1 obtained by capturing the imaging area RG including the vehicle 10 from outside the vehicle 10. In this way, the position of the vehicle 10 can be estimated without installing any devices such as sensors, markers, and transceivers on the vehicle 10. Furthermore, the position of the vehicle 10 can be estimated without mounting the position estimation device 7 on the vehicle 10. This increases the versatility of the position estimation device 7.

[0082] Furthermore, according to the first embodiment described above, when estimating the position of the vehicle 10 using the captured image Im1, the distortion of the captured image Im1 can be corrected. This further improves the accuracy of estimating the position of the vehicle 10.

[0083] Furthermore, according to the first embodiment described above, when estimating the position of the vehicle 10 using the captured image Im1, the modified image Im2 generated by processing the captured image Im1 can be rotated so that the direction of the vehicle 10's movement vector V points in a predetermined direction. In this way, the vehicle 10 included in the captured image Im1 can be detected with a unified direction of the movement vector V. This further improves the accuracy of estimating the position of the vehicle 10.

[0084] Furthermore, according to the first embodiment described above, by performing a cropping process to cut out the necessary region including the vehicle 10 from the rotated image Im3 generated by processing the captured image Im1, a processed image Im4 can be generated with unnecessary regions removed. In this way, the vehicle 10 can be detected while excluding elements other than the vehicle 10 that is the target of detection from the image. As a result, the vehicle 10 included in the captured image Im1 can be detected with high accuracy, and the accuracy of estimating the position of the vehicle 10 can be further improved.

[0085] Furthermore, according to the first embodiment described above, a processed image Im4 can be generated by performing a cropping process that cuts out the unmoved region A1 including the vehicle 10 from the rotated image Im3 generated by processing the captured image Im1. In this way, the area occupied by the vehicle 10 in the image can be increased compared to when the cropping process is not performed. This makes it easier to detect the vehicle 10 which is further away from the imaging device 9. Therefore, the accuracy of image processing for the vehicle 10 which is further away from the imaging device 9 can be further improved.

[0086] Furthermore, according to the first embodiment described above, the outline of the vehicle 10 can be detected by inputting the processed image Im4, which is generated by processing the captured image Im1, into the position detection model Md1, which is a trained machine learning model. This makes it possible to generate a first masked image Im5 in which a masked region Ms, which masks the region corresponding to the vehicle 10 from the processed image Im4, is added. In this case, according to the first embodiment described above, a DNN having the structure of a CNN that realizes semantic segmentation or instance segmentation can be used as the algorithm of the position detection model Md1. This makes it possible to suppress the decrease in the accuracy of estimating the position of the vehicle 10 due to the diversity of the background region in the captured image Im1. The background region is the region of the captured image Im1 other than the region corresponding to the vehicle 10.

[0087] Furthermore, according to the first embodiment described above, a second mask image Im6 can be generated by perspective transforming the first mask image Im5. This makes it possible to convert the camera coordinate system to the image coordinate system.

[0088] Furthermore, according to the first embodiment described above, by setting a first bounding rectangle R1 on the mask region Ms before perspective transformation of the first mask image Im5, the base coordinate point P0, which is the vertex of the first bounding rectangle R1 having the coordinates closest to the positioning point 10e of the vehicle 10, can be calculated. Then, by performing perspective transformation on the first mask image Im5 after calculating the base coordinate point P0, the first coordinate point P1, which is the coordinate point corresponding to the base coordinate point P0, can be calculated. Furthermore, by setting a second bounding rectangle R2 on the mask region Ms after perspective transformation of the first mask image Im5, the second coordinate point P2, which is the vertex of the second bounding rectangle R2 having the coordinates closest to the positioning point 10e of the vehicle 10, can be calculated. Then, by correcting the first coordinate point P1 using the second coordinate point P2, the image coordinate point P3 can be calculated. In this way, by comparing and correcting the coordinate points before and after perspective transformation, the image coordinate point P3 can be calculated more accurately. This further improves the accuracy of estimating the position of the vehicle 10.

[0089] Furthermore, according to the first embodiment described above, by inputting the captured image Im1 to the vehicle identification model Md2, which is a trained machine learning model, various images Im2 to Im4 generated by processing the captured image Im1, the type of vehicle 10 contained in the captured image Im1 can be identified. In other words, the type of vehicle 10 can be identified using machine learning.

[0090] Furthermore, according to the first embodiment described above, by referring to the vehicle database D1 pre-stored in the device storage unit 73, which links each type of vehicle 10 with vehicle parameters, the values ​​of vehicle parameters corresponding to the type of vehicle 10 can be obtained. Also, by referring to the camera database D2 pre-stored in the device storage unit 73, which links each imaging device 9 with imaging parameters, the values ​​of imaging parameters related to the imaging device 9 corresponding to the captured image Im1 can be obtained.

[0091] Furthermore, according to the first embodiment described above, when estimating the position of vehicle 10 using the captured image Im1, the position of vehicle 10 can be estimated using vehicle parameters determined according to the type of vehicle 10. In other words, the position of vehicle 10 can be estimated while taking into account the type of vehicle 10. This makes it possible to suppress a decrease in the accuracy of estimating the position of vehicle 10 even when the vehicle class, etc., differs depending on the type of vehicle 10. In other words, when estimating the position of vehicle 10 using the captured image Im1, the position of vehicle 10 can be estimated with high accuracy by using vehicle parameters determined according to the type of vehicle 10.

[0092] Furthermore, according to the first embodiment described above, the position of the vehicle 10 can be estimated taking into account the type of vehicle 10. As a result, even when multiple types of vehicles 10 are driving in remote automatic driving mode, the position of the vehicle 10 can be appropriately estimated according to the type of vehicle 10.

[0093] Furthermore, according to the first embodiment described above, when converting an image coordinate point P3 to a vehicle coordinate point Pv, the vehicle coordinate point Pv is used as the target variable, and the values ​​of the vehicle parameters corresponding to the type of vehicle 10 and the values ​​of the imaging parameters corresponding to the captured image Im1 are substituted into a relational expression that includes the image coordinate point P3, imaging parameters, and vehicle parameters as explanatory variables. This makes it possible to convert the image coordinate point P3 to a vehicle coordinate point Pv.

[0094] Furthermore, according to the first embodiment described above, the imaging parameter is the height H of the imaging device 9 from the road surface 60, calculated based on the position of the imaging device 9 in the global coordinate system. The vehicle parameter is the height h of the predetermined positioning point 10e of the vehicle 10 from the road surface 60. In this way, the observation error ΔD can be calculated based on the similarity relationship between the imaging parameter and the vehicle parameter. Then, using the calculated observation error ΔD, the image coordinate point P3 can be converted to the vehicle coordinate point Pv.

[0095] Furthermore, according to the first embodiment described above, the position of the vehicle 10 can be estimated without using data from GNSS satellites or the like. This makes it possible to estimate the position of the vehicle 10 even when the vehicle 10 is traveling on a road surface 60 located indoors or in other places where it is difficult to receive data from GNSS satellites.

[0096] B. Second Embodiment: Figure 14 is a diagram showing the details of the device CPU 72a in the second embodiment. Figure 15 is a flowchart showing the vehicle 10 position estimation method in the second embodiment. In this embodiment, the conversion method when converting image coordinate point P3 to vehicle coordinate point Pv differs from that of the first embodiment. Specifically, in this embodiment, the device CPU 72 further includes a correction value setting unit 726. The correction value setting unit 726 sets the correction parameter by calculating the difference between the vehicle parameter acquired by the device acquisition unit 721 and the reference parameter. Then, the position conversion unit 724 converts the image coordinate point P3 to the vehicle coordinate point Pv using the vehicle coordinate point Pv as the target variable and the relational expressions represented by equations (4) to (6) described later, which include the image coordinate point P3, imaging parameter, reference parameter, and correction parameter as explanatory variables. The reference parameter is a parameter that indicates the reference value of the vehicle parameter. The reference parameter is, for example, the vehicle parameter of a predetermined type of vehicle 10. The reference parameter is, for example, stored in advance in the device storage unit 73. The correction parameter is a parameter determined according to the type of vehicle 10, and is a parameter that indicates a correction value for the reference parameter. Therefore, in this embodiment, some of the functions of the device CPU 72 and some of the vehicle 10 position estimation method differ from the first embodiment. Other configurations are the same as in the first embodiment. Steps that are the same as in the first embodiment and configurations that are the same as in the first embodiment are denoted by the same reference numerals and their descriptions are omitted.

[0097] As shown in Figure 15, the correction value setting step (step S36) is performed after the parameter acquisition step. In the correction value setting step, the correction value setting unit 726 calculates the difference between the vehicle parameters acquired by the device acquisition unit 721 and the reference parameters, and sets a correction parameter determined according to the type of vehicle 10 identified by the vehicle type identification unit 723. If the correction parameter is greater than the reference parameter, the correction parameter is a positive value. If the correction parameter is less than the reference parameter, the correction parameter is a negative value. If the correction parameter is the same as the reference parameter, the correction parameter is zero.

[0098] Following the correction value setting step, the position conversion step (step S38) is executed. In the position conversion step, the position conversion unit 724 converts the image coordinate point P3 to the vehicle coordinate point Pv using the relational expressions represented by equations (4) to (6). Specifically, the position conversion unit 724 substitutes the coordinate value of the image coordinate point P3 calculated by the position calculation unit 722 into the relational expressions represented by equations (4) to (6). The position conversion unit 724 substitutes the values ​​of the imaging parameters acquired by the device acquisition unit 721 into the relational expressions represented by equations (4) to (6). The position conversion unit 724 substitutes the reference parameters pre-stored in the device storage unit 73 into the relational expressions represented by equations (4) to (6). Furthermore, the position conversion unit 724 substitutes the correction parameters set by the correction value setting unit 726, that is, the correction parameters corresponding to the type of vehicle 10 to be targeted for position estimation, into the relational expressions represented by equations (4) to (6). The details of the position transformation process in this embodiment will be described below.

[0099] Figure 16 is the first figure illustrating the details of the position change process in the second embodiment. Figure 16 shows the state of the vehicle 10 as viewed from the left side. Figure 17 is the second figure illustrating the details of the position change process in the second embodiment. Figure 17 shows the state of the vehicle 10 as viewed from the roof side.

[0100] As shown in Figure 16, let Do be the observed distance on the XgYg plane between the position of the imaging device 9 and the position of the vehicle 10 (image coordinate point P3). Let ΔD be the observation error. Let H be the height [m] of the imaging device 9 from the road surface 60 as an acquired imaging parameter. Let h be the height [m] of the positioning point 10e of the vehicle 10 from the road surface 60 as an acquired vehicle parameter. Let hs be the reference parameter. In this embodiment, the reference parameter hs is the height [m] of the positioning point 10e of a predetermined type of vehicle 10 from the road surface 60. Let Δh be the correction parameter. In this embodiment, the correction parameter Δh is the difference between the height hs as a reference parameter and the height h of the positioning point 10e of the vehicle 10 targeted for position estimation from the road surface 60. In this case, the observation error ΔD can be expressed by the following equation (4). ΔD=(hs+Δh) / H×Do Equation (4) In other words, the larger the observation distance Do, the larger the observation error ΔD becomes.

[0101] Next, if we define the first actual distance D as the actual distance between the position of the imaging device 9 and the position of the positioning point 10e of the vehicle 10, then the first actual distance D can be expressed by the following equation (5). D=Do×[{1-(hs+Δh)} / H] Equation (5) In other words, the first actual distance D is determined by the observation distance Do, the height H of the imaging device 9 as an imaging parameter, and the correction parameter Δh.

[0102] Then, as shown in Figure 17, if Dp is the estimated distance between the reference position and the estimated position of the vehicle 10, and Dt is the second actual distance which is the actual distance between the reference position and the position of the vehicle 10, the second actual distance Dt can be expressed by the following equation (6). Dt=Dp×[{1-(hs+Δh)} / H] Equation (6)

[0103] Here, the estimated distance Dp can be calculated using the third actual distance Dc, the image coordinate point P3, and the fixed coordinate point Pf, similar to the first embodiment. Therefore, the position transformation unit 724 can calculate the vehicle coordinate point Pv using the second actual distance Dt, which is obtained by correcting the estimated distance Dp using the above equation (6), and the fixed coordinate point Pf. In this way, the image coordinate point P3 can be converted to the vehicle coordinate point Pv by substituting the acquired imaging parameter value and the set correction parameter Δh value into the relational equations expressed by equations (4) to (6).

[0104] According to the second embodiment described above, when converting an image coordinate point P3 to a vehicle coordinate point Pv, the vehicle coordinate point Pv is used as the objective variable, and the imaging parameter corresponding to the captured image Im1 and the correction parameter Δh set according to the type of vehicle 10 are substituted into a relational expression that includes the image coordinate point P3, imaging parameters, reference parameter hs, and correction parameter Δh as explanatory variables. This makes it possible to convert the image coordinate point P3 to a vehicle coordinate point Pv.

[0105] C. Other embodiments: C-1. Other Embodiments 1: In other embodiments, if the type of vehicle 10 identified by the vehicle type identification unit 723 is a predetermined type, the correction value setting unit 726 may set the correction parameter Δh to zero without calculating the difference between the vehicle parameter and the reference parameter hs. The predetermined type is, for example, a type in which the difference with the reference parameter hs is less than a predetermined threshold. In this case, the vehicle database D1 may further include judgment information for determining whether the type of vehicle 10 identified by the vehicle type identification unit 723 is a type in which the difference with the reference parameter hs is less than a predetermined threshold. The judgment information is, for example, information linking vehicle type identification information and correction necessity information. The correction necessity information is information indicating whether the type is a type in which the difference with the reference parameter hs is less than a predetermined threshold. The correction value setting unit 726 determines whether to set the correction parameter Δh to zero by referring to the correction necessity information corresponding to the vehicle type identification information indicating the type of vehicle 10 identified by the vehicle type identification unit 723. In this configuration, if the difference between the vehicle parameters and the reference parameter hs for the vehicle 10 to be positioned is less than a predetermined threshold, the position of the vehicle 10 can be estimated without considering the correction parameter Δh corresponding to the type of vehicle 10. In other words, if the values ​​of the vehicle parameters for the vehicle 10 to be positioned are equal to or close to the values ​​of the reference parameter hs, the position of the vehicle 10 can be estimated without correcting the vehicle parameters. This simplifies the processing steps involved in setting the correction parameter Δh.

[0106] C-2. Other Embodiments 2: In other embodiments, the position estimation device 7 may further include a direction information generation unit as a function of the device CPU 72. The direction information generation unit generates direction information indicating the direction of movement of the vehicle 10 using a plurality of vehicle coordinate points Pv obtained at different intervals. In other words, the direction information generation unit generates direction information for the same vehicle 10 using a plurality of vehicle coordinate points Pv acquired at different timings. Specifically, the direction information generation unit generates direction information by, for example, arranging vehicle coordinate points Pv generated from a plurality of captured images Im1 captured by different imaging devices 9 in chronological order. With this configuration, the actual trajectory of the vehicle 10 can be calculated. Furthermore, with this configuration, the direction of travel of the vehicle 10 can be estimated. This allows, for example, when the position estimation method shown in Figures 8 and 15 is repeatedly executed at predetermined intervals, the direction of movement of the moving vehicle 10 can be estimated using the direction information.

[0107] C-3. Other Embodiments 3: In other embodiments, the vehicle type identification unit 723 may identify the type of vehicle 10 included in the captured image Im1 using scheduled information pre-stored in the device storage unit 73. The scheduled information indicates which type of vehicle 10 is scheduled to travel to which position on the target travel route at which time. The scheduled information is, for example, information that links vehicle type identification information, imaging position information, and timing information. The imaging position information is information that indicates the imaging area RG of the imaging device 9, which is identified by the camera identification information. Therefore, the imaging position information is information that indicates the position range on the target travel route. The timing information is information that indicates the scheduled time when the type of vehicle 10 identified by the vehicle type identification information will travel within the imaging area RG identified by the imaging position information. However, the scheduled information is not limited to this. When multiple vehicles 10 of one type are produced continuously in a factory, and the imaging device 9 acquires the captured image Im1 in order to estimate the position of the vehicles 10 traveling on the target travel route in the factory during the production process, the scheduled information may be the following information. In this case, the scheduled information may include process location information instead of imaging location information. Process location information is information indicating the location of the workplace where a production process identified by process identification information (process ID), which identifies multiple production processes executed in the production process of the vehicle 10, is performed. Therefore, the process location information is information indicating the position range on the target driving route. Furthermore, when the scheduled information includes process location information instead of imaging location information, the timing information is information indicating the scheduled time for executing the production process identified by the process identification information for a vehicle 10 of the type identified by the vehicle type identification information. In other words, in this case, the scheduled information is production plan information indicating the production plan for each type of vehicle 10. In this form, the type of vehicle 10 included in the captured image Im1 can be identified using the scheduled information. As a result, the type of vehicle 10 included in the captured image Im1 can be identified without performing image analysis, thus reducing the processing load of the vehicle type identification unit 723 in the vehicle type identification process.

[0108] C-4. Other Embodiments 4: In the position calculation process, the distortion correction process is not a mandatory step. For example, if the distortion correction process is not performed in the position calculation process, and the subsequent steps from the rotation processing step onward are executed, the rotation processing unit 722b will perform the following processing. In this case, in the rotation processing step, the rotation processing unit 722b rotates the captured image Im1 instead of the corrected image Im2. Even in this configuration, the image coordinate point P3 can be calculated.

[0109] C-5. Other Embodiments 5: In the position calculation process, the rotation process is not a mandatory step. For example, if the distortion correction process and the rotation process are not performed in the position calculation process, the cropping unit 722c performs the following process. In other words, if only the steps from the cropping process onward are performed in the position calculation process, the cropping unit 722c performs the following process. In this case, the cropping unit 722c performs the cropping process on the captured image Im1 instead of the rotated image Im3. Even in this configuration, the image coordinate point P3 can be calculated.

[0110] C-6. Other Embodiments 6: In the position calculation process, the cropping process is not a mandatory step. For example, if the distortion correction process, rotation process, and cropping process are not performed in the position calculation process, the detection unit 722d performs the following process. In other words, if only the processes from the detection process onward are performed in the position calculation process, the detection unit 722d performs the following process. In this case, the detection unit 722d generates a first mask image Im5 by masking the vehicle 10 included in the captured image Im1 instead of the processed image Im4. Even in this configuration, the image coordinate point P3 can be calculated.

[0111] C-7. Other Embodiments 7: In other embodiments, the captured image Im1 may include multiple vehicles 10. In this case, the device CPU 72 may further include a deletion unit that deletes, for example, mask regions Ms of vehicles 10 that are not subject to position estimation from the first mask image Im5. The deletion unit determines, for example, that among the mask regions Ms generated in the detection step, mask regions Ms that exist outside the recognition target region are mask regions Ms of vehicles 10 that are not subject to position estimation, and deletes them from the first mask image Im5. The recognition target region is, for example, a predetermined region in the first mask image Im5 where the vehicles 10 move. The predetermined region where the vehicles 10 move is, for example, a region corresponding to the region of the grid lines 61. The recognition target region is stored in advance in the device storage unit 73. In this configuration, when multiple vehicles 10 are captured in the captured image Im1, the influence of vehicles 10 that are not subject to position estimation can be eliminated. This improves the accuracy of estimating the position of the vehicles 10.

[0112] C-8. Other Embodiments 8: In other embodiments, the captured image Im1 may include multiple vehicles 10. In this case, a DNN that performs instance segmentation, such as YOLACT++, may be used as the algorithm for the position detection model Md1. In this configuration, the multiple vehicles 10 included in the captured image Im1 can be classified, and a first masked image Im5 can be generated, masked for each vehicle 10. This makes it possible to select the vehicle 10 to be targeted for position estimation when the captured image Im1 includes multiple vehicles 10, and to estimate the position of the selected vehicle 10.

[0113] C-9. Other Embodiments 9: In other embodiments, the captured image Im1 may include multiple vehicles 10. In this case, the vehicle type identification unit 723 uses a vehicle type identification model Md2 capable of classifying multiple vehicles 10 included in the captured image Im1 to identify the types of vehicles 10 that are subject to position estimation, without identifying the types of vehicles 10 that are not subject to position estimation. In this configuration, even if the captured image Im1 includes multiple vehicles 10, the types of vehicles 10 that are subject to position estimation can be identified.

[0114] C-10. Other Embodiments 10: In other embodiments, the position estimation device 7 may estimate the position of a stationary vehicle 10. When estimating the position of a stationary vehicle 10, the position estimation device 7 estimates the position of the vehicle 10 using, for example, the initial vector direction of the vehicle 10 estimated from the first captured image Im1 acquired after the position estimation device 7 is activated, instead of the direction of the movement vector V of the moving vehicle 10. In this configuration, even when the vehicle 10 is stationary, the position of the vehicle 10 can be estimated using the captured image Im1.

[0115] C-11. Other Embodiments 11: In other embodiments, the position estimation device 7 may process 3D images captured by a stereo camera or a TOF (Time of Flight) camera as the imaging device 9. This configuration expands the range of processing targets, thereby increasing the versatility of the position estimation device 7.

[0116] C-12. Other Embodiments 12: In other embodiments, the remote control device 5 and the position estimation device 7 may be configured as a single unit. Furthermore, in yet another embodiment, each part of the position estimation device 7 may be implemented, for example, by cloud computing, which is composed of one or more computers. In such configurations, the configuration of the position estimation device 7 can be changed as appropriate.

[0117] C-13. Other Embodiments 13: In other embodiments, the position estimation device 7 may estimate the position of a vehicle 10 traveling in remote manual driving mode. Remote manual driving mode is a driving mode in which the vehicle 10 travels without a driver on board the vehicle 10, by receiving driving conditions generated by an operator operating an operator input device. In this configuration, the position of the vehicle 10 can be estimated not only in remote automatic driving mode but also in remote manual driving mode. This increases the versatility of the position estimation device 7.

[0118] C-14. Other Embodiments 14: In other embodiments, the device CPU 72 may include two or more functional units that implement the functions of the device acquisition unit 721. For example, the device CPU 72 may include an image acquisition unit that acquires the captured image Im1, and a parameter acquisition unit that acquires vehicle parameters and imaging parameters. Even in such a configuration, the device CPU 72 can acquire various types of information as needed.

[0119] C-15. Other Embodiments 15: In the above embodiment, the vehicle parameter was the height h of the vehicle 10's positioning point 10e from the road surface 60. In other words, in the above embodiment, the vehicle parameter was the distance of the vehicle 10 to the road surface 60 along the height direction. However, the present disclosure is not limited thereto. The vehicle parameter may be, for example, the distance between the road boundary, which serves as a reference point on the vehicle 10's route, and the vehicle 10's positioning point 10e. In other words, the vehicle parameter may be the distance of the vehicle 10 to a reference point along the width direction. In such a configuration, if the dimensions of the vehicle 10 in the width direction (hereinafter referred to as vehicle width) differ depending on the type of vehicle 10, the position of the vehicle 10 can be estimated by using a value related to the vehicle width as the vehicle parameter.

[0120] C-16. Other Embodiments 16: In each of the above embodiments, the vehicle 10 only needs to have a configuration that allows it to be moved by remote control, and may, for example, be in the form of a platform having the configuration described below. Specifically, in order for the vehicle 10 to perform the three functions of "driving," "turning," and "stopping" by remote control, it only needs to be equipped with at least a vehicle control device 2, a drive device 11, a steering device 12, a braking device 13, and a vehicle communication unit 21. That is, a vehicle 10 that can be moved by remote control does not need to have at least some of its interior parts such as a driver's seat and dashboard attached, at least some of its exterior parts such as bumpers and fenders attached, and does not need to have a body shell attached. In this case, the remaining parts such as the body shell may be attached to the vehicle 10 before it leaves the factory for shipment, or the remaining parts such as the body shell may be attached to the vehicle 10 after it has left the factory for shipment, while the remaining parts such as the body shell are not attached to the vehicle 10. Position estimation can also be performed for the platform configuration in the same manner as for the vehicle 10 in each embodiment.

[0121] This disclosure is not limited to the embodiments described above, and can be implemented in various configurations without departing from its spirit. For example, the technical features of the embodiments corresponding to the technical features in each form described in the summary of the invention can be replaced or combined as appropriate in order to solve some or all of the above-described problems, or to achieve some or all of the above-described effects. Furthermore, if a technical feature is not described as essential in this specification, it can be deleted as appropriate. [Explanation of symbols]

[0122] 1...Position estimation system, 2...Vehicle control device, 5...Remote control device, 7...Position estimation device, 9...Imaging device, 10...Vehicle, 10e...Positioning point, 11...Drive device, 12...Steering device, 13...Braking device, 16...External sensor group, 17...Internal sensor group, 21...Vehicle communication unit, 22...Vehicle CPU, 23...Vehicle memory unit, 51...Remote communication unit, 52...Remote CPU, 53...Remote memory unit, 60...Road surface, 61...Grid lines, 71...Device communication unit, 72...Device CPU, 72a...Device CPU, 73...Device memory unit, 160...External sensor, 161...On-board camera 162...Radar, 163...Lider, 170...Internal sensor, 171...Wheel speed sensor, 172...Steering angle sensor, 221...Vehicle speed calculation unit, 222...Vehicle acquisition unit, 223...Vehicle transmission unit, 224...Motion control unit, 521...Remote acquisition unit, 522...Control value creation unit, 523...Remote transmission unit, 721...Device acquisition unit, 722...Position calculation unit, 722a...Distortion correction unit, 722b...Rotation processing unit, 722c...Crop processing unit, 722d...Detection unit, 722e...Perspective conversion unit, 722f...Coordinate point calculation unit, 723...Vehicle type identification unit, 724...Position conversion unit, 72 5...Device transmission unit, 726...Correction value setting unit, 901...First imaging device, 902...Second imaging device, A1...Unmoved area, A2...Moved area, C...Center of gravity, D...First actual distance, ΔD...Observation error, Dc...Third actual distance, Do...Observed distance, Dp...Estimated distance, Dt...Second actual distance, D1...Vehicle database, D2...Camera database, h...Height of vehicle positioning point from road surface, Δh...Correction parameter, hs...Reference parameter, H...Height of imaging device from road surface, Im1...Captured image, Im2...Corrected image, Im3...Rotating image, Im4...After processing Image, Im5…First mask image, Im6…Second mask image, Md1…Position detection model, Md2…Vehicle type identification model, Ms…Mask region, N…Network, P0…Base coordinate point, P1…First coordinate point, P2…Second coordinate point, P3…Image coordinate point, Pa1…Correction parameter, Pa2…Perspective transformation parameter, Pc…Imaging coordinate point, Pf…Fixed coordinate point, Pv…Vehicle coordinate point, R0…Base bounding rectangle, R1…First bounding rectangle, R2…Second bounding rectangle, RG…Imaging region, RG1…First imaging region, RG2…Second imaging region, V…Motion vector

Claims

1. A position estimation device for estimating the position of a vehicle, An acquisition unit that acquires an image of the vehicle including the aforementioned vehicle from an imaging device, A position calculation unit that uses the captured image to calculate an image coordinate point indicating the position of the vehicle in the image coordinate system, A vehicle type identification unit that identifies the type of vehicle included in the captured image, A position transformation unit that transforms the image coordinate points into vehicle coordinate points indicating the position of the vehicle in the global coordinate system, using imaging parameters calculated based on the position of the imaging device in the global coordinate system and vehicle parameters determined according to the type of the vehicle, The system includes a correction value setting unit which calculates the difference between the vehicle parameters acquired by the acquisition unit and a reference parameter indicating a reference value of the vehicle parameters, and sets a correction parameter indicating a correction value for the reference parameter, the correction parameter being determined according to the type of vehicle identified by the vehicle type identification unit, The position conversion unit is a position estimation device that, when converting the image coordinate points to the vehicle coordinate points, uses the vehicle coordinate points as the target variable and substitutes the correction parameter set by the correction value setting unit into a relational expression that includes the image coordinate points, the imaging parameter, the reference parameter, and the correction parameter as explanatory variables.

2. A position estimation device according to claim 1, further, The correction value setting unit is a position estimation device that, when the type of the vehicle identified by the vehicle type identification unit is a predetermined type, sets the correction parameter to zero without calculating the difference.

3. A position estimation device for estimating the position of a vehicle, An acquisition unit that acquires an image of the vehicle including the aforementioned vehicle from an imaging device, A position calculation unit that uses the captured image to calculate an image coordinate point indicating the position of the vehicle in the image coordinate system, A vehicle type identification unit that identifies the type of vehicle included in the captured image, The system includes a position transformation unit that transforms the image coordinate points into vehicle coordinate points indicating the position of the vehicle in the global coordinate system, using imaging parameters calculated based on the position of the imaging device in the global coordinate system and vehicle parameters determined according to the type of the vehicle, The position calculation unit, A detection unit that generates a first mask image in which a mask region is added to the captured image that masks the vehicle, A perspective transformation unit that performs perspective transformation on the first mask image, A position estimation device comprising: a coordinate point calculation unit that sets a specified vertex of a first bounding rectangle set in the mask region of the first mask image as a first coordinate point, sets a vertex of a second bounding rectangle set in the mask region of the second mask image obtained by perspective transforming the first mask image, which indicates the same position as the first coordinate point, as a second coordinate point, and corrects the first coordinate point using the second coordinate point.

4. A position estimation device according to claim 3, The position estimation device further includes a rotation processing unit that rotates the captured image so that the direction of movement of the vehicle faces a predetermined direction.

5. A position estimation device according to claim 4, The position estimation device further includes a distortion correction unit for correcting the distortion of the captured image, wherein the position calculation unit is further equipped with a distortion correction unit.

6. A position estimation device according to claim 1, further, A position estimation device comprising a direction information generation unit that generates direction information indicating the direction of movement of the vehicle using the aforementioned vehicle coordinate points.

7. A position estimation device according to claim 1, The position conversion unit is a position estimation device that, when converting the image coordinate points to the vehicle coordinate points, uses the vehicle coordinate points as the target variable and substitutes the values ​​of the vehicle parameters acquired by the acquisition unit into a relational expression that includes the image coordinate points, the imaging parameters, and the vehicle parameters as explanatory variables.

8. A position estimation device according to claim 1, The imaging parameter is the height of the imaging device from the road surface, calculated based on the position of the imaging device in the global coordinate system. A position estimation device in which the vehicle parameter is the height of a predetermined positioning point of the vehicle from the road surface.

9. A position estimation system for estimating the position of a vehicle, Vehicles and, An imaging device that acquires an image of the vehicle that is the target of position estimation, A position estimation system comprising a position estimation device according to any one of claims 1 to 8.

10. A position estimation method for estimating the position of a vehicle, The acquisition step involves acquiring an image of the vehicle, including the aforementioned vehicle, from an imaging device. A position calculation step of calculating an image coordinate point indicating the position of the vehicle in the image coordinate system using the captured image, A vehicle type identification step that identifies the type of vehicle included in the captured image, A position transformation step that transforms the image coordinate points into vehicle coordinate points indicating the position of the vehicle in the global coordinate system, using imaging parameters calculated based on the position of the imaging device in the global coordinate system and vehicle parameters determined according to the type of the vehicle; The system includes a correction value setting step, which calculates the difference between the vehicle parameters obtained by the acquisition step and a reference parameter indicating a reference value of the vehicle parameters, thereby setting a correction parameter indicating a correction value for the reference parameter, the correction parameter being determined according to the type of the vehicle identified by the vehicle type identification step, A position estimation method in which, when converting the image coordinate points to the vehicle coordinate points in the position conversion step, the vehicle coordinate points are used as the objective variable, and the correction parameters set in the correction value setting step are substituted into a relational expression that includes the image coordinate points, the imaging parameters, the reference parameters, and the correction parameters as explanatory variables.

11. A computer program used to estimate the position of a vehicle, An acquisition function that acquires an image of the vehicle including the aforementioned vehicle from the imaging device, A position calculation function that uses the captured image to calculate an image coordinate point indicating the position of the vehicle in the image coordinate system, A vehicle type identification function that identifies the type of vehicle included in the captured image, A position transformation function that transforms the image coordinate points into vehicle coordinate points indicating the position of the vehicle in the global coordinate system, using imaging parameters calculated based on the position of the imaging device in the global coordinate system and vehicle parameters determined according to the type of the vehicle, A correction value setting function is implemented in the computer, which calculates the difference between the vehicle parameters acquired by the acquisition function and a reference parameter indicating a reference value for the vehicle parameters, thereby setting a correction parameter that indicates a correction value for the reference parameter, and which is determined according to the type of the vehicle identified by the vehicle type identification function. A computer program that, in the position conversion function, converts the image coordinate points to the vehicle coordinate points, using the vehicle coordinate points as the target variable and substituting the correction parameter set by the correction value setting function into a relational expression that includes the image coordinate points, the imaging parameter, the reference parameter, and the correction parameter as explanatory variables.

Images