system

Abstract

To provide a system for suitably executing the estimation of a vehicle position by using a captured image of a vehicle regardless of imaging conditions for capturing the image.SOLUTION: A system used for estimating a vehicle position of a vehicle traveling by remote control includes: an image acquisition section for acquiring a captured image obtained by imaging the vehicle; a specification section for specifying a camera which has captured the image; and a determination section for determining contents of estimation processing for estimating the vehicle position by using the captured image according to imaging conditions of the specified camera.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a technique for running a vehicle by remote control in a vehicle manufacturing process.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In remote control of a vehicle, a technique for estimating the vehicle position using a captured image of the vehicle can be used. A technique for appropriately performing such estimation of the vehicle position regardless of the imaging conditions of the captured image is desired.

Means for Solving the Problems

[0005] The present disclosure can be realized in the following forms. According to one embodiment of the present disclosure, a system is provided for estimating a vehicle position representing the position of a vehicle that is driven by remote control. The system comprises: an image acquisition unit that acquires a captured image of the vehicle; a camera identification unit that identifies the camera that took the captured image; a determination unit that determines the content of an estimation process for estimating the vehicle position using the captured image, according to the shooting conditions of the identified camera, which relate to whether the vehicle is photographed from the left or the right side by the camera; an estimation unit that executes the estimation process; and a command generation unit that generates a control command for remote control using the vehicle position estimated by the estimation process. The captured image includes at least one of the left portion of the vehicle and the right portion of the vehicle. The determination unit, according to the shooting conditions, alternatively determines as the content of the estimation process a first process that includes detecting the left portion of the captured image as a predetermined detection location, and a second process that includes detecting the right portion of the captured image as a detection location. The estimation unit estimates the vehicle position using the detected location in the estimation process.

[0006] (1) According to one aspect of the present disclosure, a system used for estimating a vehicle position representing the position of a vehicle traveling by remote control is provided. This system includes an image acquisition unit that acquires a captured image of the vehicle, a camera identification unit that identifies the camera that captured the captured image, and a determination unit that determines the content of an estimation process for estimating the vehicle position using the captured image according to the imaging conditions of the identified camera. With this configuration, the content of the estimation process can be determined according to the shooting conditions of the camera that captured the image. Therefore, the likelihood of being able to appropriately estimate the vehicle's position using the captured image increases, regardless of the shooting conditions under which the image was taken. (2) In the above configuration, the system may include an estimation unit that performs the estimation process and a command generation unit that generates a control command for remote control using the vehicle position estimated by the estimation process. In such a configuration, a control command for remote control can be generated using an appropriate vehicle position estimated by the estimation process. (3) In the above configuration, the captured image includes at least one of the left portion of the vehicle and the right portion of the vehicle, and the determination unit selectively determines, depending on the shooting conditions, the content of the estimation process to include a first process that detects the left portion of the captured image as a predetermined detection location, and a second process that detects the right portion of the captured image as a detection location, and the estimation unit may estimate the vehicle position using the detected location in the estimation process. In this configuration, the estimation process can detect the left portion or the right portion of the vehicle depending on the shooting conditions of the camera that captured the image, and can estimate the vehicle position using the detected left portion or the right portion. (4) In the above configuration, the left portion and the right portion are portions that are symmetrically located to each other in the vehicle width direction, and either the first process or the second process is a process that generates a mirror-reversed image of the captured image and detects the detected portion in the mirror-reversed image, and the other of the first process or the second process is a process that detects the detected portion in the captured image that has not been mirror-reversed. In this configuration, the left portion and the right portion can be detected in the same procedure as when the first process is executed and when the second process is executed in the estimation process, except that the captured image is mirror-reversed. (5) In the above configuration, the second determination unit may determine the content of the estimation process by determining the program to be used in the estimation process according to the shooting conditions. In this configuration, the content of the estimation process can be determined according to the shooting conditions of the camera that took the captured image by determining the program to be used in the estimation process by the determination unit.

[0007] In addition to the system form described above, this disclosure can also be implemented in other forms, such as a control device, a control method, a computer program for implementing the control method, and a non-temporary recording medium on which the computer program is stored. Furthermore, for example, the above system may estimate the vehicle position by inputting captured images into a pre-prepared machine learning model. [Brief explanation of the drawing]

[0008] [Figure 1] A conceptual diagram showing the system configuration. [Figure 2] A block diagram showing the configuration of the vehicle and the control system. [Figure 3] A flowchart of the decision-making process. [Figure 4] Flowchart of the estimation process. [Figure 5] An explanatory diagram illustrating an example of how images are analyzed during the estimation process. [Figure 6] An explanatory diagram showing an example of how a camera for photographing the vehicle is determined in another embodiment. [Modes for carrying out the invention]

[0009] A. First Embodiment: Figure 1 is a conceptual diagram showing the configuration of the system 10 in the first embodiment. Figure 2 is a block diagram showing the configuration of the vehicle 100 and the control device 200. The system 10 is configured as a remote control system that drives the vehicle 100 by remote control. The system 10 comprises one or more vehicles 100, a control device 200 that performs remote control of the vehicle 100, a plurality of cameras 300 that capture captured images Pi including the vehicle 100, and a process control device 400 that manages the manufacturing process of the vehicle 100.

[0010] In this embodiment, vehicle 100 is an electric vehicle (BEV: Battery Electric Vehicle). However, vehicle 100 is not limited to electric vehicles; for example, it may be a gasoline vehicle, a hybrid vehicle, or a fuel cell vehicle.

[0011] In this embodiment, remote control of the vehicle 100 is performed in a factory that manufactures the vehicle 100. As shown in Figure 1, the factory in this embodiment comprises a first location PL1 and a second location PL2. The first location PL1 is, for example, the location where the vehicle 100 is assembled, and the second location PL2 is, for example, the location where the vehicle 100 is inspected. Any location within the factory is represented by xyz coordinate values ​​in a reference coordinate system Σr. The reference coordinate system Σr is defined, for example, as the world coordinate system (also called the global coordinate system).

[0012] The first location PL1 and the second location PL2 are connected by a track SR on which the vehicle 100 can travel. In this embodiment, the track SR is configured as a track that branches into two at branching point Jc and then merges at merging point Cf. The track SR includes the first track SR1, the second track SR2, the third track SR3, and the fourth track SR4. The first track SR1 extends straight from the first location PL1 towards the second location PL2. The second track SR2 to the fourth track SR4 connect branching point Jc and merging point Cf via a different path than the first track SR1. The configuration of the track SR is not limited to the above. For example, the track SR may be a track that branches into three or more parts, or it may be a track that does not branch.

[0013] A target route is set on the track SR, which the vehicle 100 travels by remote control. In this embodiment, the target route on the track SR is set as a first route Rt1 that goes straight from the branching point Jc, and a second route Rt2 that turns at the branching point Jc. The first route Rt1 corresponds to the first track SR1. The second route Rt2 includes routes Rt2a, Rt2b, Rt2c, Rt2d, and Rt2e. Route Rt2a corresponds to the portion of the first track SR1 up to the branching point Jc. Route Rt2b corresponds to the second track SR2. Route Rt2c corresponds to the third track SR3. Route Rt2d corresponds to the fourth track SR4. Route Rt2e corresponds to the portion of the first track SR1 from the merging point Cf onward. The number, shape, and arrangement of the target routes are not limited to those described above and may be arbitrary.

[0014] The target route is determined, for example, according to the following conditions: the type of vehicle 100 (e.g., vehicle type, model, grade, and power source), the inspection results of vehicle 100, the degree of congestion on each route, and the degree of congestion in the next process. For example, the manufacturing information of vehicle 100, as described later, can be used to determine this target route.

[0015] A plurality of cameras 300 configured to be able to photograph the vehicle 100 on the runway SR are installed around the runway SR. Each camera 300 is arranged such that when the vehicle 100 is at any position on the target route, at least one camera 300 can always photograph the vehicle 100. The control device 200 can acquire, in real time, the relative position and orientation of the vehicle 100 with respect to the target route and the traveling direction of the vehicle 100 by using the photographed images Pi captured by each camera 300. The position, orientation, and traveling direction thus detected are used for generating control commands for remotely controlling the vehicle 100. In the present embodiment, each camera 300 is arranged to be able to photograph an image overlooking the runway SR from above. Also, the positions and orientations of the individual cameras 300 are fixed, and the relative relationship between the reference coordinate system Σr and the device coordinate system of each individual camera 300 (hereinafter also referred to as the camera coordinate system) is known. A coordinate transformation matrix for mutually converting the coordinate values of the reference coordinate system Σr and the coordinate values of the device coordinate system of each individual camera 300 is stored in advance in the control device 200. In the remote control of the vehicle 100, for example, various sensors such as an in-vehicle camera mounted on the vehicle 100, LiDAR (Light Detection And Ranging), millimeter-wave radar, ultrasonic sensor, and infrared sensor may be used subsidiarily.

[0016] The control device 200 in the present embodiment is configured as a remote control device that generates the above-described control commands and transmits them to the vehicle 100. More specifically, the control device 200 generates a control command for causing the vehicle 100 to travel along the target route and transmits the control command to the vehicle 100. The vehicle 100 travels in accordance with the received control command. Therefore, by the system 10, the vehicle 100 can be moved from the first location PL1 to the second location PL2 by remote control without using a conveying device such as a crane or a conveyor.

[0017] As shown in Figure 2, the vehicle 100 includes a vehicle control device 110 for controlling various parts of the vehicle 100, an actuator group 120 driven under the control of the vehicle control device 110, a communication device 130 for communicating with a control device 200 via wireless communication, and a GNSS (Global Navigation Satellite System) receiver 140 for acquiring position information of the vehicle 100. In this embodiment, the actuator group 120 includes actuators for the drive system for accelerating the vehicle 100, actuators for the steering system for changing the direction of travel of the vehicle 100, and actuators for the braking system for decelerating the vehicle 100. The drive system includes a battery, a drive motor driven by the battery's power, and drive wheels rotated by the drive motor. The actuators for the drive system include the drive motor. The actuator group 120 may also include actuators for swinging the wipers of the vehicle 100, actuators for opening and closing the power windows of the vehicle 100, and so on.

[0018] The vehicle control device 110 is comprised of a computer comprising a processor 111, memory 112, input / output interface 113, and internal bus 114. The processor 111, memory 112, and input / output interface 113 are connected via the internal bus 114 to enable bidirectional communication. The input / output interface 113 is connected to an actuator group 120, a communication device 130, and a GNSS receiver 140.

[0019] In this embodiment, the processor 111 functions as a vehicle control unit 115 and a position information acquisition unit 116 by executing a program PG1 prestored in the memory 112. The vehicle control unit 115 controls the actuator group 120. When a driver is aboard the vehicle 100, the vehicle control unit 115 can drive the vehicle 100 by controlling the actuator group 120 according to the driver's operation. Whether or not a driver is aboard the vehicle 100, the vehicle control unit 115 can also drive the vehicle 100 by controlling the actuator group 120 according to a control command transmitted from the control device 200. The position information acquisition unit 116 acquires position information indicating the current location of the vehicle 100 using the GNSS receiver 140. However, the position information acquisition unit 116 and the GNSS receiver 140 may be omitted.

[0020] The control device 200 is constituted by a computer including a processor 201, a memory 202, an input / output interface 203, and an internal bus 204. The processor 201, the memory 202, and the input / output interface 203 are connected via the internal bus 204 so as to be communicable bidirectionally. A communication device 205 for communicating with the vehicle 100, the camera 300, and the process management device 400 by wireless communication is connected to the input / output interface 203.

[0021] In this embodiment, the processor 201 functions as an image acquisition unit 210, a camera identification unit 215, a camera information acquisition unit 218, a determination unit 220, an estimation unit 250, and a command generation unit 260 by executing a program PG2 prestored in the memory 202. In addition to the program PG2, the memory 202 stores target route data RD representing a set target route, correspondence data 271 described later, and an estimation program 280. The estimation program 280 in this embodiment is an inclusive program including a first analysis program 281 and a second analysis program 282 described later.

[0022] The image acquisition unit 210 acquires an image Pi of the vehicle 100 captured by the camera 300. In this embodiment, the camera 300 captures the image Pi so as to show at least one of the first portion Dp1 and the second portion Dp2 shown in Figure 1. Therefore, the image Pi in this embodiment includes at least one of the first portion Dp1 and the second portion Dp2. The first portion Dp1 is the left portion of the vehicle 100 located to the left of the center position in the vehicle width direction, and the second portion Dp2 is the right portion of the vehicle 100 located to the right of the center position. More specifically, the first portion Dp1 is the left rear corner of the vehicle 100, and the second portion Dp2 is the right rear corner of the vehicle 100. In other words, in this embodiment, the first portion Dp1 and the second portion Dp2 are portions of the vehicle 100 that are located approximately symmetrically to each other in the vehicle width direction. The first part Dp1 and the second part Dp2 correspond to detection areas, which are parts that can be detected as detection points in the estimation process described later. The detection points are the parts used in the estimation process to estimate the vehicle position, which represents the position of vehicle 100.

[0023] The camera identification unit 215 identifies the imaging camera, which is the camera 300 that captured the captured image Pi. In this embodiment, when the captured image Pi is acquired by the image acquisition unit 210, the camera identification unit 215 identifies the camera 300 that transmitted the captured image Pi to the control device 200 as the imaging camera. In this embodiment, the position and orientation of each camera 300 are fixed with respect to the track SR. Therefore, identifying the imaging camera is equivalent to identifying the shooting conditions for the captured image Pi by the imaging camera. These shooting conditions are conditions related to the detection parts included in the captured image Pi, and include, for example, conditions related to parts of the vehicle 100 that may be included as detection parts in the captured image Pi, and conditions related to the orientation in which the captured image Pi is taken. The conditions regarding the orientation in which the captured image Pi is taken refer to the conditions regarding the orientation in which the vehicle 100 is photographed when the captured image Pi is taken. For example, these are conditions that determine whether the vehicle 100 is photographed from the right or left side, or from the front or rear side. Once the orientation in which the captured image Pi is taken is specified, the detection parts included in the captured image Pi can be identified. Hereafter, the conditions for photographing the captured image Pi by camera 300 will also simply be referred to as "the conditions for photographing camera 300".

[0024] The camera information acquisition unit 218 acquires camera information Ci related to the imaging camera. In this embodiment, the identification information of each camera 300 is used as the camera information Ci. In this embodiment, each camera 300 is configured to transmit its own identification information to the control device 200 along with the captured image Pi. The camera identification unit 215 described above may identify the imaging camera using the identification information transmitted from the camera 300 in this manner. In other embodiments, the camera information acquisition unit 218 may acquire the camera information Ci from a database pre-stored in the memory 202, or from an external computer or recording medium.

[0025] In this embodiment, the system 10 is configured such that the camera 300 that captures the captured image Pi is determined according to the route the vehicle 100 travels. Camera 300A, shown in Figure 1, captures the vehicle 100 traveling on the first route Rt1, route Rt2a, or route Rt2e from the left rear side. Camera 300B captures the vehicle 100 traveling on route Rt2b from the right rear side. Camera 300C captures the vehicle 100 traveling on route Rt2c from the left rear side. Camera 300D captures the vehicle 100 traveling on route Rt2d from the right rear side. In this embodiment, "capturing the vehicle 100 from the left rear side" means capturing the vehicle 100 from the left rear side such that the first portion Dp1 (left rear corner) is visible in the captured image Pi. "Photographing vehicle 100 from the right rear" means photographing vehicle 100 from the right rear so that the second portion Dp2 (right rear corner) is visible in the captured image Pi. Therefore, an image Pi taken of vehicle 100 from the left rear usually includes the first portion Dp1 regardless of the vehicle's position. On the other hand, this captured image Pi may or may not include the second portion Dp2 depending on the vehicle's position. Also, if an image Pi taken of vehicle 100 from the left rear includes both the first portion Dp1 and the second portion Dp2, the second portion Dp2 will appear smaller than the first portion Dp1 in that image Pi. Similarly, an image Pi taken of vehicle 100 from the right rear usually includes the second portion Dp2 regardless of the vehicle's position. On the other hand, this captured image Pi may or may not include the first part Dp1, depending on the position where the vehicle 100 is traveling.

[0026] The estimation unit 250 shown in Figure 2 performs estimation processing. Estimation processing is the process of estimating the vehicle position of the remotely controlled vehicle 100 using the captured image Pi. In this embodiment, estimation processing is the process of estimating the vehicle position using detected locations in the captured image Pi. More specifically, estimation processing includes analysis processing to detect detected locations in the captured image Pi by analyzing the captured image Pi, and processing to estimate the vehicle position using the detected locations. In this embodiment, estimation processing is performed by executing estimation program 280 by the estimation unit 250. The estimation unit 250 can estimate the position and orientation of the vehicle 100 using, for example, the vehicle 100's driving history or position information detected by the GNSS receiver 140. The direction of travel of the vehicle 100 can be estimated using the vehicle 100's orientation, driving history, control command history, etc.

[0027] In this embodiment, the estimation unit 250 includes an analysis unit 251 that performs analysis processing. In this embodiment, the analysis unit 251 performs analysis processing by executing a first analysis program 281 and a second analysis program 282, which are analysis programs. An analysis program is a program that enables a computer executing the analysis program to analyze the captured image Pi and detect detection locations in the captured image Pi. In this embodiment, the first analysis program 281 is an analysis program for analyzing a captured image Pi taken from the left rear side of the vehicle 100, and is used to detect a first part Dp1 as a detection location. The second analysis program 282 is an analysis program for analyzing a captured image Pi taken from the right rear side of the vehicle 100, and is used to detect a second part Dp2 as a detection location.

[0028] The determination unit 220 determines the content of the estimation process according to the shooting conditions of the identified imaging camera. In this embodiment, the determination unit 220 changes the content of the estimation process according to the shooting conditions of the imaging camera by determining the program used in the estimation process according to the camera information Ci of the imaging camera. More specifically, the determination unit 220 determines a detection program according to the camera information Ci of the imaging camera. The detection program is an analysis program used to detect detection locations in the analysis process. As described above, in this embodiment, identifying the imaging camera is equivalent to identifying the shooting conditions of the imaging camera. Therefore, by determining the content of the estimation process according to the imaging camera, the content of the estimation process can be determined according to the shooting conditions of the imaging camera.

[0029] In this specification, "determining a program according to shooting conditions" refers to determining the processing steps of a program according to the shooting conditions. Therefore, the meaning of "determining a program according to shooting conditions" includes, for example, selecting one analysis program from a plurality of pre-prepared programs as the program to be determined, according to the shooting conditions. In this case, the selection of the detection program may be achieved, for example, by executing conditional branching according to the shooting conditions in a comprehensive program that includes multiple programs. Furthermore, the meaning of "determining a program according to shooting conditions" also includes, for example, preparing the program to be determined by rewriting some or all of the code included in the program according to the shooting conditions.

[0030] In this embodiment, the determination unit 220 determines the content of the estimation process by referring to the corresponding data 271 using the camera information Ci of the imaging camera. The corresponding data 271 is data that records the correspondence between the camera information Ci and the information for determining the content of the estimation process. More specifically, the corresponding data 271 in this embodiment is data that records the correspondence between the identification information of each camera 300 and the information for determining the program used in the estimation process. In the corresponding data 271 in this embodiment, the identification information of cameras 300A and 300C is associated with parameters for selecting the first analysis program 281. In addition, the identification information of cameras 300B and 300D is associated with parameters for selecting the second analysis program 282. In other embodiments, the corresponding data 271 may be, for example, data that records the correspondence between the identification information of each camera 300, the shooting conditions of each camera 300, and the information for determining the content of the estimation process.

[0031] The command generation unit 260 generates a control command for remote control using the estimated position and orientation of the vehicle 100 and transmits it to the vehicle 100. This control command is a command to drive the vehicle 100 according to a target route represented by the target route data RD stored in the memory 202. The control command can be generated as a command including driving force or braking force and steering angle. Alternatively, the control command may be generated as a command including at least one of the position and orientation of the vehicle 100 and the future route.

[0032] The process control device 400, for example, is configured by a computer and manages the entire manufacturing process of the vehicle 100 in the factory. For example, when a vehicle 100 starts traveling along a target route, individual information such as an identification number and model number that identifies the vehicle 100 is transmitted from the process control device 400 to the control device 200. This individual information corresponds to manufacturing information used to manage the manufacturing process of the vehicle 100. The position of the vehicle 100 detected by the control device 200 is also transmitted to the process control device 400. The functions of the process control device 400 may also be implemented in the same device as the control device 200.

[0033] Figure 3 is a flowchart of the decision process in this embodiment. This decision process is the process for realizing the control method in this embodiment. The decision process is executed by the processor 201 of the control device 200, for example, at predetermined time intervals.

[0034] In S110 of Figure 3, the image acquisition unit 210 acquires the captured image Pi. In S120, the camera identification unit 215 identifies the camera 300 that captured the captured image Pi acquired in S110 as the imaging camera. In S130, the camera information acquisition unit 218 acquires the camera information Ci of the imaging camera identified in S120. In S140, the determination unit 220 determines the detection program according to the acquired camera information Ci. In this embodiment, in S140, the detection program is determined according to the camera information Ci, and the content of the estimation process is determined according to the shooting conditions of the imaging camera. More specifically, in S140, the content of the analysis process is determined.

[0035] Figure 1 shows vehicle 100 traveling along route Rt2a of the second route Rt2. When vehicle 100 is traveling along route Rt2a, the captured image Pi is captured by camera 300A. In this case, at S110 in Figure 3, the captured image Pi captured by camera 300A is acquired. At S120, camera 300A, which captured this captured image Pi, is identified as the imaging camera. At S130, camera information Ci of camera 300A, which has been identified as the imaging camera, is acquired. At S140, the corresponding data 271 is referenced according to the acquired camera information Ci, and the first analysis program 281 is determined as the detection program according to camera 300A. The process is substantially the same when vehicle 100 is traveling along other routes of the second route Rt2 or the first route Rt1.

[0036] Figure 4 is a flowchart of the estimation process in this embodiment. In this embodiment, the estimation process is executed each time the decision process is completed.

[0037] In S210 of Figure 4, the estimation unit 250 determines, based on the result of S140 in Figure 3, whether or not to use the first analysis program 281 for the analysis of the captured image Pi acquired in S110. If it is determined in S210 to use the first analysis program 281, the analysis unit 251 executes the first process implemented by the first analysis program 281 from S220 to S230. The first process is for detecting the first part Dp1 as a detection location. If it is not determined in S210 to use the first analysis program 281, the analysis unit 251 executes the second process implemented by the second analysis program 282 from S215 to S230. The second process is for detecting the second part Dp2 as a detection location. In the following, S220 implemented by the first analysis program 281 will also be referred to as S220A, and S220 implemented by the second analysis program 282 will also be referred to as S220B. The same applies to S225 and S230.

[0038] Unlike the first process, the second process in this embodiment includes the inversion process in S215. The inversion process generates an inverted image by mirror-reversing the captured image Pi. As a result, the captured image Pi, which is not mirror-reversed, is analyzed in S220A to S230A, while the inverted image is analyzed in S220B to S230B. This process of mirror-reversing an image is also called "flipping". The first and second processes are substantially the same except that the second process includes the inversion process. Therefore, in S220B to S230B, it is possible to analyze the inverted image using the same algorithm as in S220A to S230A, that is, using the same procedure. More specifically, when the first part Dp1 is used as the detection location, in S220A to S230A, the first part Dp1 is detected as the left rear corner of the un-inverted captured image Pi. On the other hand, when the second portion Dp2 is used as the detection location, the second portion Dp2, which is actually the right rear corner, is detected as the left rear corner in the inverted image from S220B to S230B. In other words, in this embodiment, in the inverted image, the second portion Dp2 is detected as the portion corresponding to the first portion Dp1 in the captured image Pi that has not been mirror-reversed.

[0039] Figure 5 is an explanatory diagram illustrating an example of how image Im1 is analyzed in the estimation process. Image Im1 shown in Figure 5 is an image of a vehicle 100 traveling on a track SR. Image Im1 has mutually orthogonal Xc and Yc axes as coordinate axes and is represented in a camera coordinate system with the focal point of camera 300 as the origin. Image Im1 may be an inverted image or an uninverted captured image Pi. Various corrections and preprocessing may be performed on image Im1, such as distortion correction processing to correct distortion, rotation processing to rotate image Im1 so that the direction of the vehicle 100's movement vector V in image Im1 points in a predetermined direction, and cropping processing to remove unnecessary areas that do not contain the vehicle 100 from image Im1. For example, the optical flow method is used to detect the magnitude and direction of the movement vector V.

[0040] In steps S220 to S230 of Figure 4, the analysis unit 251 performs analysis processing. In the analysis processing in this embodiment, the analysis unit 251 detects the positioning point PP by calculating the coordinates of the positioning point PP shown in Figure 5 using the image Im1. The coordinates of the positioning point PP are calculated as the coordinates of the vicinity of the detection location in the image coordinate system described later. The positioning point PP detected in this way represents the detection location in the captured image Pi.

[0041] In S220, the analysis unit 251 performs a masking process as a detection process to detect the vehicle 100 in image Im1. The masking process detects the vehicle 100 in image Im1 and masks the target region containing the vehicle 100 in image Im1, thereby generating image Im2 which includes the masked region Ms. In the masking process, the estimation unit 250 generates image Im2 by inputting image Im1 to a machine learning model (not shown) that has been trained to mask the vehicle 100 contained in the input image. As this machine learning model, for example, a deep neural network (hereinafter also called DNN) having the structure of a convolutional neural network (hereinafter also called CNN) that realizes semantic segmentation or instance segmentation is used. Note that the machine learning model may be trained using algorithms other than neural networks, for example.

[0042] In S225, the analysis unit 251 performs a perspective transformation process. The perspective transformation process generates image Im3 by performing a perspective transformation on image Im2. In the perspective transformation process, the analysis unit 251 uses, for example, the position information of the camera 300 and perspective transformation parameters related to internal parameters to perform a perspective transformation on image Im2 into a bird's-eye view image viewed from above the vehicle 100, which is approximately perpendicular to the road surface of the track SR (for example, directly above the vehicle 100). Image Im3 is represented in an image coordinate system. The image coordinate system is a coordinate system that has a point on the image plane projected by the perspective transformation as its origin, and has mutually orthogonal Xi and Yi axes as its coordinate axes. Image Im3 includes a mask region Msb that corresponds to the mask region Ms transformed by the perspective transformation.

[0043] In S230, the analysis unit 251 performs a positioning point calculation process to calculate the coordinates of the positioning point PP. In the positioning point calculation process in this embodiment, the analysis unit 251 calculates the coordinates of the positioning point PP using the coordinates (Xi1, Yi1) of the first coordinate point P1 and the coordinates (Xi2, Yi2) of the second coordinate point P2.

[0044] The first coordinate point P1 is identified as the vertex of quadrilateral R1b in image Im2 that corresponds to the base coordinate point P0 in image Im1. Quadrilateral R1b corresponds to the first bounding rectangle R1, which is deformed by perspective transformation. The first bounding rectangle R1 is a rectangle in image Im2 that is bounding to the mask region Ms and has a longer side parallel to the movement vector V. The base coordinate point P0 is the vertex of this first bounding rectangle R1 that corresponds to the detection location of vehicle 100. More specifically, the base coordinate point P0 is the vertex of the first bounding rectangle R1 that is located to the left and behind the center of gravity C of vehicle 100 when the direction of the movement vector V is forward. The second coordinate point P2 is the vertex of the second bounding rectangle R2 in image Im2 that corresponds to the detection location of vehicle 100, among the four vertices of the second bounding rectangle R2. More specifically, the second coordinate point P2 represents the coordinate of a vertex of the second circumscribed rectangle R2 that is located to the left and behind the centroid C when the direction of the movement vector V is considered to be forward. The second circumscribed rectangle R2 is a rectangle that is circumscribed around the mask region Msb and has sides parallel to the Xi axis and sides parallel to the Yi axis. Thus, the first coordinate point P1 and the second coordinate point P2 are both coordinate points determined according to the detection location, and therefore have a correlation with each other.

[0045] If the coordinate value Xi1 is greater than the coordinate value Xi2, the estimation unit 250 determines the Xi coordinate value of the positioning point PP to be the coordinate value Xi1. Conversely, if the coordinate value Xi1 is less than the coordinate value Xi2, the estimation unit 250 determines the Xi coordinate value to be the coordinate value Xi2. Similarly, if the coordinate value Yi1 is greater than the coordinate value Yi2, the Yi coordinate value of the positioning point PP is determined to be the coordinate value Yi1. If the coordinate value Yi1 is less than the coordinate value Yi2, the Yi coordinate value is determined to be the coordinate value Yi2. By determining the Xi and Yi coordinate values ​​in this way, the positioning point PP is detected.

[0046] In S235, the estimation unit 250 executes a vehicle coordinate calculation process. The vehicle coordinate calculation process calculates vehicle coordinate points using positioning point PP. Vehicle coordinate points are coordinate points that represent positioning point PP in the reference coordinate system Σr. In S235, the estimation unit 250 calculates vehicle coordinate points using, for example, a predetermined estimation formula and the coordinate values ​​of positioning point PP calculated in S230. The command generation unit 260 generates a control command using the vehicle coordinate points calculated in the estimation process as the vehicle position. The control command thus generated is transmitted to the vehicle 100. In other words, in this embodiment, calculating vehicle coordinate points is equivalent to estimating the vehicle position. Note that the method for estimating the vehicle position using the captured image Pi is not limited to the above. For example, the vehicle 100 in the captured image Pi may be detected by a process different from mask processing, the detection location in the captured image Pi may be detected by a process different from perspective transformation processing or positioning point calculation processing, or coordinate points representing the vehicle position may be calculated by a process different from vehicle coordinate calculation processing.

[0047] As described above, the system 10 in this embodiment allows the content of the estimation process to be determined according to the shooting conditions of the imaging camera. Therefore, even if the shooting conditions differ for each camera 300, for example, the likelihood of appropriately estimating the vehicle position increases regardless of the shooting conditions. Furthermore, since control commands are generated using the vehicle position estimated by the estimation process, the likelihood of generating appropriate control commands increases. Therefore, the likelihood of appropriately remotely controlling the vehicle 100 increases.

[0048] Furthermore, in this embodiment, the determination unit 220 selectively determines, depending on the shooting conditions of the imaging camera, whether the estimation process includes a first process or a second process. The first process is the process of detecting the first portion Dp1 (left portion) of the captured image Pi. The second process is the process of detecting the second portion Dp2 (right portion) of the captured image Pi. In this way, in the estimation process, the left portion and the right portion of the captured image Pi can be detected depending on the shooting conditions of the imaging camera, and the vehicle position can be estimated using the detected left portion and the right portion. The left portion and the right portion can be easily captured by the camera 300 by installing the camera 300 on the left side and the right side of the track SR. In particular, in this embodiment, the first portion Dp1 is the left rear corner, and the second portion Dp2 is the right rear corner. Therefore, by using cameras 300 installed on the left and right sides of the track SR to photograph the vehicle 100 from behind, it is possible to easily capture the captured image Pi, including the first part Dp1 and the second part Dp2.

[0049] Furthermore, in this embodiment, the first process is to detect the first portion Dp1 in the captured image Pi which has not been mirror-reversed. The second process is to generate a mirror-reversed image of the captured image Pi and to detect the second portion Dp2 in the mirror-reversed image which is symmetrically positioned with respect to the first portion Dp1 in the vehicle width direction. Therefore, whether the first process or the second process is executed in the estimation process, the first portion Dp1 and the second portion Dp2 can be detected using the same procedure, except for mirror-reversing the captured image Pi. In other embodiments, for example, the first process may include a reversal process, and the second process may not include a reversal process.

[0050] Furthermore, in this embodiment, the determination unit 220 determines the program to be used in the estimation process according to the identified imaging camera. Therefore, by determining the program using the determination unit 220, the content of the estimation process can be determined according to the shooting conditions of the imaging camera.

[0051] B. Other embodiments: (B1) In the above embodiment, the system 10 is configured as a remote control system comprising a vehicle 100, a camera 300, an estimation unit 250, and a command generation unit 260, but it does not have to be configured in this way. For example, the system 10 may be configured as a system that determines the content of the estimation process and transmits the determination result to another system comprising a vehicle 100, a camera 300, an estimation unit 250, and a command generation unit 260. In this case, the system 10 does not have to comprise some or all of the vehicle 100, camera 300, estimation unit 250, and command generation unit 260. Also in this case, the system 10 may be configured to transmit the captured image Pi along with the determination result to the other system. In this case, the estimation unit 250 provided in the other system may acquire the captured image Pi transmitted from the system 10 during the estimation process.

[0052] (B2) In the above embodiment, the camera 300 that photographs the vehicle 100 is determined in accordance with the route the vehicle 100 travels. Other conditions may be applied in place of, or in addition to, the above-mentioned route-related conditions for determining the camera 300 that photographs the vehicle 100.

[0053] For example, Figure 6 is an explanatory diagram showing an example in another embodiment where the camera 300 that photographs the vehicle 100 is determined. Figure 6 shows the vehicle 100 traveling along the third route Rt3 on the track SRb. In the example of Figure 6, the camera 300 that photographs the vehicle 100 traveling along the third route Rt3 is determined by the timing of the vehicle 100's travel. In this way, for example, when the vehicle 100 is traveling outdoors by remote control, it is possible to suppress the capture of images Pi that are unsuitable for estimation processing due to the influence of sunlight. More specifically, in the example of Figure 6, during time period TP1, camera 300E photographs the vehicle 100, and during time periods other than time period TP1, camera 300F photographs the vehicle 100. Camera 300E photographs the vehicle 100 from the left rear side. Camera 300F photographs the vehicle 100 from the right rear side. In the example shown in Figure 6, for example, in the corresponding data 271, the camera information Ci of camera 300E is associated with parameters for selecting the first analysis program 281, and the camera information Ci of camera 300F is associated with parameters for selecting the second analysis program 282. Even in this configuration, the content of the estimation process can be determined according to the shooting conditions of the imaging camera, similar to the first embodiment. By configuring the control device 200 and system 10 to determine the content of the estimation process according to the shooting conditions of the imaging camera in this way, the possibility of appropriately determining the content of the estimation process increases, regardless of the method by which the camera 300 that captures the captured image Pi is determined. Therefore, the control device 200 and system 10 can be easily applied to various remote control systems.

[0054] (B3) In the above embodiment, the first and second processes include inversion processing, but inversion processing may be omitted. In this case, the estimation unit 250 may detect either the first part Dp1 or the second part Dp2 as the detection location in the captured image Pi which has not been mirror-reversed.

[0055] (B4) In the above embodiment, the first part Dp1 and the second part Dp2 do not have to be symmetrically positioned in the vehicle width direction. For example, the left rear corner and the right front corner of the vehicle 100 may be used as the first part Dp1 and the second part Dp2, respectively. Also, the left side and right side of the vehicle 100 do not have to be used as detection points. Furthermore, the number of parts used as detection points may be three or more, or it may be just one.

[0056] (B5) In the above embodiment, the determination unit 220 determines the program used in the estimation process according to the shooting conditions of the imaging camera, but this is not required. For example, the same program may be used in the estimation process even if the shooting conditions are different. In this case, the determination unit 220 may determine the content of the estimation process according to the shooting conditions of the imaging camera by determining the parameters used in the estimation process according to the shooting conditions of the imaging camera.

[0057] (B6) In the above embodiment, the camera information Ci may be information representing the shooting conditions of the camera 300, for example, information relating to a detection area that may be included in the captured image Pi taken by the camera 300 may be used. In this case, the camera information Ci may be information representing the orientation in which the camera 300 photographs the vehicle 100, or information representing a detection area that the camera 300 can photograph. By using such camera information Ci, for example, even if the position and orientation of each camera 300 is not fixed with respect to the track SR, the content of the estimation process can be determined according to the shooting conditions of the imaging camera. In other words, in this case, the camera 300 may be configured so that its position and orientation with respect to the track SR can be changed by, for example, the control device 200 or the process control device 400. In this way, the shooting conditions of the camera 300 can be changed according to, for example, the timing of the vehicle 100's movement or the position of the vehicle 100.

[0058] (B7) In the above embodiment, the vehicle 100 only needs to have a configuration that allows it to be moved by remote control, and may be in the form of a platform having the configuration described below. Specifically, the vehicle 100 only needs to have at least a vehicle control unit 115 and a communication device 130 in order to perform the three functions of "driving," "turning," and "stopping" by remote control. That is, the vehicle 100 that can be moved by remote control does not need to have at least some of the interior parts such as the driver's seat and dashboard attached, at least some of the exterior parts such as the bumper and fenders attached, and does not need to have a body shell attached. In this case, the remaining parts such as the body shell may be attached to the vehicle 100 before it is shipped from the factory, or the remaining parts such as the body shell may be attached to the vehicle 100 after it has been shipped from the factory without the remaining parts such as the body shell attached to it. The position of the platform can also be determined in the same way as the vehicle 100 in each embodiment.

[0059] This disclosure is not limited to the embodiments described above, and can be implemented in various configurations without departing from its spirit. For example, the technical features in the embodiments corresponding to the technical features in each form described in the summary of the invention can be replaced or combined as appropriate in order to solve some or all of the above-described problems, or to achieve some or all of the above-described effects. Furthermore, if a technical feature is not described as essential in this specification, it can be deleted as appropriate. [Explanation of Symbols]

[0060] 10...System, 100...Vehicle, 110...Vehicle control device, 111...Processor, 112...Memory, 113...Input / Output Interface, 114...Internal Bus, 115...Vehicle Control Unit, 116...Location Information Acquisition Unit, 120...Actuator Group, 130...Communication Device, 140...GNSS Receiver, 200...Control Device, 201...Processor, 202...Memory, 203...Input / Output Interface, 204...Internal Bus, 205...Communication device, 210...Image acquisition unit, 215...Camera identification unit, 218...Camera information acquisition unit, 220...Decision unit, 250...Estimation unit, 251...Analysis unit, 260...Command generation unit, 271...Corresponding data, 280...Estimation program, 281...First analysis program, 282...Second analysis program, 300, 300A, 300B, 300C, 300D, 300E, 300F...Camera, 400...Process control device

Claims

1. A system used to estimate the vehicle position representing the position of a vehicle that is driven by remote control, The image acquisition unit acquires the captured image of the vehicle, A camera identification unit that identifies the camera that took the aforementioned captured image, A determination unit determines the content of an estimation process that estimates the vehicle's position using the captured image, according to the shooting conditions of the specified camera, which relate to whether the vehicle is photographed from the left or right side by the camera. An estimation unit that performs the estimation process, The system includes a command generation unit that generates a control command for remote control using the vehicle position estimated by the estimation process, The aforementioned captured image includes at least one of the left portion of the vehicle and the right portion of the vehicle. The determination unit, depending on the shooting conditions, selectively determines the content of the estimation process to be either a first process that includes detecting the left portion of the captured image as a predetermined detection location, or a second process that includes detecting the right portion of the captured image as the detection location. The estimation unit is a system that estimates the vehicle position using the detected location in the estimation process.

2. The system according to claim 1, The left portion and the right portion are, respectively, symmetrically positioned relative to each other in the vehicle width direction. A system in which either the first or second process generates a mirror-reversed image of the captured image and detects the detected location in the mirror-reversed image, and the other of the first or second process detects the detected location in the captured image that has not been mirror-reversed.

3. A system according to either claim 1 or 2, The system determines the content of the estimation process by determining the program to be used in the estimation process according to the shooting conditions.

Images