system

Abstract

To provide a system for suitably executing the estimation of a vehicle position by using a captured image of a vehicle regardless of which camera is in charge of the imaging.SOLUTION: A system used for estimating a vehicle position of a vehicle traveling by remote control includes: a plurality of cameras provided in order to image the vehicle on a track; a route information acquisition section for acquiring route information on a route of the vehicle traveling on the track by the remote control; a first determination section for determining a camera-in-charge which takes charge of capturing an object image including the vehicle traveling on the route among the respective cameras according to the acquired route information; and a second determination section for determining contents of estimation processing for estimating the vehicle position by using the object image according to the determined camera-in-charge.SELECTED DRAWING: Figure 2

Description

Technical Field

[0006] , ,

[0005] , ,

[0001] This disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a technique for driving a vehicle by remote control in the manufacturing process of a vehicle.

Prior Art Document

Patent Document

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the remote control of a vehicle, a technique for estimating the vehicle position using a captured image of the vehicle can be utilized. When a plurality of cameras can capture the vehicle, a technique for appropriately performing such estimation of the vehicle position regardless of which camera is responsible for capturing the vehicle is desired.

Means for Solving the Problems

[0005] This disclosure can be realized in the following forms. <0000​​(1) According to one embodiment of the present disclosure, a system is provided for estimating a vehicle position representing the position of a vehicle that is driven by remote control. The system comprises a plurality of cameras arranged to photograph the vehicle on a road, a route information acquisition unit that acquires route information relating to the route on which the vehicle travels by remote control, a first determination unit that determines which of the cameras is responsible for photographing a target image including the vehicle traveling along the route, according to the acquired route information, and a second determination unit that determines the content of an estimation process for estimating the vehicle position using the target image according to the determined responsible camera. In this configuration, the camera responsible for capturing target images can be determined according to the route the vehicle is traveling, and the content of the estimation process can be determined according to the assigned camera. Therefore, regardless of which camera is assigned, the likelihood of appropriately estimating the vehicle's position using the captured images increases. (2) In the above configuration, the system may include an image acquisition unit that acquires the target image from the assigned camera, an estimation unit that performs the estimation process, and a command generation unit that generates a control command for remote control using the vehicle position estimated by the estimation process. In such a configuration, a control command for remote control can be generated using an appropriate vehicle position estimated by the estimation process. (3) In the above configuration, the target image includes at least one of the left portion of the vehicle and the right portion of the vehicle, and the second determination unit selectively determines, depending on the assigned camera, the content of the estimation process to include a first process that detects the left portion of the target image as a predetermined detection location, and a second process that detects the right portion of the target image as a detection location, and the estimation unit may estimate the vehicle position using the detected location in the estimation process. In this configuration, the estimation process can detect the left portion or the right portion of the vehicle depending on the assigned camera, and the vehicle position can be estimated using the detected left portion or the right portion. (4) In the above configuration, the left portion and the right portion are portions that are symmetrically located to each other in the vehicle width direction, and either the first process or the second process is a process that generates a mirror-reversed image of the target image and detects the detection location in the mirror-reversed image, while the other of the first process or the second process is a process that detects the detection location in the target image that has not been mirror-reversed. In this configuration, the left portion and the right portion can be detected in the same procedure as when the first process is executed and when the second process is executed in the estimation process, except that the target image is mirror-reversed. (5) In the above configuration, the second decision unit may determine the content of the estimation process by determining the program to be used in the estimation process according to the assigned camera. In this configuration, the content of the estimation process can be determined according to the assigned camera by determining the program to be used in the estimation process by the second decision unit.

[0007] In addition to the system form described above, this disclosure can also be implemented in other forms, such as a control device, a control method, a computer program for implementing the control method, and a non-temporary recording medium on which the computer program is stored. Furthermore, for example, the above system may estimate the vehicle position by inputting a target image into a pre-prepared machine learning model. [Brief explanation of the drawing]

[0008] [Figure 1] This is a conceptual diagram showing the system configuration. [Figure 2] This is a block diagram showing the configuration of the vehicle and the control device. [Figure 3] This is a flowchart of the decision-making process. [Figure 4] This is a diagram illustrating the decision-making process. [Figure 5] This is a flowchart of the estimation process. [Figure 6] This is an explanatory diagram illustrating an example of how images are analyzed during the estimation process. [Modes for carrying out the invention]

[0009] A. First Embodiment: Figure 1 is a conceptual diagram showing the configuration of the system 10 in the first embodiment. Figure 2 is a block diagram showing the configuration of the vehicle 100 and the control device 200. The system 10 is configured as a remote control system that drives the vehicle 100 by remote control. The system 10 comprises one or more vehicles 100, a control device 200 that performs remote control of the vehicle 100, a plurality of cameras 300 that capture captured images Pi including the vehicle 100, and a process control device 400 that manages the manufacturing process of the vehicle 100.

[0010] In this embodiment, vehicle 100 is an electric vehicle (BEV: Battery Electric Vehicle). However, vehicle 100 is not limited to electric vehicles; for example, it may be a gasoline vehicle, a hybrid vehicle, or a fuel cell vehicle.

[0011] In this embodiment, remote control of the vehicle 100 is performed in a factory that manufactures the vehicle 100. As shown in Figure 1, the factory in this embodiment comprises a first location PL1 and a second location PL2. The first location PL1 is, for example, the location where the vehicle 100 is assembled, and the second location PL2 is, for example, the location where the vehicle 100 is inspected. Any location within the factory is represented by xyz coordinate values ​​in a reference coordinate system Σr. The reference coordinate system Σr is defined, for example, as the world coordinate system (also called the global coordinate system).

[0012] The first location PL1 and the second location PL2 are connected by a track SR on which the vehicle 100 can travel. In this embodiment, the track SR is configured as a track that branches into two at branching point Jc and then merges at merging point Cf. The track SR includes the first track SR1, the second track SR2, the third track SR3, and the fourth track SR4. The first track SR1 extends straight from the first location PL1 towards the second location PL2. The second track SR2 to the fourth track SR4 connect branching point Jc and merging point Cf via a different route than the first track SR1.

[0013] A target route is set on the track SR, which the vehicle 100 will travel on by remote control. In this embodiment, a first route Rt1 that goes straight at the branching point Jc and a second route Rt2 that turns at the branching point Jc are set as target routes on the track SR. The target route is determined according to the following conditions, for example. These conditions include the type of vehicle 100 (e.g., vehicle type, model, grade, and power source), the inspection results of the vehicle 100, the degree of congestion on each track, and the degree of congestion in the next process. For example, the manufacturing information of the vehicle 100, which will be described later, can be used to determine such target routes. The number, shape, and arrangement of target routes are not limited to those described above and may be arbitrary.

[0014] Multiple cameras 300, configured to capture images of a vehicle 100 on the track SR, are installed around the track SR according to the track SR. The multiple cameras 300 are arranged so that at least one camera 300 can always capture images of the vehicle 100 when the vehicle 100 is at any position on the target route. The control device 200 can use the captured images Pi taken by each camera 300 to obtain in real time the relative position and orientation of the vehicle 100 with respect to the target route, as well as the direction of travel of the vehicle 100. The position, orientation, and direction of travel thus detected are used to generate control commands for remotely controlling the vehicle 100. In this embodiment, each camera 300 is arranged to capture an image of the track SR from above. The position and orientation of each camera 300 are fixed, and the relative relationship between the reference coordinate system Σr and the device coordinate system of each camera 300 (hereinafter also referred to as the camera coordinate system) is known. A coordinate transformation matrix for converting between the coordinate values ​​of the reference coordinate system Σr and the coordinate values ​​of the device coordinate system of each camera 300 is pre-stored in the control device 200. In the remote control of the vehicle 100, various sensors such as on-board cameras mounted on the vehicle 100, LiDAR (Light Detection and Ranging), millimeter-wave radar, ultrasonic sensors, and infrared sensors may be used as auxiliary devices.

[0015] In this embodiment, the control device 200 is configured as a remote control device that generates the control commands described above and transmits them to the vehicle 100. More specifically, the control device 200 generates control commands to cause the vehicle 100 to travel along a target route and transmits the control commands to the vehicle 100. The vehicle 100 travels according to the received control commands. Therefore, the system 10 can remotely move the vehicle 100 from the first location PL1 to the second location PL2 without using transport devices such as cranes or conveyors.

[0016] As shown in FIG. 2, the vehicle 100 includes a vehicle control device 110 for controlling each part of the vehicle 100, an actuator group 120 that is driven under the control of the vehicle control device 110, a communication device 130 for communicating with the control device 200 by wireless communication, and a GNSS (Global Navigation Satellite System) receiver 140 for acquiring the position information of the vehicle 100. In the present embodiment, the actuator group 120 includes an actuator of a driving device for accelerating the vehicle 100, an actuator of a steering device for changing the traveling direction of the vehicle 100, and an actuator of a braking device for decelerating the vehicle 100. The driving device includes a battery, a traveling motor driven by the power of the battery, and driving wheels rotated by the traveling motor. The actuator of the driving device includes the traveling motor. The actuator group 120 may include an actuator for swinging the wiper of the vehicle 100, an actuator for opening and closing the power window of the vehicle 100, and the like.

[0017] The vehicle control device 110 is constituted by a computer including a processor 111, a memory 112, an input / output interface 113, and an internal bus 114. The processor 111, the memory 112, and the input / output interface 113 are connected so as to be communicable bidirectionally via the internal bus 114. The actuator group 120, the communication device 130, and the GNSS receiver 140 are connected to the input / output interface 113.

[0018] In this embodiment, the processor 111 functions as the vehicle control unit 115 and the position information acquisition unit 116 by executing the program PG1 prestored in the memory 112. The vehicle control unit 115 controls the actuator group 120. When a driver is on board the vehicle 100, the vehicle control unit 115 can drive the vehicle 100 by controlling the actuator group 120 according to the driver's operation. Regardless of whether a driver is on board the vehicle 100 or not, the vehicle control unit 115 can also drive the vehicle 100 by controlling the actuator group 120 according to the control command transmitted from the control device 200. The position information acquisition unit 116 acquires position information indicating the current location of the vehicle 100 by using the GNSS receiver 140. However, the position information acquisition unit 116 and the GNSS receiver 140 can be omitted.

[0019] The control device 200 is composed of a computer including a processor 201, a memory 202, an input / output interface 203, and an internal bus 204. The processor 201, the memory 202, and the input / output interface 203 are connected via the internal bus 204 so as to be communicable bidirectionally. A communication device 205 for communicating with the vehicle 100, the camera 300, and the process management device 400 by wireless communication is connected to the input / output interface 203.

[0020] In this embodiment, the processor 201 functions as the route information acquisition unit 208, the image acquisition unit 210, the first determination unit 215, the camera information acquisition unit 220, the second determination unit 230, the estimation unit 250, and the command generation unit 260 by executing the program PG2 prestored in the memory 202. In addition to the program PG2, the memory 202 stores target route data RD representing the set target route, first correspondence data 271, second correspondence data 272, and an estimation program 280, which will be described later.

[0021] The route information acquisition unit 208 acquires route information for the vehicle 100. The route information is information about the route that the vehicle 100 will travel on the track SR via remote control. More specifically, the route information is information about the target route that the vehicle 100 will travel in the future. In this embodiment, the route information acquisition unit 208 acquires route information by referring to target route data RD using current location information that represents the current location of the vehicle 100. The control device 200 and the process control device 400 acquire this current location information using, for example, a GNSS receiver 140, area sensors (not shown) installed around the track SR, and various sensors used for remote control (for example, a camera 300).

[0022] The first determination unit 215 determines the assigned camera according to the route information acquired by the route information acquisition unit 208. The assigned camera is the camera 300 among the cameras 300 that is responsible for capturing the captured image Pi which includes the vehicle 100 traveling along the target route. Hereinafter, the captured image Pi which includes the vehicle 100 traveling along the target route will also be referred to as the target image. In this embodiment, the first determination unit 215 determines the assigned camera by referring to the first correspondence data 271 according to the route information. The first correspondence data 271 is, for example, data that records the correspondence between each route traveled by the vehicle 100 and each camera 300, or data that records the correspondence between each section of the track SR and each camera 300.

[0023] The image acquisition unit 210 acquires the target image from the assigned camera. In this embodiment, the camera 300 captures the captured image Pi so as to show at least one of the first part Dp1 and the second part Dp2 shown in Figure 1. Therefore, the target image in this embodiment includes at least one of the first part Dp1 and the second part Dp2. The first part Dp1 is the left portion of the vehicle 100 located to the left of the center position in the vehicle width direction, and the second part Dp2 is the right portion of the vehicle 100 located to the right of the center position. More specifically, the first part Dp1 is the left rear corner of the vehicle 100, and the second part Dp2 is the right rear corner of the vehicle 100. In other words, in this embodiment, the first part Dp1 and the second part Dp2 are portions of the vehicle 100 that are located approximately symmetrically to each other in the vehicle width direction. The first part Dp1 and the second part Dp2 correspond to detection areas, which are portions that can be detected as detection points in the estimation process described later. The detection location is the location used in the estimation process to estimate the vehicle position, which represents the position of vehicle 100.

[0024] The camera information acquisition unit 220 shown in Figure 2 acquires camera information Ci related to the assigned camera. In this embodiment, the identification information of each camera 300 is used as the camera information Ci. In this embodiment, the identification information of the camera 300 is transmitted from the camera 300 to the control device 200 along with the captured image Pi. In other embodiments, the camera information acquisition unit 220 may acquire the camera information Ci from a database pre-stored in the memory 202, or from an external computer or recording medium.

[0025] In this embodiment, the position and orientation of each camera 300 are fixed with respect to the track SR. Therefore, the identification information of the assigned camera is information that can identify the shooting conditions under which the assigned camera captures the target image. These shooting conditions are conditions relating to detection parts that may be included in the captured image Pi, and include, for example, conditions relating to parts of the vehicle 100 that may be included as detection parts in the captured image Pi, and conditions relating to the orientation in which the captured image Pi is taken. The conditions relating to the orientation in which the captured image Pi is taken relate to the orientation in which the vehicle 100 is photographed when the captured image Pi is taken, and include, for example, conditions that determine whether the vehicle 100 is photographed from the right or left side of the vehicle 100, or conditions that determine whether the vehicle 100 is photographed from the front or rear side of the vehicle 100. In this embodiment, once the orientation in which the captured image Pi is taken is identified, the detection parts that may be included in the captured image Pi are identified. The camera information Ci that can identify these shooting conditions is not limited to identification information of camera 300, but may also include information indicating the direction in which camera 300 is photographing vehicle 100, or information indicating the detection area that camera 300 can photograph.

[0026] The estimation unit 250 performs estimation processing. Estimation processing is the process of estimating the vehicle position of the remotely controlled vehicle 100 using the target image. In this embodiment, estimation processing is the process of estimating the vehicle position using detected locations in the target image. More specifically, estimation processing includes analysis processing to detect detected locations in the target image by analyzing the target image, and processing to estimate the vehicle position using the detected locations. In this embodiment, estimation processing is performed by the estimation unit 250 executing the estimation program 280. The estimation unit 250 can estimate the position and orientation of the vehicle 100 using, for example, the vehicle 100's driving history or position information detected by the GNSS receiver 140. The direction of travel of the vehicle 100 can be estimated using the vehicle 100's orientation, driving history, control command history, etc.

[0027] In this embodiment, the estimation unit 250 includes an analysis unit 251 that performs analysis processing. In this embodiment, the analysis unit 251 performs analysis processing by executing a first analysis program 281 and a second analysis program 282, which are analysis programs, respectively. An analysis program is a program that enables a computer executing the analysis program to analyze a target image and detect detection locations in the target image.

[0028] The second determination unit 230 determines the content of the estimation process according to the assigned camera. In this embodiment, the second determination unit 230 changes the content of the estimation process according to the assigned camera by determining the program used in the estimation process according to the camera information Ci. More specifically, the second determination unit 230 determines a detection program according to the camera information Ci. The detection program is an analysis program used to detect detection locations in the analysis process.

[0029] In this specification, "determining the program according to the assigned camera" refers to determining the processing steps of the program according to the assigned camera. Therefore, the meaning of "determining the program according to the assigned camera" includes, for example, selecting one analysis program from a plurality of pre-prepared programs as the program to be determined, according to the assigned camera. In this case, the selection of the detection program may be achieved, for example, by executing conditional branching according to the assigned camera in a comprehensive program that includes multiple programs. Furthermore, the meaning of "determining the program according to the assigned camera" also includes, for example, preparing the program to be determined by rewriting part or all of the code included in the program according to the assigned camera.

[0030] In this embodiment, the second determination unit 230 determines the content of the estimation process by referring to the second correspondence data 272 using the camera information Ci. The second correspondence data 272 is data that records the correspondence between the camera information Ci and the information for determining the content of the estimation process. More specifically, in this embodiment, the second correspondence data 272 is data that records the correspondence between the identification information of each camera 300 and the information for determining the program used in the estimation process. In other embodiments, the second correspondence data 272 may be, for example, data that records the correspondence between the identification information of each camera 300, the orientation in which each camera 300 photographs the vehicle 100 in order to capture a target image, and the information for determining the content of the estimation process.

[0031] The estimation program 280 described above is a comprehensive program that includes a first analysis program 281 and a second analysis program 282 as analysis programs. Details of the first analysis program 281 and the second analysis program 282 will be described later.

[0032] The command generation unit 260 generates a control command using the estimated position and orientation of the vehicle 100 and transmits it to the vehicle 100. This control command is a command to drive the vehicle 100 according to a target route represented by the target route data RD stored in the memory 202. The control command can be generated as a command including driving force or braking force and steering angle. Alternatively, the control command may be generated as a command including at least one of the position and orientation of the vehicle 100 and the route that the vehicle 100 will travel in the future.

[0033] The process control device 400, for example, is configured by a computer and manages the entire manufacturing process of the vehicle 100 in the factory. For example, when a vehicle 100 starts traveling along a target route, individual information such as an identification number and model number that identifies the vehicle 100 is transmitted from the process control device 400 to the control device 200. This individual information corresponds to manufacturing information used to manage the manufacturing process of the vehicle 100. The position of the vehicle 100 detected by the control device 200 is also transmitted to the process control device 400. The functions of the process control device 400 may also be implemented in the same device as the control device 200.

[0034] Figure 3 is a flowchart of the decision process in this embodiment. This decision process is a process for realizing the control method in this embodiment. The decision process is executed by the processor 201 of the control device 200, for example, at predetermined time intervals.

[0035] In S110 of Figure 3, the route information acquisition unit 208 acquires route information for the remotely controlled vehicle 100. In S120, the first determination unit 215 determines the assigned camera according to the acquired route information. In S130, the camera information acquisition unit 220 acquires camera information Ci of the determined assigned camera. In S140, the second determination unit 230 determines the detection program according to the acquired camera information Ci. In this embodiment, in S140, the detection program is determined according to the camera information Ci, and the content of the estimation process is determined according to the assigned camera. More specifically, in S140, the content of the analysis process is determined.

[0036] Figure 4 is a diagram illustrating the decision process. Figure 4 shows an example of a vehicle 100 traveling along the track SR according to a target route via remote control. Figure 4 shows the first route Rt1 and the second route Rt2 described above. The first route Rt1 corresponds to the first track SR1. The second route Rt2 includes routes Rt2a, Rt2b, Rt2c, Rt2d, and Rt2e. Route Rt2a corresponds to the portion of the first track SR1 up to branching point Jc. Route Rt2b corresponds to the second track SR2. Route Rt2c corresponds to the third track SR3. Route Rt2d corresponds to the fourth track SR4. Route Rt2e corresponds to the portion of the first track SR1 from the merging point Cf onward.

[0037] In the first correspondence data 271 of this embodiment, the first route Rt1, route Rt2a, and route Rt2e are associated with camera 300A, route Rt2b is associated with camera 300B, route Rt2c is associated with camera 300C, and route Rt2d is associated with camera 300D. In other words, in the first correspondence data 271, each route included in the target route is associated with each camera on a one-to-one basis.

[0038] Camera 300A is positioned to photograph the vehicle 100 traveling on the first route Rt1, route Rt2a, and route Rt2e from the left rear. Camera 300B is positioned to photograph the vehicle 100 traveling on route Rt2b from the right rear. Camera 300C is positioned to photograph the vehicle 100 traveling on route Rt2c from the left rear. Camera 300D is positioned to photograph the vehicle 100 traveling on route Rt2d from the right rear. In this embodiment, "photographing the vehicle 100 from the left rear" means photographing the vehicle 100 from the left rear so that the first portion Dp1 (left rear corner) is visible in the captured image Pi. "Photographing the vehicle 100 from the right rear" means photographing the vehicle 100 from the right rear so that the second portion Dp2 (right rear corner) is visible in the captured image Pi. Therefore, an image Pi taken of vehicle 100 from the left rear side usually includes the first part Dp1 regardless of the position of vehicle 100. On the other hand, this image Pi may or may not include the second part Dp2 depending on the position of vehicle 100. Also, when an image Pi taken of vehicle 100 from the left rear side includes both the first part Dp1 and the second part Dp2, the second part Dp2 appears smaller than the first part Dp1 in that image Pi. Similarly, an image Pi taken of vehicle 100 from the right rear side usually includes the second part Dp2 regardless of the position of vehicle 100. On the other hand, this image Pi may or may not include the first part Dp1 depending on the position of vehicle 100.

[0039] In the second corresponding data 272 of this embodiment, the camera information Ci of camera 300A and the camera information Ci of camera 300C are associated with parameters for selecting the first analysis program 281. In addition, the camera information Ci of cameras 300B and 300D are associated with parameters for selecting the second analysis program 282. The first analysis program 281 in this embodiment is an analysis program for analyzing a target image taken from the left rear side of the vehicle 100, and is used to detect a first portion Dp1 as a detection location. The second analysis program 282 is an analysis program for analyzing a target image taken from the right rear side of the vehicle 100, and is used to detect a second portion Dp2 as a detection location.

[0040] In Figure 4, vehicle 100 is traveling on route Rt2a in the second route Rt2. Therefore, in the example in Figure 4, route information representing route Rt2a is acquired in S110 of Figure 3. Next, in S120, the first corresponding data 271 is referenced according to the acquired route information, and camera 300A is determined as the camera in charge of route Rt2a. In S130, camera information Ci related to the determined camera 300A is acquired. Then, in S140, the second corresponding data 272 is referenced according to the acquired camera information Ci, and the first analysis program 281 is determined as the detection program according to camera 300A. The process is substantially the same when vehicle 100 travels on other routes in the second route Rt2 or on the first route Rt1. In Figure 4, the movements of vehicle 100 traveling on each route from route Rt2b to route Rt2e are shown by dashed lines.

[0041] In addition, in S110 of Figure 3, for example, if vehicle 100 is traveling near the end of a certain route, route information representing the next route after that route may be acquired as the target route to travel. More specifically, for example, in Figure 4, if vehicle 100 is traveling near route Rt2b within route Rt2a, route information representing route Rt2b may be acquired as the target route to travel.

[0042] Figure 5 is a flowchart of the estimation process in this embodiment. In this embodiment, the estimation process is performed each time a target image is transmitted from the assigned camera to the control device 200.

[0043] In step S205 of Figure 5, the image acquisition unit 210 acquires the target image transmitted from the assigned camera. In step S210, the estimation unit 250 determines whether or not to use the first analysis program 281 for analyzing the target image, based on the result of step S140 of Figure 3. If it is determined in step S210 to use the first analysis program 281, the analysis unit 251 executes the first process implemented by the first analysis program 281 from steps S220 to S230. The first process is for detecting the first part Dp1 as a detection location. If it is not determined in step S210 to use the first analysis program 281, the analysis unit 251 executes the second process implemented by the second analysis program 282 from steps S215 to S230. The second process is for detecting the second part Dp2 as a detection location. In the following, S220 realized by the first analysis program 281 will also be referred to as S220A, and S220 realized by the second analysis program 282 will also be referred to as S220B. The same applies to S225 and S230.

[0044] Unlike the first process, the second process in this embodiment includes the inversion process in S215. The inversion process generates an inverted image by mirror-reversing the target image. As a result, the unmirrored target image is analyzed in S220A to S230A, while the inverted image is analyzed in S220B to S230B. This process of mirror-reversing an image is also called "flipping". The first and second processes are substantially the same except that the second process includes the inversion process. Therefore, in S220B to S230B, it is possible to analyze the inverted image using the same algorithm as in S220A to S230A, that is, using the same procedure. More specifically, when the first part Dp1 is used as the detection location, in S220A to S230A, the first part Dp1 is detected as the left rear corner of the unmirrored target image. On the other hand, when the second portion Dp2 is used as the detection location, the second portion Dp2, which is actually the right rear corner, is detected as the left rear corner in the inverted image from S220B to S230B. In other words, in this embodiment, in the inverted image, the second portion Dp2 is detected as the portion corresponding to the first portion Dp1 in the target image that has not been mirror-reversed.

[0045] Figure 6 is an explanatory diagram illustrating an example of how image Im1 is analyzed in the estimation process. Image Im1 shown in Figure 6 is an image of vehicle 100 traveling on track SR. Image Im1 has mutually orthogonal Xc and Yc axes as coordinate axes and is represented in a camera coordinate system with the focal point of camera 300 as the origin. Image Im1 may be an inverted image or an uninverted target image. Various corrections and preprocessing may be performed on image Im1 as appropriate, such as distortion correction processing to correct distortion, rotation processing to rotate image Im1 so that the direction of the vehicle 100's movement vector V in image Im1 points in a predetermined direction, and cropping processing to remove unnecessary areas that do not contain vehicle 100 from image Im1. For example, the optical flow method is used to detect the magnitude and direction of the movement vector V.

[0046] In steps S220 to S230 of Figure 5, the analysis unit 251 performs analysis processing. In the analysis processing in this embodiment, the analysis unit 251 detects the positioning point PP shown in Figure 6 by calculating the coordinates of the positioning point PP using the image Im1. The coordinates of the positioning point PP are calculated as the coordinates of the vicinity of the detected location in the image coordinate system described later. The positioning point PP detected in this way represents the detected location in the target image.

[0047] In S220, the analysis unit 251 performs a masking process as a detection process to detect the vehicle 100 in image Im1. The masking process detects the vehicle 100 in image Im1 and masks the target region containing the vehicle 100 in image Im1, thereby generating image Im2 which includes the masked region Ms. In the masking process, the estimation unit 250 generates image Im2 by inputting image Im1 to a machine learning model (not shown) that has been trained to mask the vehicle 100 contained in the input image. As this machine learning model, for example, a deep neural network (hereinafter also called DNN) having the structure of a convolutional neural network (hereinafter also called CNN) that realizes semantic segmentation or instance segmentation is used. Note that the machine learning model may be trained using algorithms other than neural networks, for example.

[0048] In S225, the analysis unit 251 performs a perspective transformation process. The perspective transformation process generates image Im3 by performing a perspective transformation on image Im2. In the perspective transformation process, the analysis unit 251 uses, for example, the position information of the camera 300 and perspective transformation parameters related to internal parameters to perform a perspective transformation on image Im2 into a bird's-eye view image viewed from above the vehicle 100, which is approximately perpendicular to the road surface of the track SR (for example, directly above the vehicle 100). Image Im3 is represented in an image coordinate system. The image coordinate system is a coordinate system that has a point on the image plane projected by the perspective transformation as its origin, and has mutually orthogonal Xi and Yi axes as its coordinate axes. Image Im3 includes a mask region Msb that corresponds to the mask region Ms transformed by the perspective transformation.

[0049] In S230, the analysis unit 251 performs a positioning point calculation process to calculate the coordinates of the positioning point PP. In the positioning point calculation process in this embodiment, the analysis unit 251 calculates the coordinates of the positioning point PP using the coordinates (Xi1, Yi1) of the first coordinate point P1 and the coordinates (Xi2, Yi2) of the second coordinate point P2.

[0050] The first coordinate point P1 is identified as the vertex of quadrilateral R1b in image Im2 that corresponds to the base coordinate point P0 in image Im1. Quadrilateral R1b corresponds to the first bounding rectangle R1, which is deformed by perspective transformation. The first bounding rectangle R1 is a rectangle in image Im2 that is bounding to the mask region Ms and has a longer side parallel to the movement vector V. The base coordinate point P0 is the vertex of this first bounding rectangle R1 that corresponds to the detection location of vehicle 100. More specifically, the base coordinate point P0 is the vertex of the first bounding rectangle R1 that is located to the left and behind the center of gravity C of vehicle 100 when the direction of the movement vector V is forward. The second coordinate point P2 is the vertex of the second bounding rectangle R2 in image Im2 that corresponds to the detection location of vehicle 100, among the four vertices of the second bounding rectangle R2. More specifically, the second coordinate point P2 represents the coordinate of a vertex of the second circumscribed rectangle R2 that is located to the left and behind the centroid C when the direction of the movement vector V is considered to be forward. The second circumscribed rectangle R2 is a rectangle that is circumscribed around the mask region Msb and has sides parallel to the Xi axis and sides parallel to the Yi axis. Thus, the first coordinate point P1 and the second coordinate point P2 are both coordinate points determined according to the detection location, and therefore have a correlation with each other.

[0051] If the coordinate value Xi1 is greater than the coordinate value Xi2, the estimation unit 250 determines the Xi coordinate value of the positioning point PP to be the coordinate value Xi1. Conversely, if the coordinate value Xi1 is less than the coordinate value Xi2, the estimation unit 250 determines the Xi coordinate value to be the coordinate value Xi2. Similarly, if the coordinate value Yi1 is greater than the coordinate value Yi2, the Yi coordinate value of the positioning point PP is determined to be the coordinate value Yi1. If the coordinate value Yi1 is less than the coordinate value Yi2, the Yi coordinate value is determined to be the coordinate value Yi2. By determining the Xi and Yi coordinate values ​​in this way, the positioning point PP is detected.

[0052] In S235, the estimation unit 250 performs a vehicle coordinate calculation process. The vehicle coordinate calculation process calculates vehicle coordinate points using positioning point PP. Vehicle coordinate points are coordinate points that represent positioning point PP in the reference coordinate system Σr. In S235, the estimation unit 250 calculates vehicle coordinate points using, for example, a predetermined estimation formula and the coordinate values ​​of positioning point PP calculated in S230. The command generation unit 260 generates a control command using the vehicle coordinate points calculated in the estimation process as the vehicle position. The control command thus generated is transmitted to the vehicle 100. In other words, in this embodiment, calculating vehicle coordinate points is equivalent to estimating the vehicle position. Note that the method for estimating the vehicle position using the target image is not limited to the above. For example, the vehicle 100 in the target image may be detected by a process different from masking, the detection location in the target image may be detected by a process different from perspective transformation processing or positioning point calculation processing, or coordinate points representing the vehicle position may be calculated by a process different from the vehicle coordinate calculation process.

[0053] As described above, the system 10 in this embodiment can determine the assigned camera according to the route information of the vehicle 100 to be remotely controlled, and can determine the content of the estimation process according to the assigned camera. Therefore, regardless of which camera 300 is determined to be the assigned camera, the likelihood of appropriately estimating the vehicle's position increases. Furthermore, in this embodiment, a control command is generated using the appropriate vehicle position estimated by the above estimation process. Therefore, the likelihood of generating an appropriate control command increases, and the likelihood of appropriately remotely controlling the vehicle 100 increases.

[0054] Furthermore, in this embodiment, the second decision unit 230 selectively decides, depending on the assigned camera, to execute either the first process or the second process in the estimation process. The first process is the process of detecting the first portion Dp1 (left portion) of the target image. The second process is the process of detecting the second portion Dp2 (right portion) of the target image. In this way, in the estimation process, the left portion or the right portion of the target image can be detected depending on the assigned camera, and the vehicle position can be estimated using the detected left portion or right portion. These left portion and right portion can be easily photographed by the camera 300 by installing the camera 300 on the left or right side of the track SR. In particular, in this embodiment, the first portion Dp1 is the left rear corner, and the second portion Dp2 is the right rear corner. Therefore, by photographing the vehicle 100 from behind with the camera 300 installed on the left or right side of the track SR, the target image including the first portion Dp1 and the second portion Dp2 can be easily photographed.

[0055] Furthermore, in this embodiment, the first process is the process of detecting the first portion Dp1 in the target image that has not been mirror-reversed. The second process is the process of generating a mirror-reversed image of the target image and detecting the second portion Dp2 in the mirror-reversed image that is symmetrically positioned with respect to the first portion Dp1 in the vehicle width direction. Therefore, in the estimation process, the first portion Dp1 and the second portion Dp2 can be detected using the same procedure as when the first process is executed and when the second process is executed, except that the target image is mirror-reversed. In other embodiments, for example, the first process may include a reversal process, and the second process may not include a reversal process.

[0056] Furthermore, in this embodiment, the second determination unit 230 determines the program to be used in the estimation process according to the assigned camera. Therefore, by determining the program using the second determination unit 230, the content of the estimation process can be determined according to the assigned camera.

[0057] B. Other embodiments: (B1) In the above embodiment, system 10 is configured as a remote control system comprising an image acquisition unit 210, an estimation unit 250, and a command generation unit 260, but it does not have to be configured in this way. For example, system 10 may be configured as a system that determines the content of the estimation process and transmits the determination result to another system comprising an image acquisition unit 210, an estimation unit 250, and a command generation unit 260. In this case, system 10 does not have to comprise some or all of the image acquisition unit 210, the estimation unit 250, and the command generation unit 260.

[0058] (B2) In the above embodiment, the first part Dp1 and the second part Dp2 do not have to be symmetrically positioned in the vehicle width direction. For example, the left rear corner and the right front corner of the vehicle 100 may be used as the first part Dp1 and the second part Dp2, respectively. Also, the left side and right side of the vehicle 100 do not have to be used as detection points. Furthermore, the number of parts used as detection points may be three or more, or it may be just one.

[0059] (B3) In the above embodiment, the first and second processes include inversion processing, but inversion processing may not be included. In this case, the estimation unit 250 may detect the first part Dp1 or the second part Dp2 in the target image that has not been mirror-reversed, regardless of whether it detects the first part Dp1 or the second part Dp2 as the detection location.

[0060] (B4) In the above embodiment, the second determination unit 230 determines the program used in the estimation process according to the assigned camera, but this is not required. For example, even if the cameras 300 determined as assigned cameras are all different, the same program may be used in the estimation process. In this case, the second determination unit 230 may determine the content of the estimation process according to the assigned camera by determining the parameters used in the estimation process according to the assigned camera.

[0061] (B5) In the above embodiment, the track SR is configured as a track that branches into two and then merges, but it is not limited to this and may be configured as any type of track. For example, the track SR may be a track that branches into three or more, or it may be a track that does not branch.

[0062] (B6) In the above embodiment, the vehicle 100 only needs to have a configuration that allows it to be moved by remote control, and may be in the form of a platform having the configuration described below. Specifically, the vehicle 100 only needs to have at least a vehicle control unit 115 and a communication device 130 in order to perform the three functions of "driving," "turning," and "stopping" by remote control. That is, the vehicle 100 that can be moved by remote control does not need to have at least some of the interior parts such as the driver's seat and dashboard attached, at least some of the exterior parts such as the bumper and fenders attached, and does not need to have a body shell attached. In this case, the remaining parts such as the body shell may be attached to the vehicle 100 before it is shipped from the factory, or the remaining parts such as the body shell may be attached to the vehicle 100 after it has been shipped from the factory without the remaining parts such as the body shell attached to it. The position of the platform can also be determined in the same way as the vehicle 100 in each embodiment.

[0063] This disclosure is not limited to the embodiments described above, and can be implemented in various configurations without departing from its spirit. For example, the technical features in the embodiments corresponding to the technical features in each form described in the summary of the invention can be replaced or combined as appropriate in order to solve some or all of the above-described problems, or to achieve some or all of the above-described effects. Furthermore, if a technical feature is not described as essential in this specification, it can be deleted as appropriate. [Explanation of Symbols]

[0064] 10...System, 100...Vehicle, 110...Vehicle control device, 111...Processor, 112...Memory, 113...Input / Output Interface, 114...Internal Bus, 115...Vehicle Control Unit, 116...Location Information Acquisition Unit, 120...Actuator Group, 130...Communication Device, 140...GNSS Receiver, 200...Control Device, 201...Processor, 202...Memory, 203...Input / Output Interface, 204...Internal Bus, 205...Communication Device, 208...Route information acquisition unit, 210...Image acquisition unit, 215...First determination unit, 220...Camera information acquisition unit, 230...Second determination unit, 250...Estimation unit, 251...Analysis unit, 260...Command generation unit, 271...First corresponding data, 272...Second corresponding data, 280...Estimation program, 281...First analysis program, 282...Second analysis program, 300, 300A, 300B, 300C, 300D...Camera, 400...Process control device

Claims

1. A system used to estimate the vehicle position representing the position of a vehicle that is driven by remote control, Multiple cameras positioned to photograph the vehicle on the track, A route information acquisition unit that acquires route information relating to the route the vehicle travels on the road via the aforementioned remote control, A first determination unit determines, based on the acquired route information, which camera is responsible for capturing target images including the vehicle traveling along the route. A second determination unit determines the content of the estimation process for estimating the vehicle position using the target image, depending on the assigned camera that is determined, An image acquisition unit that acquires the target image from the camera in charge, An estimation unit that performs the estimation process, The system includes a command generation unit that generates a control command for remote control using the vehicle position estimated by the estimation process, The aforementioned target image includes at least one of the left portion of the vehicle and the right portion of the vehicle. The second determination unit, depending on the assigned camera, selectively determines the content of the estimation process to include a first process that detects the left portion of the target image as a predetermined detection location, and a second process that detects the right portion of the target image as a detection location. The estimation unit is a system that estimates the vehicle position using the detected location in the estimation process.

2. The system according to claim 1, The left portion and the right portion are, respectively, symmetrically positioned relative to each other in the vehicle width direction. A system in which either the first or second process generates a mirror-reversed image of the target image and detects the detection location in the mirror-reversed image, and the other of the first or second process detects the detection location in the target image that has not been mirror-reversed.

3. The system according to claim 1 or 2, The second determination unit determines the content of the estimation process by determining the program to be used in the estimation process according to the assigned camera.

Images