System

Abstract

To provide a system for suitably executing the estimation of a vehicle position by using a captured image of a vehicle regardless of which camera is in charge of the imaging.SOLUTION: A system used for estimating a vehicle position of a vehicle traveling by remote control includes: a plurality of cameras provided in order to image the vehicle on a track; a route information acquisition section for acquiring route information on a route of the vehicle traveling on the track by the remote control; a first determination section for determining a camera-in-charge which takes charge of capturing an object image including the vehicle traveling on the route among the respective cameras according to the acquired route information; and a second determination section for determining contents of estimation processing for estimating the vehicle position by using the object image according to the determined camera-in-charge.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to a system. [Background technology]

[0002] Patent Document 1 discloses a technique for running a vehicle by remote control during the vehicle manufacturing process. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2017-538619 Summary of the Invention [Problem to be solved by the invention]

[0004] In remote control of a vehicle, a technique for estimating the vehicle position using images captured by the vehicle can be used. When multiple cameras can capture images of the vehicle, a technique for appropriately estimating the vehicle position is desired regardless of which camera is responsible for capturing the image of the vehicle. [Means for solving the problem]

[0005] The present disclosure can be realized in the following forms.

[0006] (1) According to one aspect of the present disclosure, there is provided a system used for estimating a vehicle position representing a position of a vehicle traveling by remote control, the system including: a plurality of cameras arranged to capture an image of the vehicle on a road; a route information acquisition unit that acquires route information related to a route along which the vehicle will travel on the road by remote control; a first determination unit that determines, among the cameras, a camera in charge of capturing an image of a target including the vehicle traveling on the route according to the acquired route information; and a second determination unit that determines, according to the determined camera in charge, the content of an estimation process for estimating the vehicle position using the image of the target. In this manner, the camera responsible for capturing the target image can be determined according to the route the vehicle is traveling, and the content of the estimation process can be determined according to the camera responsible. Therefore, regardless of which camera is determined as the responsible camera, the possibility of appropriately estimating the vehicle position using the captured image is increased. (2) In the above embodiment, the vehicle control system may include an image acquisition unit that acquires the target image from the assigned camera, an estimation unit that executes the estimation process, and a command generation unit that generates a control command for the remote control using the vehicle position estimated by the estimation process. With this embodiment, the control command for the remote control can be generated using an appropriate vehicle position estimated by the estimation process. (3) In the above embodiment, the target image includes at least one of a left side portion of the vehicle and a right side portion of the vehicle, and the second determination unit determines, according to the assigned camera, the content of the estimation process alternatively between a first process that detects the left side portion in the target image as a predetermined detection location and a second process that detects the right side portion in the target image as the detection location, and the estimation unit may estimate the vehicle position using the detection location detected in the estimation process. With this embodiment, the estimation process can detect the left side or right side of the vehicle according to the assigned camera, and can estimate the vehicle position using the detected left side or right side. (4) In the above embodiment, the left side portion and the right side portion are portions located symmetrically to each other in the vehicle width direction of the vehicle, and one of the first process and the second process may be a process of generating an inverted image by mirror-inverting the target image and detecting the detection location in the inverted image, and the other of the first process and the second process may be a process of detecting the detection location in the target image that is not mirror-inverted. In this embodiment, the left side portion and the right side portion can be detected in the same procedure when the first process is performed and when the second process is performed in the estimation process, except for mirror-inverting the target image. (5) In the above embodiment, the second determination unit may determine the content of the estimation process by determining a program to be used in the estimation process according to the responsible camera. In this embodiment, the content of the estimation process can be determined according to the responsible camera by determining the program to be used in the estimation process by the second determination unit.

[0007] In addition to the above-described system form, the present disclosure can be realized in the form of, for example, a control device, a control method, a computer program for implementing the control method, a non-transitory recording medium on which the computer program is recorded, etc. Also, for example, the above-described system may estimate the vehicle position by inputting a target image into a machine learning model prepared in advance. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a conceptual diagram showing a system configuration. [Diagram 2] 2 is a block diagram showing the configuration of a vehicle and a control device. FIG. [Diagram 3] 13 is a flowchart of a determination process. [Figure 4] FIG. 11 is a diagram illustrating a determination process. [Diagram 5] 13 is a flowchart of an estimation process. [Figure 6] FIG. 11 is an explanatory diagram showing an example of how an image is analyzed in the estimation process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] A. First embodiment: Fig. 1 is a conceptual diagram showing the configuration of a system 10 in the first embodiment. Fig. 2 is a block diagram showing the configuration of a vehicle 100 and a control device 200. The system 10 is configured as a remote control system that causes the vehicle 100 to run by remote control. The system 10 includes one or more vehicles 100, a control device 200 that executes remote control of the vehicle 100, a plurality of cameras 300 that capture images Pi including the vehicle 100, and a process management device 400 that manages the manufacturing process of the vehicle 100.

[0010] The vehicle 100 in this embodiment is a battery electric vehicle (BEV). Note that the vehicle 100 is not limited to an electric vehicle, and may be, for example, a gasoline vehicle, a hybrid vehicle, or a fuel cell vehicle.

[0011] In this embodiment, remote control of the vehicle 100 is performed in a factory where the vehicle 100 is manufactured. As shown in FIG. 1, the factory in this embodiment includes a first location PL1 and a second location PL2. The first location PL1 is, for example, a location where the vehicle 100 is assembled, and the second location PL2 is, for example, a location where the vehicle 100 is inspected. Any position in the factory is expressed by xyz coordinate values ​​of a reference coordinate system Σr. The reference coordinate system Σr is defined, for example, as a world coordinate system (also called a global coordinate system).

[0012] The first location PL1 and the second location PL2 are connected by a track SR on which the vehicle 100 can travel. In this embodiment, the track SR is configured as a track that branches into two at a branching point Jc and then merges at a merging point Cf. The track SR includes a first track SR1, a second track SR2, a third track SR3, and a fourth track SR4. The first track SR1 extends straight from the first location PL1 to the second location PL2. The second track SR2 to the fourth track SR4 connect the branching point Jc and the merging point Cf via a route different from the first track SR1.

[0013] A target route along which the vehicle 100 travels is set on the road SR by remote control. In this embodiment, a first route Rt1 that goes straight through the branch point Jc and a second route Rt2 that turns at the branch point Jc are set on the road SR as target routes. The target route is determined, for example, according to the following conditions. That is, the conditions are the type of the vehicle 100 (for example, the model, type, grade, and power source), the inspection results of the vehicle 100, the degree of congestion of each road, the degree of congestion of the next process, and the like. For example, manufacturing information of the vehicle 100 described later can be used to determine such a target route. Note that the number, shape, arrangement, and the like of the target routes are not limited to those described above and may be arbitrary.

[0014] A plurality of cameras 300 configured to be able to photograph the vehicle 100 on the road SR are installed around the road SR along the road SR. The plurality of cameras 300 are arranged so that when the vehicle 100 is at any position on the target route, at least one camera 300 can always photograph the vehicle 100. The control device 200 can obtain the relative position and orientation of the vehicle 100 with respect to the target route and the traveling direction of the vehicle 100 in real time using the photographed images Pi photographed by each camera 300. The position, orientation, and traveling direction thus detected are used to generate a control command for remotely controlling the vehicle 100. In this embodiment, each camera 300 is arranged so that it can photograph an image overlooking the road SR from above. In addition, the position and orientation of each camera 300 are fixed, and the relative relationship between the reference coordinate system Σr and the device coordinate system (hereinafter also referred to as the camera coordinate system) of each camera 300 is known. A coordinate transformation matrix for mutually transforming coordinate values ​​of the reference coordinate system Σr and coordinate values ​​of the device coordinate system of each camera 300 is stored in advance in the control device 200. In remote control of the vehicle 100, for example, various sensors such as an on-board camera mounted on the vehicle 100, a LiDAR (Light Detection And Ranging), a millimeter wave radar, an ultrasonic sensor, and an infrared sensor may be used as auxiliary sensors.

[0015] The control device 200 in this embodiment is configured as a remote control device that generates the above-mentioned control command and transmits it to the vehicle 100. More specifically, the control device 200 generates a control command for causing the vehicle 100 to travel along a target route, and transmits the control command to the vehicle 100. The vehicle 100 travels according to the received control command. Therefore, the system 10 can move the vehicle 100 from the first location PL1 to the second location PL2 by remote control without using a transport device such as a crane or a conveyor.

[0016] As shown in FIG. 2, the vehicle 100 includes a vehicle control device 110 for controlling each part of the vehicle 100, an actuator group 120 that operates under the control of the vehicle control device 110, a communication device 130 for communicating with the control device 200 by wireless communication, and a GNSS (Global Navigation Satellite System) receiver 140 for acquiring position information of the vehicle 100. In this embodiment, the actuator group 120 includes an actuator of a drive device for accelerating the vehicle 100, an actuator of a steering device for changing the traveling direction of the vehicle 100, and an actuator of a braking device for decelerating the vehicle 100. The drive device includes a battery, a driving motor driven by the power of the battery, and a driving wheel rotated by the driving motor. The actuator of the drive device includes the driving motor. The actuator group 120 may include an actuator for swinging the wipers of the vehicle 100, an actuator for opening and closing the power windows of the vehicle 100, and the like.

[0017] The vehicle control device 110 is configured by a computer including a processor 111, a memory 112, an input / output interface 113, and an internal bus 114. The processor 111, the memory 112, and the input / output interface 113 are connected to each other via the internal bus 114 so as to be able to communicate bidirectionally. The actuator group 120, the communication device 130, and the GNSS receiver 140 are connected to the input / output interface 113.

[0018] In this embodiment, the processor 111 functions as the vehicle control unit 115 and the position information acquisition unit 116 by executing a program PG1 stored in advance in the memory 112. The vehicle control unit 115 controls the actuator group 120. When a driver is on board the vehicle 100, the vehicle control unit 115 can drive the vehicle 100 by controlling the actuator group 120 in response to an operation by the driver. The vehicle control unit 115 can also drive the vehicle 100 by controlling the actuator group 120 in response to a control command transmitted from the control device 200, regardless of whether a driver is on board the vehicle 100. The position information acquisition unit 116 acquires position information indicating the current location of the vehicle 100 using the GNSS receiver 140. However, the position information acquisition unit 116 and the GNSS receiver 140 can be omitted.

[0019] The control device 200 is configured by a computer including a processor 201, a memory 202, an input / output interface 203, and an internal bus 204. The processor 201, the memory 202, and the input / output interface 203 are connected via the internal bus 204 to enable bidirectional communication. A communication device 205 is connected to the input / output interface 203 for communicating with the vehicle 100, the camera 300, and the process management device 400 by wireless communication.

[0020] In this embodiment, the processor 201 executes a program PG2 pre-stored in the memory 202 to function as a route information acquisition unit 208, an image acquisition unit 210, a first determination unit 215, a camera information acquisition unit 220, a second determination unit 230, an estimation unit 250, and a command generation unit 260. In addition to the program PG2, the memory 202 stores target route data RD representing a set target route, first correspondence data 271, second correspondence data 272, and an estimation program 280, which will be described later.

[0021] The route information acquisition unit 208 acquires route information of the vehicle 100. The route information is information about the route along which the vehicle 100 travels on the road SR by remote control. More specifically, the route information is information about a target route to be traveled by the vehicle 100 in the future. In this embodiment, the route information acquisition unit 208 acquires route information by using current position information indicating the current location of the vehicle 100 and referring to the target route data RD. The control device 200 and the process management device 400 acquire such current position information by using, for example, the GNSS receiver 140, an area sensor (not shown) provided around the road SR, or various sensors (for example, the camera 300) used for remote control.

[0022] The first determination unit 215 determines the responsible camera according to the route information acquired by the route information acquisition unit 208. The responsible camera is the camera 300 that is responsible for capturing the captured image Pi including the vehicle 100 traveling on the target route among the cameras 300. Hereinafter, the captured image Pi including the vehicle 100 traveling on the target route is also referred to as a target image. In this embodiment, the first determination unit 215 determines the responsible camera by referring to the first correspondence data 271 according to the route information. The first correspondence data 271 is, for example, data recording the correspondence between each route on which the vehicle 100 travels and each camera 300, or data recording the correspondence between each section on the road SR and each camera 300.

[0023] The image acquisition unit 210 acquires a target image from a responsible camera. The camera 300 in this embodiment captures a captured image Pi so as to capture at least one of the first portion Dp1 and the second portion Dp2 shown in FIG. 1. Therefore, the target image in this embodiment includes at least one of the first portion Dp1 and the second portion Dp2. The first portion Dp1 is a left portion located to the left of the center position in the vehicle width direction of the vehicle 100, and the second portion Dp2 is a right portion located to the right of the center position of the vehicle 100. More specifically, the first portion Dp1 is a left rear corner of the vehicle 100, and the second portion Dp2 is a right rear corner of the vehicle 100. That is, in this embodiment, the first portion Dp1 and the second portion Dp2 are portions located approximately symmetrically to each other in the vehicle width direction of the vehicle 100. The first portion Dp1 and the second portion Dp2 correspond to detection portions that can be detected as detection points in the estimation process described later. The detection location is a location that is used to estimate a vehicle position that represents the location of the vehicle 100 in the estimation process.

[0024] The camera information acquisition unit 220 shown in Fig. 2 acquires camera information Ci related to the responsible camera. In this embodiment, identification information of each camera 300 is used as the camera information Ci. In this embodiment, the identification information of the camera 300 is transmitted from the camera 300 to the control device 200 together with the captured image Pi. In other embodiments, the camera information acquisition unit 220 may acquire the camera information Ci from a database stored in advance in the memory 202, an external computer, a recording medium, or the like.

[0025] In this embodiment, the position and orientation of each camera 300 are fixed with respect to the running road SR. Therefore, the identification information of the responsible camera is information that can specify the shooting conditions under which the responsible camera shoots the target image. The shooting conditions are conditions related to the detection parts that may be included in the captured image Pi, and include, for example, conditions related to the parts of the vehicle 100 that may be included as detection parts in the captured image Pi and conditions related to the direction in which the captured image Pi is shot. The conditions related to the direction in which the captured image Pi is shot are conditions related to the direction in which the vehicle 100 is shot when the captured image Pi is shot, and are, for example, conditions that determine whether the vehicle 100 is shot from the right side or the left side of the vehicle 100, and conditions that determine whether the vehicle 100 is shot from the front side or the rear side of the vehicle 100. In this embodiment, if the direction in which the captured image Pi is shot is specified, the detection parts that may be included in the captured image Pi are specified. The camera information Ci that can identify such shooting conditions is not limited to the identification information of the camera 300, but may also include information indicating the direction in which the camera 300 shoots the vehicle 100, or information indicating the detection areas that the camera 300 can shoot.

[0026] The estimation unit 250 executes an estimation process. The estimation process is a process of estimating the vehicle position of the vehicle 100 to be remotely controlled using a target image. The estimation process in this embodiment is a process of estimating the vehicle position using a detection point in the target image. More specifically, the estimation process includes an analysis process of detecting a detection point in the target image by analyzing the target image, and a process of estimating the vehicle position using the detected detection point. In this embodiment, the estimation process is executed by the estimation unit 250 executing the estimation program 280. Note that the estimation unit 250 can estimate the position and orientation of the vehicle 100 using, for example, the driving history of the vehicle 100 and the position information detected by the GNSS receiver 140. In addition, the driving direction of the vehicle 100 can be estimated using the direction, driving history, control command history, etc. of the vehicle 100.

[0027] The estimation unit 250 in this embodiment has an analysis unit 251 that executes an analysis process. The analysis unit 251 in this embodiment executes the analysis process by executing a first analysis program 281 and a second analysis program 282, which are analysis programs. The analysis programs are programs that realize a function of making a computer that executes the analysis program analyze a target image and detect a detection point in the target image.

[0028] The second determination unit 230 determines the contents of the estimation process according to the determined responsible camera. In this embodiment, the second determination unit 230 changes the contents of the estimation process according to the responsible camera by determining a program to be used in the estimation process according to the camera information Ci. More specifically, the second determination unit 230 determines a detection program according to the camera information Ci. The detection program is an analysis program used to detect the detection point in the analysis process.

[0029] In this specification, "determining a program according to a camera in charge" refers to determining the processing steps of a program according to a camera in charge. Therefore, the meaning of "determining a program according to a camera in charge" includes, for example, selecting one analysis program as a program to be determined from a plurality of programs prepared in advance according to a camera in charge. In this case, the selection of the detection program may be realized, for example, by executing a conditional branch according to the camera in charge in a comprehensive program including a plurality of programs. Furthermore, the meaning of "determining a program according to a camera in charge" includes, for example, preparing a program to be determined by rewriting part or all of the code included in the program according to the camera in charge.

[0030] In this embodiment, the second determination unit 230 determines the content of the estimation process by referring to the second correspondence data 272 using the camera information Ci. The second correspondence data 272 is data that records the correspondence between the camera information Ci and information for determining the content of the estimation process. More specifically, the second correspondence data 272 in this embodiment is data that records the correspondence between the identification information of each camera 300 and information for determining the program used in the estimation process. In other embodiments, the second correspondence data 272 may be data that records the correspondence between, for example, the identification information of each camera 300, the direction in which each camera 300 photographs the vehicle 100 to photograph the target image, and information for determining the content of the estimation process.

[0031] The above-mentioned estimation program 280 is a comprehensive program that includes, as analysis programs, a first analysis program 281 and a second analysis program 282. The first analysis program 281 and the second analysis program 282 will be described in detail later.

[0032] The command generating unit 260 generates a control command using the estimated position and orientation of the vehicle 100 and transmits it to the vehicle 100. This control command is a command to make the vehicle 100 travel along the target route represented by the target route data RD stored in the memory 202. The control command can be generated as a command including a driving force or braking force and a steering angle. Alternatively, the control command may be generated as a command including at least one of the position and orientation of the vehicle 100 and the route that the vehicle 100 will travel in the future.

[0033] The process management device 400 is, for example, configured by a computer, and executes management of the overall manufacturing process of the vehicle 100 in a factory. For example, when one vehicle 100 starts traveling along a target route, individual information indicating an identification number and model number for identifying the vehicle 100 is transmitted from the process management device 400 to the control device 200. This individual information corresponds to manufacturing information used for managing the manufacturing process of the vehicle 100. In addition, the position of the vehicle 100 detected by the control device 200 is also transmitted to the process management device 400. The function of the process management device 400 may be implemented in the same device as the control device 200.

[0034] 3 is a flowchart of the decision process in this embodiment. This decision process is a process for implementing the control method in this embodiment. The decision process is executed by the processor 201 of the control device 200, for example, at predetermined time intervals.

[0035] At S110 in FIG. 3, the route information acquisition unit 208 acquires route information of the vehicle 100 to be remotely controlled. At S120, the first determination unit 215 determines the responsible camera according to the acquired route information. At S130, the camera information acquisition unit 220 acquires camera information Ci of the determined responsible camera. At S140, the second determination unit 230 determines a detection program according to the acquired camera information Ci. At S140 in this embodiment, the detection program is determined according to the camera information Ci, and the contents of the estimation process are determined according to the responsible camera. More specifically, at S140, the contents of the analysis process are determined.

[0036] FIG. 4 is a diagram for explaining the determination process. FIG. 4 shows an example of a state in which the vehicle 100 travels along the road SR according to the target route by remote control. FIG. 4 shows the first route Rt1 and the second route Rt2 described above. The first route Rt1 corresponds to the first road SR1. The second route Rt2 includes routes Rt2a, Rt2b, Rt2c, Rt2d, and Rt2e. The route Rt2a corresponds to the portion of the first road SR1 up to the branch point Jc. The route Rt2b corresponds to the second road SR2. The route Rt2c corresponds to the third road SR3. The route Rt2d corresponds to the fourth road SR4. The route Rt2e corresponds to the portion of the first road SR1 after the junction point Cf.

[0037] In the first correspondence data 271 in this embodiment, the first route Rt1, the route Rt2a, and the route Rt2e are associated with the camera 300A, the route Rt2b is associated with the camera 300B, the route Rt2c is associated with the camera 300C, and the route Rt2d is associated with the camera 300D. That is, in the first correspondence data 271, each route included in the target route is associated with each camera on a one-to-one basis.

[0038] The camera 300A is disposed so as to be able to photograph the vehicle 100 traveling on the first route Rt1, the route Rt2a, or the route Rt2e from the left rear side. The camera 300B is disposed so as to be able to photograph the vehicle 100 traveling on the route Rt2b from the right rear side. The camera 300C is disposed so as to be able to photograph the vehicle 100 traveling on the route Rt2c from the left rear side. The camera 300D is disposed so as to be able to photograph the vehicle 100 traveling on the route Rt2d from the right rear side. In this embodiment, "photographing the vehicle 100 from the left rear side" refers to photographing the vehicle 100 from the left rear side so that the first portion Dp1 (left rear corner) is captured in the captured image Pi. "Photographing the vehicle 100 from the right rear side" refers to photographing the vehicle 100 from the right rear side so that the second portion Dp2 (right rear corner) is captured in the captured image Pi. Therefore, the captured image Pi of the vehicle 100 taken from the left rear side usually includes the first portion Dp1 regardless of the position where the vehicle 100 runs. On the other hand, the captured image Pi may or may not include the second portion Dp2 depending on the position where the vehicle 100 runs. Also, when the captured image Pi of the vehicle 100 taken from the left rear side includes the first portion Dp1 and the second portion Dp2, the second portion Dp2 appears smaller than the first portion Dp1 in the captured image Pi. Similarly, the captured image Pi of the vehicle 100 taken from the right rear side usually includes the second portion Dp2 regardless of the position where the vehicle 100 runs. On the other hand, the captured image Pi may or may not include the first portion Dp1 depending on the position where the vehicle 100 runs.

[0039] In the second correspondence data 272 in this embodiment, the camera information Ci of the camera 300A and the camera information Ci of the camera 300C are associated with parameters for selecting the first analysis program 281. In addition, the camera information Ci of the camera 300B and the camera 300D are associated with parameters for selecting the second analysis program 282. Note that the first analysis program 281 in this embodiment is an analysis program for analyzing a target image captured from the left rear side of the vehicle 100, and is used to detect the first part Dp1 as a detection location. In addition, the second analysis program 282 is an analysis program for analyzing a target image captured from the right rear side of the vehicle 100, and is used to detect the second part Dp2 as a detection location.

[0040] In FIG. 4, the vehicle 100 is traveling along the route Rt2a in the second route Rt2. Therefore, in the example of FIG. 4, route information representing the route Rt2a is acquired in S110 in FIG. 3. Next, in S120, the first correspondence data 271 is referenced according to the acquired route information, and the camera 300A is determined as the responsible camera corresponding to the route Rt2a. In S130, the camera information Ci regarding the determined camera 300A is acquired. Then, in S140, the second correspondence data 272 is referenced according to the acquired camera information Ci, and the first analysis program 281 is determined as the detection program according to the camera 300A. The above is substantially the same for the case where the vehicle 100 travels along other routes in the second route Rt2 or the first route Rt1. In FIG. 4, the vehicle 100 traveling along each route from the route Rt2b to the route Rt2e is shown by dashed lines.

[0041] In S110 in Fig. 3, for example, when the vehicle 100 is traveling near the end of a certain route, route information representing the next route of the route may be acquired as the target traveling route. More specifically, for example, in Fig. 4, when the vehicle 100 is traveling near the route Rt2b of the route Rt2a, route information representing the route Rt2b may be acquired as the target traveling route.

[0042] 5 is a flowchart of the estimation process in this embodiment. In this embodiment, the estimation process is executed every time a target image is transmitted to the control device 200 from the responsible camera.

[0043] In S205 of FIG. 5, the image acquisition unit 210 acquires a target image transmitted from the responsible camera. In S210, the estimation unit 250 determines whether or not to use the first analysis program 281 to analyze the target image based on the result of S140 of FIG. 3. If it is determined in S210 that the first analysis program 281 is to be used, the analysis unit 251 executes a first process realized by the first analysis program 281 in S220 to S230. The first process is a process for detecting the first part Dp1 as the detection location. If it is not determined in S210 that the first analysis program 281 is to be used, the analysis unit 251 executes a second process realized by the second analysis program 282 in S215 to S230. The second process is a process for detecting the second part Dp2 as the detection location. In the following, S220 realized by the first analysis program 281 will also be referred to as S220A, and S220 realized by the second analysis program 282 will also be referred to as S220B. The same applies to S225 and S230.

[0044] Unlike the first process, the second process in this embodiment includes the inversion process of S215. The inversion process is a process of generating an inverted image by mirror-inverting the target image. As a result, the target image that is not mirror-inverted is analyzed in S220A to S230A, whereas the inverted image is analyzed in S220B to S230B. The process of mirror-inverting an image in this manner is also called "flip". The first process and the second process are substantially similar except that the second process includes the inversion process. Therefore, in S220B to S230B, it is possible to analyze the inverted image using the same algorithm as in S220A to S230A, that is, in the same procedure. More specifically, when the first portion Dp1 is used as the detection location, in S220A to S230A, the first portion Dp1 is detected as the left rear corner in the non-inverted target image. On the other hand, when the second portion Dp2 is used as the detection point, the second portion Dp2, which is actually the right rear corner, is detected as the left rear corner in the pseudo-inverted image in S220B to S230B. That is, in this embodiment, the second portion Dp2 in the inverted image is detected as a portion corresponding to the first portion Dp1 in the target image that is not mirror-inverted.

[0045] FIG. 6 is an explanatory diagram showing an example of how the image Im1 is analyzed in the estimation process. The image Im1 shown in FIG. 6 is an image captured of the vehicle 100 traveling on the road SR. The image Im1 is expressed in a camera coordinate system having mutually orthogonal Xc and Yc axes as coordinate axes, and the focal point of the camera 300 as the origin. The image Im1 may be an inverted image, or may be a target image that is not mirror-inverted. The image Im1 may be appropriately subjected to various corrections and pre-processing, such as a distortion correction process for correcting distortion, a rotation process for rotating the image Im1 so that the direction of the movement vector V of the vehicle 100 in the image Im1 faces a predetermined direction, and a crop process for removing unnecessary areas not including the vehicle 100 from the image Im1. For example, an optical flow method is used to detect the magnitude and direction of the movement vector V.

[0046] 5, the analysis unit 251 executes the analysis process. In the analysis process in this embodiment, the analysis unit 251 detects the positioning point PP by calculating the coordinates of the positioning point PP shown in FIG. 6 using the image Im1. The coordinates of the positioning point PP are calculated as coordinates in the vicinity of the detection point in an image coordinate system described later. The positioning point PP detected in this manner represents the detection point in the target image.

[0047] In S220, the analysis unit 251 executes a mask process as a detection process for detecting the vehicle 100 in the image Im1. The mask process is a process for detecting the vehicle 100 in the image Im1 and masking a target area including the vehicle 100 in the image Im1 to generate an image Im2 including a mask area Ms in which the target area is masked. In the mask process, the estimation unit 250 generates the image Im2 by inputting the image Im1 to a machine learning model (not shown) that has been machine-learned to mask the vehicle 100 included in the input image. As the machine learning model, for example, a deep neural network (hereinafter also referred to as DNN) having a structure of a convolutional neural network (hereinafter also referred to as CNN) that realizes semantic segmentation and instance segmentation is used. Note that the machine learning model may be machine-learned using an algorithm other than a neural network, for example.

[0048] In S225, the analysis unit 251 executes a perspective transformation process. The perspective transformation process is a process for generating an image Im3 by performing a perspective transformation on the image Im2. In the perspective transformation process, the analysis unit 251 performs perspective transformation on the image Im2 into a bird's-eye image viewed from above the vehicle 100 (for example, directly above the vehicle 100) approximately perpendicular to the road surface of the road SR, for example, using perspective transformation parameters related to the position information and internal parameters of the camera 300. The image Im3 is expressed in an image coordinate system. The image coordinate system is a coordinate system having a point on an image plane projected by the perspective transformation as its origin and having mutually orthogonal Xi-axis and Yi-axis as its coordinate axes. The image Im3 includes a mask area Msb corresponding to the mask area Ms transformed by the perspective transformation.

[0049] In S230, the analysis unit 251 executes a positioning point calculation process to calculate the coordinates of the positioning point PP. In the positioning point calculation process in this embodiment, the analysis unit 251 calculates the coordinates of the positioning point PP using the coordinates (Xi1, Yi1) of the first coordinate point P1 and the coordinates (Xi2, Yi2) of the second coordinate point P2.

[0050] The first coordinate point P1 is identified as a coordinate point corresponding to the base coordinate point P0 in the image Im1 among the vertices of the rectangle R1b in the image Im2. The rectangle R1b corresponds to the first circumscribing rectangle R1 transformed by perspective transformation. The first circumscribing rectangle R1 is a rectangle in the image Im2 that circumscribes the mask area Ms and has a long side parallel to the movement vector V. The base coordinate point P0 is a coordinate point representing a vertex of the first circumscribing rectangle R1 that corresponds to the detection location of the vehicle 100. More specifically, the base coordinate point P0 is a coordinate point representing a vertex located to the left and rear of the center of gravity C of the vehicle 100 when the direction of the movement vector V is the forward direction among the vertices of the first circumscribing rectangle R1. The second coordinate point P2 is a coordinate point representing a vertex corresponding to the detection location of the vehicle 100 among the four vertices of the second circumscribing rectangle R2 in the image Im2. More specifically, the second coordinate point P2 is a coordinate representing a vertex of the second circumscribing rectangle R2 that is located to the left and rear of the center of gravity C when the direction of the movement vector V is forward. The second circumscribing rectangle R2 is a rectangle that circumscribes the mask area Msb and has a side parallel to the Xi axis and a side parallel to the Yi axis. In this way, the first coordinate point P1 and the second coordinate point P2 are both coordinate points determined according to the detection location, and therefore have a correlation with each other.

[0051] When the coordinate value Xi1 is greater than the coordinate value Xi2, the estimation unit 250 determines the Xi coordinate value of the positioning point PP to be the coordinate value Xi1. Conversely, when the coordinate value Xi1 is smaller than the coordinate value Xi2, the estimation unit 250 determines the Xi coordinate value to be the coordinate value Xi2. Similarly, when the coordinate value Yi1 is greater than the coordinate value Yi2, the Yi coordinate value of the positioning point PP is determined to be the coordinate value Yi1. When the coordinate value Yi1 is smaller than the coordinate value Yi2, the Yi coordinate value is determined to be the coordinate value Yi2. By determining the Xi coordinate value and the Yi coordinate value in this manner, the positioning point PP is detected.

[0052] At S235, the estimation unit 250 executes a vehicle coordinate calculation process. The vehicle coordinate calculation process is a process for calculating a vehicle coordinate point using the positioning point PP. The vehicle coordinate point is a coordinate point that represents the positioning point PP in the reference coordinate system Σr. At S235, the estimation unit 250 calculates the vehicle coordinate point using, for example, a predetermined estimation formula and the coordinate value of the positioning point PP calculated at S230. The command generation unit 260 generates a control command using the vehicle coordinate point calculated in the estimation process as the vehicle position. The control command thus generated is transmitted to the vehicle 100. That is, in this embodiment, calculating the vehicle coordinate point corresponds to estimating the vehicle position. Note that the estimation method of the vehicle position using the target image is not limited to the above. For example, the vehicle 100 in the target image may be detected by a process different from the mask process, a detection point in the target image may be detected by a process different from the perspective transformation process or the positioning point calculation process, or a coordinate point that represents the vehicle position may be calculated by a process different from the vehicle coordinate calculation process.

[0053] According to the system 10 in the present embodiment described above, the camera in charge can be determined according to the route information of the vehicle 100 to be remotely controlled, and the content of the estimation process can be determined according to the camera in charge. Therefore, regardless of which camera 300 is determined as the camera in charge, the possibility of appropriately estimating the vehicle position is increased. Furthermore, in the present embodiment, a control command is generated using the appropriate vehicle position estimated by the above estimation process. Therefore, the possibility of an appropriate control command being generated is increased, and the possibility of appropriately remotely controlling the vehicle 100 is increased.

[0054] In this embodiment, the second determination unit 230 alternatively determines whether to execute the first process in the estimation process or the second process in the estimation process according to the responsible camera. The first process is a process for detecting the first portion Dp1 (left portion) in the target image. The second process is a process for detecting the second portion Dp2 (right portion) in the target image. In this way, in the estimation process, the left portion or the right portion in the target image can be detected according to the responsible camera, and the vehicle position can be estimated using the detected left portion or the right portion. Such left portion or right portion can be easily photographed by the camera 300 by installing the camera 300 on the left side or the right side of the road SR. In particular, in this embodiment, the first portion Dp1 is a left rear corner, and the second portion Dp2 is a right rear corner. Therefore, the target image including the first portion Dp1 and the second portion Dp2 can be easily photographed by photographing the vehicle 100 from behind the vehicle 100 with the camera 300 installed on the left side or the right side of the road SR.

[0055] In this embodiment, the first process is a process of detecting the first portion Dp1 in a target image that is not mirror-inverted. The second process is a process of generating an inverted image by mirror-inverting the target image, and detecting the second portion Dp2 located symmetrically to the first portion Dp1 in the vehicle width direction in the inverted image. Therefore, in the estimation process, the first portion Dp1 and the second portion Dp2 can be detected in the same procedure when the first process is performed and when the second process is performed, except that the target image is mirror-inverted. Note that in other embodiments, for example, the first process may include the inversion process, and the second process may not include the inversion process.

[0056] In this embodiment, the second determination unit 230 determines the program to be used in the estimation process according to the determined responsible camera. Therefore, by determining the program by the second determination unit 230, the content of the estimation process can be determined according to the responsible camera.

[0057] B. Other embodiments: (B1) In the above embodiment, the system 10 is configured as a remote control system including the image acquisition unit 210, the estimation unit 250, and the command generation unit 260, but it does not have to be configured in this manner. For example, the system 10 may be configured as a system that determines the content of the estimation process and transmits the determination result to another system that includes the image acquisition unit 210, the estimation unit 250, and the command generation unit 260. In this case, the system 10 does not have to include, for example, some or all of the image acquisition unit 210, the estimation unit 250, and the command generation unit 260.

[0058] (B2) In the above embodiment, the first portion Dp1 and the second portion Dp2 do not have to be portions located symmetrically to each other in the vehicle width direction. For example, the left rear corner and the right front corner of the vehicle 100 may be used as the first portion Dp1 and the second portion Dp2, respectively. Furthermore, the left side and the right side of the vehicle 100 do not have to be used as the detection points. Furthermore, the number of parts used as the detection points may be three or more, or may be one.

[0059] (B3) In the above embodiment, the first process and the second process include the inversion process, but the inversion process may not be included. In this case, the estimation unit 250 may detect the first part Dp1 or the second part Dp2 in the target image that is not mirror-inverted, regardless of whether the detection part is the first part Dp1 or the second part Dp2.

[0060] (B4) In the above embodiment, the second determination unit 230 determines the program used in the estimation process according to the responsible camera, but this is not necessary. For example, even if the cameras 300 determined as the responsible cameras are different, the same program may be used for the estimation process. In this case, the second determination unit 230 may determine the parameters to be used in the estimation process according to the responsible camera, thereby determining the content of the estimation process according to the responsible camera.

[0061] (B5) In the above embodiment, the track SR is configured as a track that branches into two and then merges, but the present invention is not limited to this and may be configured as a track in any manner. For example, the track SR may be a track that branches into three or more tracks, or may not branch at all.

[0062] (B6) In the above embodiment, the vehicle 100 may have a configuration capable of moving by remote control, and may be in the form of a platform having the configuration described below, for example. Specifically, the vehicle 100 may have at least the vehicle control unit 115 and the communication device 130 in order to perform the three functions of "running", "turning" and "stopping" by remote control. That is, the vehicle 100 capable of moving by remote control may not have at least some of the interior parts such as the driver's seat and the dashboard, may not have at least some of the exterior parts such as the bumper and the fenders, and may not have a body shell. In this case, the remaining parts such as the body shell may be attached to the vehicle 100 before the vehicle 100 is shipped from the factory, or the remaining parts such as the body shell may be attached to the vehicle 100 after the vehicle 100 is shipped from the factory in a state in which the remaining parts such as the body shell are not attached to the vehicle 100. Note that the position of the platform may be determined in the same manner as the vehicle 100 in each embodiment.

[0063] The present disclosure is not limited to the above-mentioned embodiment, and can be realized in various configurations without departing from the spirit of the present disclosure. For example, the technical features in the embodiments corresponding to the technical features in each aspect described in the Summary of the Invention column can be appropriately replaced or combined to solve some or all of the above-mentioned problems or to achieve some or all of the above-mentioned effects. Furthermore, if the technical feature is not described as essential in this specification, it can be appropriately deleted. [Explanation of symbols]

[0064] 10...system, 100...vehicle, 110...vehicle control device, 111...processor, 112...memory, 113...input / output interface, 114...internal bus, 115...vehicle control unit, 116...position information acquisition unit, 120...actuator group, 130...communication device, 140...GNSS receiver, 200...control device, 201...processor, 202...memory, 203...input / output interface, 204...internal bus, 205...communication device, 208... route information acquisition unit, 210... image acquisition unit, 215... first determination unit, 220... camera information acquisition unit, 230... second determination unit, 250... estimation unit, 251... analysis unit, 260... command generation unit, 271... first corresponding data, 272... second corresponding data, 280... estimation program, 281... first analysis program, 282... second analysis program, 300, 300A, 300B, 300C, 300D... cameras, 400... process management device

Claims

1. 1. A system for use in estimating a vehicle position representative of a position of a vehicle traveling by remote control, comprising: A plurality of cameras arranged to capture images of the vehicle on a road; a route information acquisition unit that acquires route information regarding a route along which the vehicle will travel on the road by remote control; a first determination unit that determines, among the cameras, a camera that is responsible for capturing a target image including the vehicle traveling along the route according to the acquired route information; A second determination unit that determines the content of an estimation process for estimating the vehicle position using the target image according to the determined assigned camera.

2. 2. The system of claim 1, an image acquisition unit that acquires the target image from the assigned camera; An estimation unit that executes the estimation process; a command generating unit that generates a control command for the remote control using the vehicle position estimated by the estimation process.

3. 3. The system of claim 2, the target image includes at least one of a left side portion of the vehicle and a right side portion of the vehicle; the second determination unit, depending on the responsible camera, alternatively determines, as the content of the estimation process, a content including a first process of detecting the left side portion in the target image as a predetermined detection location, and a content including a second process of detecting the right side portion in the target image as the detection location; The estimation unit estimates the vehicle position using the detection location detected in the estimation process.

4. 4. The system of claim 3, the left side portion and the right side portion are portions located symmetrically to each other in a vehicle width direction of the vehicle, A system in which one of the first process and the second process is a process for generating an inverted image by mirror-inverting the target image and detecting the detection location in the inverted image, and the other of the first process and the second process is a process for detecting the detection location in the target image that is not mirror-inverted.

5. A system according to any one of claims 1 to 4, comprising: The second determination unit determines content of the estimation process by determining a program to be used in the estimation process according to the camera in charge.

Images