System

Abstract

To provide a system for suitably executing the estimation of a vehicle position by using a captured image of a vehicle regardless of imaging conditions for capturing the image.SOLUTION: A system used for estimating a vehicle position of a vehicle traveling by remote control includes: an image acquisition section for acquiring a captured image obtained by imaging the vehicle; a specification section for specifying a camera which has captured the image; and a determination section for determining contents of estimation processing for estimating the vehicle position by using the captured image according to imaging conditions of the specified camera.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to a system. [Background technology]

[0002] Patent Document 1 discloses a technique for running a vehicle by remote control during the vehicle manufacturing process. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2017-538619 Summary of the Invention [Problem to be solved by the invention]

[0004] In remote control of a vehicle, a technology is available for estimating the vehicle position using an image of the vehicle. There is a demand for a technology that can appropriately estimate the vehicle position regardless of the shooting conditions of the image. [Means for solving the problem]

[0005] The present disclosure can be realized in the following forms.

[0006] (1) According to one aspect of the present disclosure, there is provided a system for estimating a vehicle position representing a position of a vehicle traveling by remote control, the system including an image acquisition unit that acquires an image of the vehicle, a camera identification unit that identifies a camera that captured the image, and a determination unit that determines the content of an estimation process for estimating the vehicle position using the captured image according to the image capturing conditions of the identified camera. In this case, the content of the estimation process can be determined according to the shooting conditions of the camera that captured the captured image, which increases the possibility of appropriately estimating the vehicle position using the captured image regardless of the shooting conditions under which the captured image is captured. (2) In the above embodiment, the vehicle control device may further include an estimation unit that executes the estimation process, and a command generation unit that generates a control command for the remote control by using the vehicle position estimated by the estimation process. With this embodiment, the control command for the remote control can be generated by using an appropriate vehicle position estimated by the estimation process. (3) In the above embodiment, the captured image includes at least one of a left side portion of the vehicle and a right side portion of the vehicle, and the determination unit determines, according to the capturing conditions, the content of the estimation process alternatively between a first process that detects the left side portion in the captured image as a predetermined detection location and a second process that detects the right side portion in the captured image as the detection location, and the estimation unit may estimate the vehicle position using the detection location detected in the estimation process. With this embodiment, in the estimation process, the left side portion or the right side portion of the vehicle can be detected according to the capturing conditions of a camera that captured the captured image, and the vehicle position can be estimated using the detected left side portion or the right side portion. (4) In the above embodiment, the left side portion and the right side portion are portions located symmetrically to each other in the vehicle width direction of the vehicle, and one of the first process and the second process may be a process of generating an inverted image by mirror-inverting the captured image and detecting the detection location in the inverted image, and the other of the first process and the second process may be a process of detecting the detection location in the captured image that is not mirror-inverted. In this embodiment, the left side portion and the right side portion can be detected in the same procedure when the first process is performed and when the second process is performed in the estimation process, except for mirror-inverting the captured image. (5) In the above embodiment, the second determination unit may determine the content of the estimation process by determining a program to be used in the estimation process in accordance with the shooting conditions. In this embodiment, the content of the estimation process can be determined in accordance with the shooting conditions of a camera that captured the captured image by determining a program to be used in the estimation process by the determination unit.

[0007] In addition to the above-described system form, the present disclosure can be realized in the form of, for example, a control device, a control method, a computer program for implementing the control method, a non-transitory recording medium on which the computer program is recorded, etc. Furthermore, for example, the above-described system may estimate the vehicle position by inputting a captured image into a machine learning model prepared in advance. [Brief description of the drawings]

[0008] [Figure 1] A conceptual diagram showing the system configuration. [Diagram 2] FIG. 2 is a block diagram showing the configuration of a vehicle and a control device. [Diagram 3] 13 is a flowchart of a decision process. [Figure 4] 13 is a flowchart of an estimation process. [Diagram 5] FIG. 11 is an explanatory diagram showing an example of how an image is analyzed in the estimation process. [Figure 6] FIG. 11 is an explanatory diagram showing an example in which a camera that captures an image of a vehicle is determined in another embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] A. First embodiment: Fig. 1 is a conceptual diagram showing the configuration of a system 10 in the first embodiment. Fig. 2 is a block diagram showing the configuration of a vehicle 100 and a control device 200. The system 10 is configured as a remote control system that causes the vehicle 100 to run by remote control. The system 10 includes one or more vehicles 100, a control device 200 that executes remote control of the vehicle 100, a plurality of cameras 300 that capture images Pi including the vehicle 100, and a process management device 400 that manages the manufacturing process of the vehicle 100.

[0010] The vehicle 100 in this embodiment is a battery electric vehicle (BEV). Note that the vehicle 100 is not limited to an electric vehicle, and may be, for example, a gasoline vehicle, a hybrid vehicle, or a fuel cell vehicle.

[0011] In this embodiment, remote control of the vehicle 100 is performed in a factory where the vehicle 100 is manufactured. As shown in FIG. 1, the factory in this embodiment includes a first location PL1 and a second location PL2. The first location PL1 is, for example, a location where the vehicle 100 is assembled, and the second location PL2 is, for example, a location where the vehicle 100 is inspected. Any position in the factory is expressed by xyz coordinate values ​​of a reference coordinate system Σr. The reference coordinate system Σr is defined, for example, as a world coordinate system (also called a global coordinate system).

[0012] The first location PL1 and the second location PL2 are connected by a track SR on which the vehicle 100 can travel. In this embodiment, the track SR is configured as a track that branches into two at a branching point Jc and then merges at a merging point Cf. The track SR includes a first track SR1, a second track SR2, a third track SR3, and a fourth track SR4. The first track SR1 extends straight from the first location PL1 to the second location PL2. The second track SR2 to the fourth track SR4 connect the branching point Jc and the merging point Cf via a route different from the first track SR1. Note that the mode of the track SR is not limited to the above. For example, the track SR may be a track that branches into three or more tracks, or may not branch.

[0013] A target route along which the vehicle 100 travels is set on the road SR by remote control. In this embodiment, a first route Rt1 that goes straight through the branch point Jc and a second route Rt2 that turns at the branch point Jc are set on the road SR as target routes. The first route Rt1 corresponds to the first road SR1. The second route Rt2 includes routes Rt2a, Rt2b, Rt2c, Rt2d, and Rt2e. The route Rt2a corresponds to the portion of the first road SR1 up to the branch point Jc. The route Rt2b corresponds to the second road SR2. The route Rt2c corresponds to the third road SR3. The route Rt2d corresponds to the fourth road SR4. The route Rt2e corresponds to the portion of the first road SR1 from the junction point Cf onward. The number, shape, arrangement, and the like of the target routes are not limited to those described above and may be arbitrary.

[0014] The target route is determined, for example, according to the following conditions: the type of vehicle 100 (for example, the model, type, grade, and power source), the inspection results of the vehicle 100, the degree of congestion on each road, the degree of congestion in the next process, etc. For example, manufacturing information of the vehicle 100 described later can be used to determine such a target route.

[0015] A plurality of cameras 300 are installed around the road SR so that the vehicle 100 on the road SR can be photographed by at least one camera 300 at all times when the vehicle 100 is located at any position on the target route. The control device 200 can obtain the relative position and orientation of the vehicle 100 with respect to the target route and the traveling direction of the vehicle 100 in real time using the photographed images Pi photographed by each camera 300. The position, orientation, and traveling direction thus detected are used to generate a control command for remotely controlling the vehicle 100. In this embodiment, each camera 300 is arranged so that it can photograph an image overlooking the road SR from above. In addition, the position and orientation of each camera 300 are fixed, and the relative relationship between the reference coordinate system Σr and the device coordinate system (hereinafter also referred to as the camera coordinate system) of each camera 300 is known. A coordinate transformation matrix for mutually transforming coordinate values ​​of the reference coordinate system Σr and coordinate values ​​of the device coordinate system of each camera 300 is stored in advance in the control device 200. In remote control of the vehicle 100, for example, various sensors such as an on-board camera mounted on the vehicle 100, a LiDAR (Light Detection And Ranging), a millimeter wave radar, an ultrasonic sensor, and an infrared sensor may be used as auxiliary sensors.

[0016] The control device 200 in this embodiment is configured as a remote control device that generates the above-mentioned control command and transmits it to the vehicle 100. More specifically, the control device 200 generates a control command for causing the vehicle 100 to travel along a target route, and transmits the control command to the vehicle 100. The vehicle 100 travels according to the received control command. Therefore, the system 10 can move the vehicle 100 from the first location PL1 to the second location PL2 by remote control without using a transport device such as a crane or a conveyor.

[0017] As shown in FIG. 2, the vehicle 100 includes a vehicle control device 110 for controlling each part of the vehicle 100, an actuator group 120 that operates under the control of the vehicle control device 110, a communication device 130 for communicating with the control device 200 by wireless communication, and a GNSS (Global Navigation Satellite System) receiver 140 for acquiring position information of the vehicle 100. In this embodiment, the actuator group 120 includes an actuator of a drive device for accelerating the vehicle 100, an actuator of a steering device for changing the traveling direction of the vehicle 100, and an actuator of a braking device for decelerating the vehicle 100. The drive device includes a battery, a driving motor driven by the power of the battery, and a driving wheel rotated by the driving motor. The actuator of the drive device includes the driving motor. The actuator group 120 may include an actuator for swinging the wipers of the vehicle 100, an actuator for opening and closing the power windows of the vehicle 100, and the like.

[0018] The vehicle control device 110 is configured by a computer including a processor 111, a memory 112, an input / output interface 113, and an internal bus 114. The processor 111, the memory 112, and the input / output interface 113 are connected to each other via the internal bus 114 so as to be able to communicate bidirectionally. The actuator group 120, the communication device 130, and the GNSS receiver 140 are connected to the input / output interface 113.

[0019] In this embodiment, the processor 111 functions as the vehicle control unit 115 and the position information acquisition unit 116 by executing a program PG1 stored in advance in the memory 112. The vehicle control unit 115 controls the actuator group 120. When a driver is on board the vehicle 100, the vehicle control unit 115 can drive the vehicle 100 by controlling the actuator group 120 in response to an operation by the driver. The vehicle control unit 115 can also drive the vehicle 100 by controlling the actuator group 120 in response to a control command transmitted from the control device 200, regardless of whether a driver is on board the vehicle 100. The position information acquisition unit 116 acquires position information indicating the current location of the vehicle 100 using the GNSS receiver 140. However, the position information acquisition unit 116 and the GNSS receiver 140 can be omitted.

[0020] The control device 200 is configured by a computer including a processor 201, a memory 202, an input / output interface 203, and an internal bus 204. The processor 201, the memory 202, and the input / output interface 203 are connected via the internal bus 204 to enable bidirectional communication. A communication device 205 is connected to the input / output interface 203 for communicating with the vehicle 100, the camera 300, and the process management device 400 by wireless communication.

[0021] In this embodiment, the processor 201 executes a program PG2 prestored in the memory 202 to function as an image acquisition unit 210, a camera identification unit 215, a camera information acquisition unit 218, a determination unit 220, an estimation unit 250, and a command generation unit 260. In addition to the program PG2, the memory 202 stores target route data RD representing a set target route, corresponding data 271 described later, and an estimation program 280. The estimation program 280 in this embodiment is a comprehensive program including a first analysis program 281 and a second analysis program 282 described later.

[0022] The image acquisition unit 210 acquires a captured image Pi of the vehicle 100 captured by the camera 300. The camera 300 in this embodiment captures the captured image Pi so as to capture at least one of the first portion Dp1 and the second portion Dp2 shown in FIG. 1. Therefore, the captured image Pi in this embodiment includes at least one of the first portion Dp1 and the second portion Dp2. The first portion Dp1 is a left portion located to the left of the center position in the vehicle width direction of the vehicle 100, and the second portion Dp2 is a right portion located to the right of the center position of the vehicle 100. More specifically, the first portion Dp1 is a left rear corner of the vehicle 100, and the second portion Dp2 is a right rear corner of the vehicle 100. That is, in this embodiment, the first portion Dp1 and the second portion Dp2 are portions located approximately symmetrically to each other in the vehicle width direction of the vehicle 100. The first portion Dp1 and the second portion Dp2 correspond to detection portions that can be detected as detection points in the estimation process described below. The detection points are points that are used to estimate the vehicle position that represents the position of the vehicle 100 in the estimation process.

[0023] The camera identification unit 215 identifies the imaging camera, which is the camera 300 that captured the captured image Pi. In this embodiment, when the captured image Pi is acquired by the image acquisition unit 210, the camera identification unit 215 identifies the camera 300 that transmitted the captured image Pi to the control device 200 as the imaging camera. In this embodiment, the position and orientation of each camera 300 are fixed with respect to the road SR. Therefore, identifying the imaging camera corresponds to identifying the imaging conditions of the captured image Pi by the imaging camera. The imaging conditions are conditions related to the detection parts included in the captured image Pi, and include, for example, conditions related to the parts of the vehicle 100 that may be included as detection parts in the captured image Pi and conditions related to the orientation in which the captured image Pi is captured. The condition related to the direction in which the captured image Pi is captured is a condition related to the direction in which the vehicle 100 is captured when the captured image Pi is captured, such as a condition that determines whether the vehicle 100 is captured from the right side or the left side of the vehicle 100, or a condition that determines whether the vehicle 100 is captured from the front side or the rear side of the vehicle 100. If the direction in which the captured image Pi is captured is specified, the detection portion included in the captured image Pi is specified. Hereinafter, the capturing conditions of the captured image Pi by the camera 300 are also simply referred to as the "capturing conditions of the camera 300".

[0024] The camera information acquisition unit 218 acquires camera information Ci related to the imaging camera. In this embodiment, identification information of each camera 300 is used as the camera information Ci. In this embodiment, each camera 300 is configured to transmit its own identification information to the control device 200 together with the captured image Pi. Note that the above-mentioned camera identification unit 215 may identify the imaging camera using the identification information transmitted from the camera 300 in this manner. In other embodiments, the camera information acquisition unit 218 may acquire the camera information Ci from a database previously stored in the memory 202, an external computer, a recording medium, or the like.

[0025] In the system 10 in this embodiment, the camera 300 that captures the captured image Pi is determined in accordance with the route along which the vehicle 100 travels. The camera 300A shown in FIG. 1 captures the vehicle 100 traveling along the first route Rt1, the route Rt2a, or the route Rt2e from the left rear side. The camera 300B captures the vehicle 100 traveling along the route Rt2b from the right rear side. The camera 300C captures the vehicle 100 traveling along the route Rt2c from the left rear side. The camera 300D captures the vehicle 100 traveling along the route Rt2d from the right rear side. In this embodiment, "capturing the vehicle 100 from the left rear side" refers to capturing the vehicle 100 from the left rear side so that the first portion Dp1 (left rear corner) is captured in the captured image Pi. "Photographing the vehicle 100 from the right rear side" refers to photographing the vehicle 100 from the right rear side so that the second portion Dp2 (right rear corner) appears in the photographed image Pi. Therefore, the photographed image Pi photographed from the left rear side of the vehicle 100 usually includes the first portion Dp1 regardless of the position where the vehicle 100 is traveling. On the other hand, the photographed image Pi may or may not include the second portion Dp2 depending on the position where the vehicle 100 is traveling. Also, when the photographed image Pi photographed from the left rear side of the vehicle 100 includes the first portion Dp1 and the second portion Dp2, the second portion Dp2 appears smaller than the first portion Dp1 in the photographed image Pi. Similarly, the photographed image Pi photographed from the right rear side of the vehicle 100 usually includes the second portion Dp2 regardless of the position where the vehicle 100 is traveling. On the other hand, the captured image Pi may or may not include the first portion Dp1 depending on the position where the vehicle 100 is traveling.

[0026] The estimation unit 250 shown in FIG. 2 executes an estimation process. The estimation process is a process of estimating the vehicle position of the vehicle 100 to be remotely controlled by using the captured image Pi. The estimation process in this embodiment is a process of estimating the vehicle position by using a detection point in the captured image Pi. More specifically, the estimation process includes an analysis process of detecting a detection point in the captured image Pi by analyzing the captured image Pi, and a process of estimating the vehicle position by using the detected detection point. In this embodiment, the estimation process is executed by the estimation unit 250 executing the estimation program 280. Note that the estimation unit 250 can estimate the position and orientation of the vehicle 100 by using, for example, the driving history of the vehicle 100 and the position information detected by the GNSS receiver 140. In addition, the driving direction of the vehicle 100 can be estimated by using the direction, driving history, control command history, etc. of the vehicle 100.

[0027] The estimation unit 250 in this embodiment has an analysis unit 251 that executes an analysis process. The analysis unit 251 in this embodiment executes the analysis process by executing a first analysis program 281 and a second analysis program 282, which are analysis programs. The analysis programs are programs that realize a function of making a computer that executes the analysis programs analyze a photographed image Pi and detect a detection point in the photographed image Pi. The first analysis program 281 in this embodiment is an analysis program for analyzing a photographed image Pi photographed from the left rear side of the vehicle 100, and is used to detect a first part Dp1 as a detection point. The second analysis program 282 is an analysis program for analyzing a photographed image Pi photographed from the right rear side of the vehicle 100, and is used to detect a second part Dp2 as a detection point.

[0028] The determination unit 220 determines the contents of the estimation process according to the shooting conditions of the identified imaging camera. The determination unit 220 in this embodiment determines the program used in the estimation process according to the camera information Ci of the imaging camera, thereby changing the contents of the estimation process according to the shooting conditions of the imaging camera. More specifically, the determination unit 220 determines the detection program according to the camera information Ci of the imaging camera. The detection program is an analysis program used to detect the detection point in the analysis process. As described above, in this embodiment, the identification of the imaging camera corresponds to the identification of the shooting conditions of the imaging camera. Therefore, by determining the contents of the estimation process according to the imaging camera, the contents of the estimation process can be determined according to the shooting conditions of the imaging camera.

[0029] In this specification, "determining a program according to the imaging conditions" refers to determining the processing steps of the program according to the imaging conditions. Therefore, the meaning of "determining a program according to the imaging conditions" includes, for example, selecting one analysis program as a program to be determined from a plurality of programs prepared in advance according to the imaging conditions. In this case, the selection of the detection program may be realized, for example, by executing a conditional branch according to the imaging conditions in a comprehensive program including a plurality of programs. In addition, the meaning of "determining a program according to the imaging conditions" includes, for example, preparing a program to be determined by rewriting a part or all of the code included in the program according to the imaging conditions.

[0030] In this embodiment, the determination unit 220 determines the content of the estimation process by referring to the correspondence data 271 using the camera information Ci of the imaging camera. The correspondence data 271 is data that records the correspondence between the camera information Ci and information for determining the content of the estimation process. More specifically, the correspondence data 271 in this embodiment is data that records the correspondence between the identification information of each camera 300 and information for determining the program used in the estimation process. In the correspondence data 271 in this embodiment, the identification information of the cameras 300A and 300C is associated with a parameter for selecting the first analysis program 281. Also, the identification information of the cameras 300B and 300D is associated with a parameter for selecting the second analysis program 282. In another embodiment, the correspondence data 271 may be data that records the correspondence between the identification information of each camera 300, the shooting conditions of each camera 300, and information for determining the content of the estimation process, for example.

[0031] The command generating unit 260 generates a control command for remote control using the estimated position and orientation of the vehicle 100 and transmits it to the vehicle 100. This control command is a command to make the vehicle 100 travel along the target route represented by the target route data RD stored in the memory 202. The control command can be generated as a command including a driving force or braking force, and a steering angle. Alternatively, the control command may be generated as a command including at least one of the position and orientation of the vehicle 100, and a future route.

[0032] The process management device 400 is, for example, configured by a computer, and executes management of the overall manufacturing process of the vehicle 100 in a factory. For example, when one vehicle 100 starts traveling along a target route, individual information indicating an identification number and model number for identifying the vehicle 100 is transmitted from the process management device 400 to the control device 200. This individual information corresponds to manufacturing information used for managing the manufacturing process of the vehicle 100. In addition, the position of the vehicle 100 detected by the control device 200 is also transmitted to the process management device 400. The function of the process management device 400 may be implemented in the same device as the control device 200.

[0033] 3 is a flowchart of the decision process in this embodiment. This decision process is a process for implementing the control method in this embodiment. The decision process is executed by the processor 201 of the control device 200, for example, at predetermined time intervals.

[0034] 3, the image acquisition unit 210 acquires a captured image Pi. In S120, the camera identification unit 215 identifies the camera 300 that captured the captured image Pi acquired in S110 as the imaging camera. In S130, the camera information acquisition unit 218 acquires camera information Ci of the imaging camera identified in S120. In S140, the determination unit 220 determines a detection program according to the acquired camera information Ci. In S140 in this embodiment, the detection program is determined according to the camera information Ci, and the content of the estimation process is determined according to the imaging conditions of the imaging camera. More specifically, in S140, the content of the analysis process is determined.

[0035] FIG. 1 shows a state in which the vehicle 100 is traveling on the route Rt2a of the second route Rt2. When the vehicle 100 is traveling on the route Rt2a, the captured image Pi is captured by the camera 300A. In this case, in S110 of FIG. 3, the captured image Pi captured by the camera 300A is acquired. In S120, the camera 300A that captured the captured image Pi is identified as the imaging camera. In S130, the camera information Ci of the camera 300A identified as the imaging camera is acquired. In S140, the corresponding data 271 is referred to according to the acquired camera information Ci, and the first analysis program 281 is determined as the detection program according to the camera 300A. The same is true when the vehicle 100 travels on another route in the second route Rt2 or the first route Rt1.

[0036] 4 is a flowchart of the estimation process in this embodiment. In this embodiment, the estimation process is executed every time the determination process is completed.

[0037] In S210 of FIG. 4, the estimation unit 250 determines whether or not to use the first analysis program 281 in the analysis of the photographed image Pi acquired in S110 based on the result of S140 of FIG. 3. If it is determined in S210 that the first analysis program 281 is to be used, the analysis unit 251 executes a first process realized by the first analysis program 281 in S220 to S230. The first process is a process for detecting the first part Dp1 as the detection part. If it is not determined in S210 that the first analysis program 281 is to be used, the analysis unit 251 executes a second process realized by the second analysis program 282 in S215 to S230. The second process is a process for detecting the second part Dp2 as the detection part. In the following, S220 realized by the first analysis program 281 is also referred to as S220A, and S220 realized by the second analysis program 282 is also referred to as S220B. The same is true for S225 and S230.

[0038] Unlike the first process, the second process in this embodiment includes the inversion process of S215. The inversion process is a process of generating an inverted image by mirror-inverting the captured image Pi. As a result, the captured image Pi that is not mirror-inverted is analyzed in S220A to S230A, whereas the inverted image is analyzed in S220B to S230B. The process of mirror-inverting an image in this manner is also called "flip". The first process and the second process are substantially similar except that the second process includes the inversion process. Therefore, in S220B to S230B, it is possible to analyze the inverted image using the same algorithm as in S220A to S230A, that is, in the same procedure. More specifically, when the first portion Dp1 is used as the detection location, in S220A to S230A, the first portion Dp1 is detected as the left rear corner in the non-inverted captured image Pi. On the other hand, when the second portion Dp2 is used as the detection point, the second portion Dp2, which is actually the right rear corner, is detected as the left rear corner in the inverted image in a pseudo manner in S220B to S230B. That is, in this embodiment, the second portion Dp2 is detected in the inverted image as a portion corresponding to the first portion Dp1 in the captured image Pi that is not mirror-inverted.

[0039] FIG. 5 is an explanatory diagram showing an example of how the image Im1 is analyzed in the estimation process. The image Im1 shown in FIG. 5 is an image captured of the vehicle 100 traveling on the road SR. The image Im1 is expressed in a camera coordinate system having mutually orthogonal Xc and Yc axes as coordinate axes, and the focal point of the camera 300 as the origin. The image Im1 may be an inverted image, or may be a captured image Pi that is not mirror-inverted. The image Im1 may be subjected to various corrections and pre-processing, such as a distortion correction process for correcting distortion, a rotation process for rotating the image Im1 so that the direction of the movement vector V of the vehicle 100 in the image Im1 faces a predetermined direction, and a crop process for removing unnecessary areas not including the vehicle 100 from the image Im1. For example, an optical flow method is used to detect the magnitude and direction of the movement vector V.

[0040] 4, the analysis unit 251 executes the analysis process. In the analysis process in this embodiment, the analysis unit 251 detects the positioning point PP by calculating the coordinates of the positioning point PP shown in FIG. 5 using the image Im1. The coordinates of the positioning point PP are calculated as coordinates in the vicinity of the detection point in an image coordinate system described later. The positioning point PP detected in this manner represents the detection point in the captured image Pi.

[0041] In S220, the analysis unit 251 executes a mask process as a detection process for detecting the vehicle 100 in the image Im1. The mask process is a process for detecting the vehicle 100 in the image Im1 and masking a target area including the vehicle 100 in the image Im1 to generate an image Im2 including a mask area Ms in which the target area is masked. In the mask process, the estimation unit 250 generates the image Im2 by inputting the image Im1 to a machine learning model (not shown) that has been machine-learned to mask the vehicle 100 included in the input image. As the machine learning model, for example, a deep neural network (hereinafter also referred to as DNN) having a structure of a convolutional neural network (hereinafter also referred to as CNN) that realizes semantic segmentation and instance segmentation is used. Note that the machine learning model may be machine-learned using an algorithm other than a neural network, for example.

[0042] In S225, the analysis unit 251 executes a perspective transformation process. The perspective transformation process is a process for generating an image Im3 by performing a perspective transformation on the image Im2. In the perspective transformation process, the analysis unit 251 performs perspective transformation on the image Im2 into a bird's-eye image viewed from above the vehicle 100 (for example, directly above the vehicle 100) approximately perpendicular to the road surface of the road SR, for example, using perspective transformation parameters related to the position information and internal parameters of the camera 300. The image Im3 is expressed in an image coordinate system. The image coordinate system is a coordinate system having a point on an image plane projected by the perspective transformation as its origin and having mutually orthogonal Xi-axis and Yi-axis as its coordinate axes. The image Im3 includes a mask area Msb corresponding to the mask area Ms transformed by the perspective transformation.

[0043] In S230, the analysis unit 251 executes a positioning point calculation process to calculate the coordinates of the positioning point PP. In the positioning point calculation process in this embodiment, the analysis unit 251 calculates the coordinates of the positioning point PP using the coordinates (Xi1, Yi1) of the first coordinate point P1 and the coordinates (Xi2, Yi2) of the second coordinate point P2.

[0044] The first coordinate point P1 is identified as a coordinate point corresponding to the base coordinate point P0 in the image Im1 among the vertices of the rectangle R1b in the image Im2. The rectangle R1b corresponds to the first circumscribing rectangle R1 transformed by perspective transformation. The first circumscribing rectangle R1 is a rectangle in the image Im2 that circumscribes the mask area Ms and has a long side parallel to the movement vector V. The base coordinate point P0 is a coordinate point representing a vertex of the first circumscribing rectangle R1 that corresponds to the detection location of the vehicle 100. More specifically, the base coordinate point P0 is a coordinate point representing a vertex located to the left and rear of the center of gravity C of the vehicle 100 when the direction of the movement vector V is the forward direction among the vertices of the first circumscribing rectangle R1. The second coordinate point P2 is a coordinate point representing a vertex corresponding to the detection location of the vehicle 100 among the four vertices of the second circumscribing rectangle R2 in the image Im2. More specifically, the second coordinate point P2 is a coordinate representing a vertex of the second circumscribing rectangle R2 that is located to the left and rear of the center of gravity C when the direction of the movement vector V is forward. The second circumscribing rectangle R2 is a rectangle that circumscribes the mask area Msb and has a side parallel to the Xi axis and a side parallel to the Yi axis. In this way, the first coordinate point P1 and the second coordinate point P2 are both coordinate points determined according to the detection location, and therefore have a correlation with each other.

[0045] When the coordinate value Xi1 is greater than the coordinate value Xi2, the estimation unit 250 determines the Xi coordinate value of the positioning point PP to be the coordinate value Xi1. Conversely, when the coordinate value Xi1 is smaller than the coordinate value Xi2, the estimation unit 250 determines the Xi coordinate value to be the coordinate value Xi2. Similarly, when the coordinate value Yi1 is greater than the coordinate value Yi2, the Yi coordinate value of the positioning point PP is determined to be the coordinate value Yi1. When the coordinate value Yi1 is smaller than the coordinate value Yi2, the Yi coordinate value is determined to be the coordinate value Yi2. By determining the Xi coordinate value and the Yi coordinate value in this manner, the positioning point PP is detected.

[0046] At S235, the estimation unit 250 executes a vehicle coordinate calculation process. The vehicle coordinate calculation process is a process for calculating a vehicle coordinate point using the positioning point PP. The vehicle coordinate point is a coordinate point that represents the positioning point PP in the reference coordinate system Σr. At S235, the estimation unit 250 calculates the vehicle coordinate point using, for example, a predetermined estimation formula and the coordinate value of the positioning point PP calculated at S230. The command generation unit 260 generates a control command using the vehicle coordinate point calculated in the estimation process as the vehicle position. The control command thus generated is transmitted to the vehicle 100. That is, in this embodiment, calculating the vehicle coordinate point corresponds to estimating the vehicle position. Note that the estimation method of the vehicle position using the captured image Pi is not limited to the above. For example, the vehicle 100 in the captured image Pi may be detected by a process other than the mask process, a detection location in the captured image Pi may be detected by a process other than the perspective transformation process or the positioning point calculation process, or a coordinate point that represents the vehicle position may be calculated by a process other than the vehicle coordinate calculation process.

[0047] According to the system 10 in the present embodiment described above, the contents of the estimation process can be determined according to the imaging conditions of the imaging cameras. Therefore, for example, even if the imaging conditions differ for each camera 300, the possibility of appropriately estimating the vehicle position regardless of the imaging conditions is increased. In addition, since a control command is generated using the vehicle position estimated by the estimation process, the possibility of an appropriate control command being generated is increased. Therefore, the possibility of appropriately remotely controlling the vehicle 100 is increased.

[0048] In this embodiment, the determination unit 220 selectively determines the content of the estimation process to include either the first process or the second process, depending on the shooting conditions of the imaging camera. The first process is a process for detecting the first portion Dp1 (left side portion) in the captured image Pi. The second process is a process for detecting the second portion Dp2 (right side portion) in the captured image Pi. In this way, in the estimation process, the left side portion or the right side portion in the captured image Pi can be detected depending on the shooting conditions of the imaging camera, and the vehicle position can be estimated using the detected left side portion or the right side portion. The left side portion or the right side portion can be easily photographed by the camera 300 by installing the camera 300 on the left side or the right side of the road SR. Particularly, in this embodiment, the first portion Dp1 is a left rear corner, and the second portion Dp2 is a right rear corner. Therefore, by photographing the vehicle 100 from behind using a camera 300 installed on the left or right side of the road SR, it is possible to easily capture an image Pi including the first portion Dp1 and the second portion Dp2.

[0049] In this embodiment, the first process is a process of detecting the first portion Dp1 in the captured image Pi that has not been mirror-inverted. The second process is a process of generating an inverted image by mirror-inverting the captured image Pi, and detecting the second portion Dp2 located symmetrically to the first portion Dp1 in the vehicle width direction in the inverted image. Therefore, in the estimation process, the first and second processes can be detected in the same procedure except that the captured image Pi is mirror-inverted. Note that in other embodiments, for example, the first process may include the inversion process, and the second process may not include the inversion process.

[0050] In the present embodiment, the determination unit 220 determines a program to be used in the estimation process according to the identified imaging camera. Therefore, by determining the program by the determination unit 220, the content of the estimation process can be determined according to the shooting conditions of the imaging camera.

[0051] B. Other embodiments: (B1) In the above embodiment, the system 10 is configured as a remote control system including the vehicle 100, the camera 300, the estimation unit 250, and the command generation unit 260, but it does not have to be configured in this way. For example, the system 10 may be configured as a system that determines the contents of the estimation process and transmits the determination result to another system including the vehicle 100, the camera 300, the estimation unit 250, and the command generation unit 260. In this case, the system 10 may not include, for example, some or all of the vehicle 100, the camera 300, the estimation unit 250, and the command generation unit 260. Also, in this case, the system 10 may be configured to transmit, for example, the captured image Pi to the other system together with the determination result. In this case, the estimation unit 250 provided in the other system may acquire the captured image Pi transmitted from the system 10 in the estimation process.

[0052] (B2) In the above embodiment, the camera 300 that will capture an image of the vehicle 100 is determined in accordance with the route traveled by the vehicle 100. As a condition for determining the camera 300 that will capture an image of the vehicle 100, other conditions may be applied instead of or in addition to the above-described conditions related to the route.

[0053] For example, FIG. 6 is an explanatory diagram showing an example in which the camera 300 that captures the vehicle 100 is determined in another embodiment. FIG. 6 shows the vehicle 100 traveling on the third route Rt3 on the road SRb. In the example of FIG. 6, the camera 300 that captures the vehicle 100 traveling on the third route Rt3 is determined by the traveling timing of the vehicle 100. In this way, for example, when the vehicle 100 travels outdoors by remote control, it is possible to suppress the capture of a captured image Pi that is not suitable for estimation processing due to the influence of sunlight. More specifically, in the example of FIG. 6, the camera 300E captures the vehicle 100 in the time period TP1, and the camera 300F captures the vehicle 100 in the time period other than the time period TP1. The camera 300E captures the vehicle 100 from the left rear side. The camera 300F captures the vehicle 100 from the right rear side. In the example of FIG. 6, for example, in the correspondence data 271, the camera information Ci of the camera 300E is associated with a parameter for selecting the first analysis program 281, and the camera information Ci of the camera 300F is associated with a parameter for selecting the second analysis program 282. Even in this form, as in the first embodiment, the content of the estimation process can be determined according to the shooting conditions of the imaging camera. In this way, by configuring the control device 200 and the system 10 to determine the content of the estimation process according to the shooting conditions of the imaging camera, the possibility of appropriately determining the content of the estimation process increases regardless of the method for determining the camera 300 that captures the captured image Pi. Therefore, the control device 200 and the system 10 can be easily applied to various remote control systems.

[0054] (B3) In the above embodiment, the first process and the second process include the inversion process, but the inversion process may not be included. In this case, the estimation unit 250 may detect the first part Dp1 or the second part Dp2 in the captured image Pi that is not mirror-inverted, regardless of whether the first part Dp1 or the second part Dp2 is detected as the detection location.

[0055] (B4) In the above embodiment, the first portion Dp1 and the second portion Dp2 do not have to be portions located symmetrically to each other in the vehicle width direction. For example, the left rear corner and the right front corner of the vehicle 100 may be used as the first portion Dp1 and the second portion Dp2, respectively. Furthermore, the left portion and the right portion of the vehicle 100 do not have to be used as the detection location. Furthermore, the number of parts used as the detection location may be three or more, or may be one.

[0056] (B5) In the above embodiment, the determination unit 220 determines the program used in the estimation process according to the shooting conditions of the imaging camera, but this is not necessary. For example, even if the shooting conditions are variously different, the same program may be used for the estimation process. In this case, the determination unit 220 may determine the parameters to be used in the estimation process according to the shooting conditions of the imaging camera, thereby determining the content of the estimation process according to the shooting conditions of the imaging camera.

[0057] (B6) In the above embodiment, the camera information Ci may be information representing the photographing conditions of the camera 300, for example, information relating to a detection portion that may be included in the photographed image Pi photographed by the camera 300. In this case, the camera information Ci may be information representing the orientation in which the camera 300 photographs the vehicle 100, or information representing a detection portion that the camera 300 may photograph. By using such camera information Ci, for example, even if the position and orientation of each camera 300 are not fixed with respect to the running path SR, the content of the estimation process can be determined according to the photographing conditions of the imaging camera. That is, in this case, the camera 300 may be configured so that the position and orientation of the camera 300 with respect to the running path SR can be changed by, for example, the control device 200 or the process management device 400. In this way, the photographing conditions of the camera 300 can be changed according to, for example, the timing at which the vehicle 100 runs or the position of the vehicle 100.

[0058] (B7) In the above embodiment, the vehicle 100 may have a configuration capable of moving by remote control, and may be in the form of a platform having the configuration described below, for example. Specifically, the vehicle 100 may have at least the vehicle control unit 115 and the communication device 130 in order to perform the three functions of "running", "turning" and "stopping" by remote control. That is, the vehicle 100 capable of moving by remote control may not have at least some of the interior parts such as the driver's seat and the dashboard, may not have at least some of the exterior parts such as the bumper and the fenders, and may not have a body shell. In this case, the remaining parts such as the body shell may be attached to the vehicle 100 before the vehicle 100 is shipped from the factory, or the remaining parts such as the body shell may be attached to the vehicle 100 after the vehicle 100 is shipped from the factory in a state in which the remaining parts such as the body shell are not attached to the vehicle 100. Note that the position of the platform may be determined in the same manner as the vehicle 100 in each embodiment.

[0059] The present disclosure is not limited to the above-mentioned embodiment, and can be realized in various configurations without departing from the spirit of the present disclosure. For example, the technical features in the embodiments corresponding to the technical features in each aspect described in the Summary of the Invention column can be appropriately replaced or combined to solve some or all of the above-mentioned problems or to achieve some or all of the above-mentioned effects. Furthermore, if the technical feature is not described as essential in this specification, it can be appropriately deleted. [Explanation of symbols]

[0060] 10...system, 100...vehicle, 110...vehicle control device, 111...processor, 112...memory, 113...input / output interface, 114...internal bus, 115...vehicle control unit, 116...position information acquisition unit, 120...actuator group, 130...communication device, 140...GNSS receiver, 200...control device, 201...processor, 202...memory, 203...input / output interface, 204...internal bus, 205...communication device, 210...image acquisition unit, 215...camera identification unit, 218...camera information acquisition unit, 220...determination unit, 250...estimation unit, 251...analysis unit, 260...command generation unit, 271...corresponding data, 280...estimation program, 281...first analysis program, 282...second analysis program, 300, 300A, 300B, 300C, 300D, 300E, 300F...camera, 400...process management device

Claims

1. 1. A system for use in estimating a vehicle position representative of a position of a vehicle traveling by remote control, comprising: an image acquisition unit that acquires an image of the vehicle; a camera identification unit for identifying a camera that captured the captured image; a determination unit that determines the content of an estimation process for estimating the vehicle position using the captured image according to the identified shooting conditions of the camera.

2. 2. The system of claim 1, An estimation unit that executes the estimation process; A system comprising: a command generating unit that generates a control command for the remote control using the vehicle position estimated by the estimation process.

3. 3. The system of claim 2, The captured image includes at least one of a left side portion of the vehicle and a right side portion of the vehicle, The determination unit determines, according to the shooting conditions, as the content of the estimation process, alternatively, between a first process including a first process for detecting the left side portion in the captured image as a predetermined detection location and a second process including a second process for detecting the right side portion in the captured image as the detection location; The estimation unit estimates the vehicle position using the detection location detected in the estimation process.

4. 4. The system of claim 3, the left side portion and the right side portion are portions located symmetrically to each other in a vehicle width direction of the vehicle, A system in which one of the first process and the second process is a process for generating an inverted image by mirror-inverting the captured image and detecting the detection location in the inverted image, and the other of the first process and the second process is a process for detecting the detection location in the captured image that is not mirror-inverted.

5. A system according to any one of claims 1 to 4, comprising: The determination unit determines content of the estimation processing by determining a program to be used in the estimation processing in accordance with the shooting conditions.

Images