system

Abstract

To appropriately execute estimation of a vehicle position using a captured image of the vehicle.SOLUTION: A system for use in estimating a vehicle position for a vehicle that travels by remote control includes: an image acquisition unit which acquires a plurality of captured images of the vehicle, the captured images being captured by a plurality of cameras; an evaluation unit which executes, for each captured image, evaluation on features of the captured images that may affect detecting the vehicle in the captured images; and a first determination unit which determines an estimation image for use in estimation processing for estimating a vehicle position from the captured images, in accordance with results of the evaluation.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a technique for driving a vehicle by remote control in a vehicle manufacturing process.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In remote control of a vehicle, a technique for estimating the vehicle position using a captured image of the vehicle can be used. A technique for appropriately performing such estimation of the vehicle position is desired.

Means for Solving the Problems

[0005] The present disclosure can be realized in the following forms. According to one embodiment of the present disclosure, a system is provided for estimating the vehicle position of a vehicle that is driven by remote control. The system comprises: an image acquisition unit that acquires a plurality of captured images of the vehicle, the plurality of captured images being captured by a plurality of cameras; an evaluation unit that performs an evaluation of feature quantities of the captured images that may affect the detection of the vehicle in the captured images for each captured image; a first determination unit that determines an estimation image to be used for estimation processing to estimate the vehicle position from each of the captured images according to the results of the evaluation; and a second determination unit that determines the content of the estimation processing according to the shooting conditions of the camera that captured the estimation image, which relate to whether the vehicle is photographed from the left or the right side by the camera. The estimation image includes at least one of the left portion of the vehicle and the right portion of the vehicle, and the second determination unit, depending on the shooting conditions, selectively determines as the content of the estimation process a content that includes a first process of detecting the left portion of the estimation image as a predetermined detection location, and a second process of detecting the right portion of the estimation image as a detection location. The estimation process is a process of estimating the vehicle position using the detected location.

[0006] (1) According to one aspect of the present disclosure, a system is provided for estimating a vehicle position representing the position of a vehicle traveling by remote control. This system includes an image acquisition unit that acquires a plurality of captured images of the vehicle, where the plurality of captured images are captured by a plurality of cameras, an evaluation unit that performs an evaluation on a feature amount of the captured image that can affect the detection of the vehicle in the captured image for each of the captured images, and a first determination unit that determines, according to the result of the evaluation, an estimation image used for an estimation process for estimating the vehicle position from each of the captured images. In this configuration, based on the evaluation results of the feature quantities of multiple images captured by multiple cameras, the image to be used for estimation can be determined from each captured image. Therefore, the likelihood of accurately estimating the vehicle's position is increased. (2) In the above configuration, the evaluation unit may calculate a value for each captured image that represents the difference between the color tone feature quantity of the captured image and the color tone feature quantity of a pre-prepared reference image, and output each of the above values ​​as the result of the evaluation. In this configuration, since color tone correlates well with factors that can affect vehicle detection in the captured image, the evaluation of the feature quantity of each captured image can be effectively performed by comparing the color tone feature quantities of the captured image and the reference image. (3) In the above configuration, a second determination unit may be provided that determines the content of the estimation process according to the shooting conditions of the camera that captured the estimation image. In such a configuration, the content of the estimation process can be determined according to the shooting conditions of the camera that captured the estimation image. Therefore, the possibility of estimating the vehicle position with greater accuracy in the estimation process is increased. (4) In the above configuration, the second determination unit may determine the content of the estimation process by determining the program to be used in the estimation process according to the shooting conditions. In this configuration, the content of the estimation process can be determined according to the shooting conditions of the camera that captured the estimation image by determining the program by the second determination unit. (5) In the above configuration, the system may include an estimation unit that performs the estimation process using the estimation image, and a command generation unit that generates a control command for remote control using the vehicle position estimated by the estimation process. In such a configuration, a control command for remote control can be generated using an appropriate vehicle position estimated by the estimation process.

[0007] In addition to the system form described above, this disclosure can also be implemented in other forms, such as a control device, a control method, a computer program for implementing the control method, and a non-temporary recording medium on which the computer program is stored. Furthermore, for example, the above system may perform feature evaluation by inputting captured images into a pre-prepared machine learning model. [Brief explanation of the drawing]

[0008] [Figure 1] This is a conceptual diagram showing the system configuration. [Figure 2] This is a block diagram showing the configuration of the vehicle and the control device. [Figure 3] This is a flowchart of the decision-making process. [Figure 4] This is a diagram illustrating the decision-making process. [Figure 5] This is a flowchart of the estimation process. [Figure 6] This is an explanatory diagram illustrating an example of how images are analyzed during the estimation process. [Modes for carrying out the invention]

[0009] A. First Embodiment: Figure 1 is a conceptual diagram showing the configuration of system 10 in the first embodiment. Figure 2 is a block diagram showing the configuration of vehicle 100 and control device 200. System 10 is configured as a remote control system that drives vehicle 100 by remote control. System 10 comprises one or more vehicles 100, a control device 200 that performs remote control of vehicle 100, a camera group 300 including multiple cameras 301 that capture captured images Pi of vehicle 100, and a process control device 400 that manages the manufacturing process of vehicle 100.

[0010] In this embodiment, vehicle 100 is an electric vehicle (BEV: Battery Electric Vehicle). However, vehicle 100 is not limited to electric vehicles; for example, it may be a gasoline vehicle, a hybrid vehicle, or a fuel cell vehicle.

[0011] In this embodiment, remote control of the vehicle 100 is performed in a factory that manufactures the vehicle 100. As shown in Figure 1, the factory in this embodiment comprises a first location PL1 and a second location PL2. The first location PL1 is, for example, the location where the vehicle 100 is assembled, and the second location PL2 is, for example, the location where the vehicle 100 is inspected. Any location within the factory is represented by xyz coordinate values ​​in a reference coordinate system Σr. The reference coordinate system Σr is defined, for example, as the world coordinate system (also called the global coordinate system).

[0012] The first location PL1 and the second location PL2 are connected by a track SR on which the vehicle 100 can travel. The track SR does not have to be a series of tracks as shown in Figure 1; for example, it may be a track that branches into multiple tracks along the way, or a track on which multiple tracks merge along the way. A target route is set on the track SR on which the vehicle 100 travels by remote control. In this embodiment, the target route includes routes Rt1, Rt2, and Rt3. The target route may be determined according to the type of vehicle 100 (for example, vehicle type, model, grade, and power source), the inspection results of the vehicle 100, the degree of congestion on the track, the degree of congestion in the next process, etc. For example, manufacturing information described later may be used to determine such a target route.

[0013] Multiple cameras 301 are installed around the track SR. Each camera 301 is positioned so that at least one camera 301 can always photograph the vehicle 100 when the vehicle is at any position on the target route. The control device 200 can use the captured images Pi taken by each camera 301 to obtain the relative position and orientation of the vehicle 100 with respect to the target route, as well as the direction of travel of the vehicle 100, in real time. The position, orientation, and direction of travel thus detected are used to generate control commands for remotely controlling the vehicle 100. In this embodiment, each camera 301 is positioned to capture an image of the track SR from above. The position of each camera 301 is fixed, and the relative relationship between the reference coordinate system Σr and the device coordinate system of each camera 301 (hereinafter also referred to as the camera coordinate system) is known. A coordinate transformation matrix for converting between the coordinate values ​​of the reference coordinate system Σr and the coordinate values ​​of the device coordinate system of each camera 301 is stored in advance in the control device 200. Furthermore, in the remote control of the vehicle 100, various sensors such as various on-board cameras mounted on the vehicle 100, LiDAR (Light Detection And Ranging), millimeter-wave radar, ultrasonic sensors, and infrared sensors may be used as auxiliary devices.

[0014] In this embodiment, the control device 200 is configured as a remote control device that generates the control commands described above and transmits them to the vehicle 100. More specifically, the control device 200 generates control commands to cause the vehicle 100 to travel along a target route and transmits the control commands to the vehicle 100. The vehicle 100 travels according to the received control commands. Therefore, the system 10 can remotely move the vehicle 100 from the first location PL1 to the second location PL2 without using transport devices such as cranes or conveyors.

[0015] As shown in FIG. 2, the vehicle 100 includes a vehicle control device 110 for controlling each part of the vehicle 100, an actuator group 120 that is driven under the control of the vehicle control device 110, a communication device 130 for communicating with the control device 200 via wireless communication, and a GNSS (Global Navigation Satellite System) receiver 140 for acquiring the position information of the vehicle 100. In the present embodiment, the actuator group 120 includes an actuator of a driving device for accelerating the vehicle 100, an actuator of a steering device for changing the traveling direction of the vehicle 100, and an actuator of a braking device for decelerating the vehicle 100. The driving device includes a battery, a traveling motor driven by the power of the battery, and driving wheels rotated by the traveling motor. The actuator of the driving device includes the traveling motor. The actuator group 120 may further include an actuator for swinging the wiper of the vehicle 100, an actuator for opening and closing the power window of the vehicle 100, and the like.

[0016] The vehicle control device 110 is constituted by a computer including a processor 111, a memory 112, an input / output interface 113, and an internal bus 114. The processor 111, the memory 112, and the input / output interface 113 are connected so as to be communicable bidirectionally via the internal bus 114. The actuator group 120, the communication device 130, and the GNSS receiver 140 are connected to the input / output interface 113.

[0017] In this embodiment, the processor 111 functions as the vehicle control unit 115 and the position information acquisition unit 116 by executing the program PG1 stored in advance in the memory 112. The vehicle control unit 115 controls the actuator group 120. When a driver is on board the vehicle 100, the vehicle control unit 115 can drive the vehicle 100 by controlling the actuator group 120 according to the driver's operation. Whether or not a driver is on board the vehicle 100, the vehicle control unit 115 can also drive the vehicle 100 by controlling the actuator group 120 according to a control command transmitted from the control device 200. The position information acquisition unit 116 acquires position information indicating the current location of the vehicle 100 using the GNSS receiver 140. However, the position information acquisition unit 116 and the GNSS receiver 140 can be omitted.

[0018] The control device 200 is composed of a computer including a processor 201, a memory 202, an input / output interface 203, and an internal bus 204. The processor 201, the memory 202, and the input / output interface 203 are connected via the internal bus 204 so as to be communicable bidirectionally. A communication device 205 for communicating with the vehicle 100, the camera 301, and the process management device 400 by wireless communication is connected to the input / output interface 203.

[0019] In this embodiment, the processor 201 functions as the image acquisition unit 210, the evaluation unit 220, the first decision unit 230, the second decision unit 240, the estimation unit 250, and the command generation unit 260 by executing the program PG2 stored in advance in the memory 202. In addition to the program PG2, the memory 202 stores correspondence data 271, an estimation program 280, and reference data 275, which will be described later. The estimation program 280 in this embodiment is a comprehensive program including a first analysis program 281 and a second analysis program 282, which will be described later. The reference data 275 includes data related to a reference image, which will be described later.

[0020] The image acquisition unit 210 acquires the captured image Pi from the camera 301. More specifically, as will be described later, the image acquisition unit 210 acquires multiple captured images Pi from each of the multiple cameras 301. The captured image Pi is preferably a color image, but may also be a grayscale image. In this embodiment, the camera 301 captures the captured image Pi so as to show at least one of the first part Dp1 and the second part Dp2 shown in Figure 1. The first part Dp1 is the left part of the vehicle 100 located to the left of the center position in the vehicle width direction, and the second part Dp2 is the right part of the vehicle 100 located to the right of the center position. More specifically, the first part Dp1 is the left rear corner of the vehicle 100, and the second part Dp2 is the right rear corner of the vehicle 100. In other words, in this embodiment, the first part Dp1 and the second part Dp2 are parts that are positioned approximately symmetrically to each other in the vehicle width direction of the vehicle 100. The first part Dp1 and the second part Dp2 correspond to detection areas, which are parts that can be detected as detection points in the estimation process described later. The detection points are the parts that are used in the estimation process to estimate the vehicle position, which represents the position of the vehicle 100.

[0021] In the system 10 of this embodiment, cameras 301A and 301B shown in Figure 1 photograph the vehicle 100 traveling on route Rt1. Cameras 301C and 301D photograph the vehicle 100 traveling on route Rt2. Cameras 301E and 301F photograph the vehicle 100 traveling on route Rt3. In addition, cameras 301A, 301C and 301E photograph the vehicle 100 traveling on track SR from the left rear side. Cameras 301B, 301D and 301F photograph the vehicle 100 traveling on track SR from the right rear side. In this embodiment, "photographing the vehicle 100 from the left rear side" means photographing the vehicle 100 from the left rear side such that the first portion Dp1 (left rear corner) is visible in the captured image Pi. "Photographing vehicle 100 from the right rear" means photographing vehicle 100 from the right rear so that the second portion Dp2 (right rear corner) is visible in the captured image Pi. Therefore, an image Pi taken of vehicle 100 from the left rear usually includes the first portion Dp1 regardless of the vehicle's position. On the other hand, this captured image Pi may or may not include the second portion Dp2 depending on the vehicle's position. Also, if an image Pi taken of vehicle 100 from the left rear includes both the first portion Dp1 and the second portion Dp2, the second portion Dp2 will appear smaller than the first portion Dp1 in that image Pi. Similarly, an image Pi taken of vehicle 100 from the right rear usually includes the second portion Dp2 regardless of the vehicle's position. On the other hand, this captured image Pi may or may not include the first part Dp1, depending on the position where the vehicle 100 is traveling.

[0022] The evaluation unit 220 shown in Figure 2 performs image evaluation for each captured image Pi acquired by the image acquisition unit 210. The image evaluation is an evaluation of the feature quantities of the captured image Pi. These feature quantities are those that can affect the detection of the vehicle 100 in the captured image Pi and reflect the influencing factors in the captured image Pi. The influencing factors include disturbance factors related to foreign objects, reflected light, and shadows that appear in the captured image Pi, as well as internal factors that are different from disturbance factors. Internal factors include, for example, abnormalities in the camera 301, or delays or abnormalities in communication between the camera 301 and the control device 200, which cause noise and degradation of image quality in the captured image Pi. Foreign objects that appear in the captured image Pi include, for example, leaves or dust attached to the lens of the camera 301 that captures the captured image Pi, or puddles on the track SR, or images of the vehicle 100 reflected in puddles. Such influencing factors can affect the detection of the vehicle 100 in the captured image Pi. For example, the disturbance factor may appear in the captured image Pi in a way that overlaps with the vehicle 100, or it may appear in the captured image Pi in a way that could lead to it being mistakenly detected as the vehicle 100.

[0023] In the image evaluation of this embodiment, features related to the color tone of the captured image Pi are evaluated. "Evaluation of features related to the color tone of the captured image Pi" refers to the evaluation of features related to at least one of the brightness, saturation, and hue of the captured image Pi. For example, the luminance of the captured image Pi corresponds to features related to brightness and hue. When the captured image Pi contains the above-mentioned disturbance factors or internal factors, the brightness, saturation, and hue of the captured image Pi change compared to when the disturbance factors or internal factors are not present. For example, the brightness and luminance of areas in the captured image Pi where shadows or foreign objects attached to the lens of the camera 301 are reflected tend to be lower compared to the brightness and luminance of the same area when shadows or foreign objects are not reflected. Also, when relatively high-intensity light such as sunlight or illumination is reflected in the captured image Pi, the brightness and luminance in the captured image Pi are higher locally or overall compared to when such light is not reflected in the captured image Pi. Furthermore, for example, the color tone of a region in the captured image Pi that contains noise due to internal factors will differ from the color tone of the same region when it is not noisy. Thus, the color tone features of the captured image Pi correlate well with the influencing factors in the captured image Pi and influence the detection of vehicle 100 in the captured image Pi. In particular, the color tone features of the captured image Pi correlate even better with the disturbance factors in the captured image Pi. In image evaluation, these features that correlate with influencing factors can be used as the features to be evaluated.

[0024] In this embodiment, the evaluation unit 220 performs image evaluation by comparing the color tone feature quantities of the captured image Pi with the color tone feature quantities of a pre-prepared reference image for each captured image Pi. More specifically, in the image evaluation, the evaluation unit 220 calculates a difference value representing the difference in color tone feature quantities between the captured image Pi and the reference image, and outputs the calculated difference value as the image evaluation result. More specifically, in the image evaluation, the evaluation unit 220 calculates, for example, color tone statistics and frequency distributions of the captured image Pi and the reference image, and outputs a value representing the difference between the statistics of both images and the difference between the frequency distributions as a difference value. A value representing the difference between frequency distributions is, for example, the sum of the absolute values ​​or the sum of the squared values ​​of the differences between the frequency distributions. Hereinafter, a value output as the result of the image evaluation, such as this difference value, will also be called an evaluation score. In this embodiment, the evaluation unit 220 calculates a difference value as this difference value, which represents the difference between the brightness in the captured image Pi and the brightness in the reference image.

[0025] As a reference image, for example, an image of the scene corresponding to the captured image Pi, taken under ideal conditions where the influence of disturbance factors is below a predetermined level, is used. In this case, the reference image is preferably an image in which the track SR is taken in the same orientation as the captured image Pi, and more preferably an image that includes a vehicle traveling on this track SR. Furthermore, if the reference image includes a vehicle, it is preferable that the vehicle included in the reference image is a vehicle that may be included in the captured image Pi, that is, a vehicle having the same shape and body color as the vehicle 100 to be remotely controlled. The reference image may also be created by simulation.

[0026] The evaluation unit 220 can, for example, use known algorithms for detecting color tone features of an image, a pre-built rule base for detecting color tone features of an input image, or various machine learning models that have been pre-trained to detect color tone features of an input image. Such machine learning models can include convolutional neural networks (hereinafter also called CNNs), deep neural networks (hereinafter also called DNNs), support vector machines, decision trees, etc. The reference data 275 mentioned above may include, for example, pre-prepared reference images, or pre-calculated color tone statistics and frequency distributions of reference images.

[0027] The first determination unit 230 determines an estimation image from each captured image Pi, which is the captured image Pi used for estimation processing, according to the evaluation result by the evaluation unit 220. In this embodiment, the first determination unit 230 determines the captured image Pi with the better image evaluation result as the estimation image. In this embodiment, "better image evaluation result" means that the influence of the influencing factors in the captured image Pi is smaller, and in this embodiment, it means a smaller evaluation score. This is because the smaller the influence of external and internal factors in the captured image Pi, the smaller the difference value tends to be. In other embodiments, for example, the evaluation score may be defined to be larger when the image evaluation result is better.

[0028] The estimation unit 250 shown in Figure 2 performs estimation processing. Estimation processing involves detecting the remotely controlled vehicle 100 in the estimation image and detecting the vehicle's position using the detection results. In this embodiment, estimation processing is a process of estimating the vehicle's position using detected locations in the estimation image, and includes an analysis process that detects detected locations in the estimation image by analyzing the estimation image, and a process that estimates the vehicle's position using the detected locations. In this embodiment, estimation processing is performed by the estimation unit 250 executing the estimation program 280. The estimation unit 250 can estimate the position and orientation of the vehicle 100 using, for example, the vehicle's driving history or position information detected by the GNSS receiver 140. The direction of travel of the vehicle 100 can be estimated using the vehicle's orientation, driving history, control command history, etc.

[0029] In this embodiment, the estimation unit 250 includes an analysis unit 251 that performs analysis processing. In this embodiment, the analysis unit 251 performs analysis processing by executing a first analysis program 281 and a second analysis program 282, which are analysis programs. An analysis program is a program that enables a computer executing the analysis program to analyze the captured image Pi and detect detection locations in the captured image Pi. In this embodiment, the first analysis program 281 is an analysis program for analyzing a captured image Pi taken from the left rear side of the vehicle 100, and is used to detect a first part Dp1 as a detection location. The second analysis program 282 is an analysis program for analyzing a captured image Pi taken from the right rear side of the vehicle 100, and is used to detect a second part Dp2 as a detection location.

[0030] The second determination unit 240 determines the content of the estimation process according to the shooting conditions of the camera 301 that captured the estimation image. Hereinafter, the camera 301 that captured the estimation image will also be referred to as the target camera. In this embodiment, the second determination unit 240 identifies the target camera and determines the program used in the estimation process according to the camera information Ci of the target camera, thereby changing the content of the estimation process according to the shooting conditions. More specifically, the second determination unit 240 acquires the camera information Ci of the target camera and determines the detection program by referring to the corresponding data 271 using the acquired camera information Ci. The detection program is an analysis program used to detect detection locations in the analysis process.

[0031] In this specification, "determining a program according to shooting conditions" refers to determining the processing steps of a program according to the shooting conditions. Therefore, "determining a program according to shooting conditions" includes, for example, selecting one analysis program from a plurality of pre-prepared programs as the program to be determined, according to the shooting conditions. In this case, the selection of the detection program may be achieved, for example, by executing conditional branching according to the shooting conditions in a comprehensive program that includes multiple programs. Furthermore, "determining a program according to shooting conditions" also includes, for example, preparing the program to be determined by rewriting some or all of the code included in the program according to the shooting conditions.

[0032] In this embodiment, the position and orientation of each camera 301 are fixed with respect to the track SR. Therefore, identifying a target camera is equivalent to identifying the shooting conditions for the image Pi captured by the target camera. These shooting conditions are conditions relating to detection parts that may be included in the captured image Pi, and include, for example, conditions relating to parts of the vehicle 100 that may be included as detection parts in the captured image Pi, and conditions relating to the orientation in which the captured image Pi is taken. The conditions relating to the orientation in which the captured image Pi is taken relate to the orientation in which the vehicle 100 is photographed when the captured image Pi is taken, and include, for example, conditions that determine whether the vehicle 100 is photographed from the right or left side of the vehicle 100, or whether the vehicle 100 is photographed from the front or rear side of the vehicle 100. In this embodiment, if the orientation in which the captured image Pi is taken is identified, the detection parts that may be included in the captured image Pi are identified. Hereinafter, the shooting conditions for the image Pi captured by camera 301 will also be simply referred to as "shooting conditions for camera 301".

[0033] Correspondence data 271 is data that records the correspondence between camera information Ci and information for determining the content of the estimation process. In this embodiment, correspondence data 271 is data that records the correspondence between the identification information of each camera 301 and information for determining the program used in the estimation process. In this embodiment, the identification information of cameras 301A, 301C, and 301E is associated with parameters for selecting the first analysis program 281. In addition, the identification information of cameras 301B, 301D, and 301F is associated with parameters for selecting the second analysis program 282. In other embodiments, correspondence data 271 may be, for example, data that records the correspondence between the identification information of each camera 301, the shooting conditions of each camera 301, and information for determining the content of the estimation process.

[0034] In this embodiment, each camera 301 is configured to transmit its own identification information as camera information Ci to the control device 200 along with the captured image Pi. This identification information may be used to identify the target camera. In other embodiments, the second determination unit 240 may obtain the camera information Ci from a database pre-stored in the memory 202, or from an external computer or recording medium. In other embodiments, information different from the identification information of the camera 301 may be used as camera information Ci; for example, other information that can identify the shooting conditions may be used.

[0035] The command generation unit 260 generates a control command for remote control using the position and orientation of the vehicle 100 estimated by the estimation unit 250 and transmits it to the vehicle 100. This control command is a command to drive the vehicle 100 according to a target route stored in the memory 202. The control command can be generated as a command including driving force or braking force and steering angle. Alternatively, the control command may be generated as a command including at least one of the position and orientation of the vehicle 100 and the future driving route.

[0036] The process control device 400, for example, is configured by a computer and manages the entire manufacturing process of the vehicle 100 in the factory. For example, when a vehicle 100 starts traveling along a target route, individual information such as an identification number and model number that identifies the vehicle 100 is transmitted from the process control device 400 to the control device 200. This individual information corresponds to manufacturing information used to manage the manufacturing process of the vehicle 100. The position of the vehicle 100 detected by the control device 200 is also transmitted to the process control device 400. The functions of the process control device 400 may also be implemented in the same device as the control device 200.

[0037] Figure 3 is a flowchart of the decision process in this embodiment. This decision process is a process for realizing the control method in this embodiment. The decision process is executed by the processor 201 of the control device 200, for example, at predetermined time intervals.

[0038] In step S110 of Figure 3, the image acquisition unit 210 acquires multiple captured images Pi. Each captured image Pi acquired in S110 in this embodiment is a captured image Pi taken by a different camera 301. In step S120, the evaluation unit 220 performs an image evaluation for each captured image Pi acquired in S110. In step S130, the first determination unit 230 determines an estimation image from each captured image Pi acquired in S110, according to the evaluation result in S120. In step S140, the second determination unit 240 determines a detection program according to the shooting conditions of the target camera, that is, the camera 301 that took the captured image Pi determined as the estimation image in S130. In step S140 of this embodiment, the content of the estimation process is determined according to the shooting conditions by determining the detection program in this way. More specifically, in step S140, the content of the analysis process is determined. Each captured image Pi acquired in S110 may be an image taken at the same time, or it may be an image taken at different times. In other embodiments, the multiple captured images Pi acquired in S110 may include two or more captured images Pi taken by the same camera 301.

[0039] Figure 4 is a diagram illustrating the decision process. Figure 4 shows a vehicle 100 traveling on route Rt1 on track SR, as captured by cameras 301A and 301B. In this case, at S110 in Figure 3, captured image PiA taken by camera 301A and captured image PiB taken by camera 301B are acquired. At S120, image evaluation is performed on captured image PiA and captured image PiB, and the evaluation score EA for captured image PiA and the evaluation score EB for captured image PiB are calculated. In the example in Figure 4, since the evaluation score EA is lower than the evaluation score EB, at S130, captured image PiA is determined as the image for estimation. In this case, at S140, the first analysis program 281 is determined as the detection program according to the shooting conditions of camera 301A. Conversely, if the evaluation score EB is lower than the evaluation score EA, in S130 the captured image PiB is determined as the estimation image, and in S140 the second analysis program 282 is determined as the detection program according to the shooting conditions of camera 301B. In other embodiments, in S110 of Figure 3, the image acquisition unit 210 may acquire each captured image Pi taken by each of the three or more cameras 301.

[0040] Figure 5 is a flowchart of the estimation process in this embodiment. In this embodiment, the estimation process is performed each time an estimation image is transmitted from the assigned camera to the control device 200.

[0041] In S210 of Figure 5, the estimation unit 250 determines, based on the result of S140 in Figure 3, whether or not to use the first analysis program 281 for the analysis of the estimation image determined in S130. If it is determined in S210 to use the first analysis program 281, the analysis unit 251 executes the first process implemented by the first analysis program 281 from S220 to S230. The first process is for detecting the first part Dp1 as a detection location. If it is not determined in S210 to use the first analysis program 281, the analysis unit 251 executes the second process implemented by the second analysis program 282 from S215 to S230. The second process is for detecting the second part Dp2 as a detection location. In the following, S220 implemented by the first analysis program 281 will also be referred to as S220A, and S220 implemented by the second analysis program 282 will also be referred to as S220B. The same applies to S225 and S230.

[0042] Unlike the first process, the second process in this embodiment includes the inversion process in S215. The inversion process generates an inverted image by mirror-reversing the estimation image. As a result, the estimation image that has not been mirror-reversed is analyzed in S220A to S230A, while the inverted image is analyzed in S220B to S230B. This process of mirror-reversing an image is also called "flipping". The first and second processes are substantially the same except that the second process includes the inversion process. Therefore, in S220B to S230B, it is possible to analyze the inverted image using the same algorithm as in S220A to S230A, that is, using the same procedure. More specifically, when the first part Dp1 is used as the detection location, in S220A to S230A, the first part Dp1 is detected as the left rear corner of the estimation image that has not been mirror-reversed. On the other hand, when the second portion Dp2 is used as the detection location, the second portion Dp2, which is actually the right rear corner, is detected as the left rear corner in the inverted image from S220B to S230B. In other words, in this embodiment, in the inverted image, the second portion Dp2 is detected as the portion corresponding to the first portion Dp1 in the estimation image that has not been mirror-reversed.

[0043] Figure 6 is an explanatory diagram illustrating an example of how image Im1 is analyzed in the estimation process. Image Im1 shown in Figure 6 is an image of a vehicle 100 traveling on a track SR. Image Im1 has mutually orthogonal Xc and Yc axes as coordinate axes and is represented in a camera coordinate system with the focal point of camera 301 as the origin. Image Im1 may be an inverted image or an estimation image that has not been mirrored. Various corrections and preprocessing may be performed on image Im1, such as distortion correction processing to correct distortion, rotation processing to rotate image Im1 so that the direction of the vehicle 100's movement vector V in image Im1 points in a predetermined direction, and cropping processing to remove unnecessary areas that do not contain the vehicle 100 from image Im1. For example, the optical flow method is used to detect the magnitude and direction of the movement vector V.

[0044] In steps S220 to S230 of Figure 5, the analysis unit 251 performs analysis processing. In the analysis processing in this embodiment, the analysis unit 251 detects the positioning point PP shown in Figure 6 by calculating the coordinates of the positioning point PP using the image Im1. The coordinates of the positioning point PP are calculated as the coordinates of the vicinity of the detected location in the image coordinate system described later. The positioning point PP detected in this way represents the detected location in the estimation image.

[0045] In S220, the analysis unit 251 performs a masking process as a detection process to detect the vehicle 100 in image Im1. The masking process detects the vehicle 100 in image Im1 and masks the target region containing the vehicle 100 in image Im1, thereby generating image Im2 which includes the masked region Ms. In the masking process, the estimation unit 250 generates image Im2 by inputting image Im1 to a machine learning model (not shown) that has been trained to mask the vehicle 100 contained in the input image. As this machine learning model, for example, a DNN having the structure of a CNN that realizes semantic segmentation or instance segmentation can be used. Note that the machine learning model may be trained using algorithms other than neural networks, for example.

[0046] In S225, the analysis unit 251 performs a perspective transformation process. The perspective transformation process generates image Im3 by performing a perspective transformation on image Im2. In the perspective transformation process, the analysis unit 251 uses, for example, the position information of camera 301 and perspective transformation parameters related to internal parameters to perform a perspective transformation on image Im2 into a bird's-eye view image viewed from above (for example, directly above vehicle 100) which is approximately perpendicular to the road surface of track SR. Image Im3 is represented in an image coordinate system. The image coordinate system is a coordinate system that has a point on the image plane projected by the perspective transformation as its origin, and mutually orthogonal Xi axes and Yi axes as its coordinate axes. Image Im3 includes a mask region Msb which corresponds to the mask region Ms that has been transformed by the perspective transformation.

[0047] In S230, the analysis unit 251 performs a positioning point calculation process to calculate the coordinates of the positioning point PP. In the positioning point calculation process in this embodiment, the analysis unit 251 calculates the coordinates of the positioning point PP using the coordinates (Xi1, Yi1) of the first coordinate point P1 and the coordinates (Xi2, Yi2) of the second coordinate point P2.

[0048] The first coordinate point P1 is identified as the vertex of quadrilateral R1b in image Im2 that corresponds to the base coordinate point P0 in image Im1. Quadrilateral R1b corresponds to the first bounding rectangle R1, which is deformed by perspective transformation. The first bounding rectangle R1 is a rectangle in image Im2 that is bounding to the mask region Ms and has a longer side parallel to the movement vector V. The base coordinate point P0 is the vertex of this first bounding rectangle R1 that corresponds to the detection location of vehicle 100. More specifically, the base coordinate point P0 is the vertex of the first bounding rectangle R1 that is located to the left and behind the center of gravity C of vehicle 100 when the direction of the movement vector V is forward. The second coordinate point P2 is the vertex of the second bounding rectangle R2 in image Im2 that corresponds to the detection location of vehicle 100, among the four vertices of the second bounding rectangle R2. More specifically, the second coordinate point P2 represents the coordinate of a vertex of the second circumscribed rectangle R2 that is located to the left and behind the centroid C when the direction of the movement vector V is considered to be forward. The second circumscribed rectangle R2 is a rectangle that is circumscribed around the mask region Msb and has sides parallel to the Xi axis and sides parallel to the Yi axis. Thus, the first coordinate point P1 and the second coordinate point P2 are both coordinate points determined according to the detection location, and therefore have a correlation with each other.

[0049] If the coordinate value Xi1 is greater than the coordinate value Xi2, the estimation unit 250 determines the Xi coordinate value of the positioning point PP to be the coordinate value Xi1. Conversely, if the coordinate value Xi1 is less than the coordinate value Xi2, the estimation unit 250 determines the Xi coordinate value to be the coordinate value Xi2. Similarly, if the coordinate value Yi1 is greater than the coordinate value Yi2, the Yi coordinate value of the positioning point PP is determined to be the coordinate value Yi1. If the coordinate value Yi1 is less than the coordinate value Yi2, the Yi coordinate value is determined to be the coordinate value Yi2. By determining the Xi and Yi coordinate values ​​in this way, the positioning point PP is detected.

[0050] In S235, the estimation unit 250 performs a vehicle coordinate calculation process. The vehicle coordinate calculation process calculates vehicle coordinate points using positioning point PP. Vehicle coordinate points are coordinate points that represent positioning point PP in the reference coordinate system Σr. In S235, the estimation unit 250 calculates vehicle coordinate points using, for example, a predetermined estimation formula and the coordinate values ​​of positioning point PP calculated in S230. The command generation unit 260 generates a control command using the vehicle coordinate points calculated in the estimation process as the vehicle position. The control command thus generated is transmitted to the vehicle 100. In other words, in this embodiment, calculating vehicle coordinate points is equivalent to estimating the vehicle position. Note that the method for estimating the vehicle position using the estimation image is not limited to the above. For example, the vehicle 100 in the estimation image may be detected by a process different from masking, the detection location in the estimation image may be detected by a process different from perspective transformation processing or positioning point calculation processing, or coordinate points representing the vehicle position may be calculated by a process different from the vehicle coordinate calculation process.

[0051] According to the system 10 of this embodiment described above, the first determination unit 230 determines an estimation image to be used for estimation processing from each captured image Pi, according to the result of the image evaluation for each captured image Pi acquired by the image acquisition unit 210. Therefore, compared to, for example, a configuration in which only one captured image Pi is acquired by the image acquisition unit 210 for estimation processing, or a configuration in which estimation processing is performed without image evaluation, the likelihood of appropriately estimating the vehicle position for the remotely controlled vehicle 100 is increased.

[0052] Furthermore, in this embodiment, the evaluation unit 220 calculates a difference value for each captured image Pi, representing the difference between the color tone feature quantity of the captured image Pi and the color tone feature quantity of the reference image, and outputs each difference value as the result of the image evaluation. Therefore, image evaluation can be performed effectively. In addition, in this embodiment, by determining the captured image Pi with the smaller difference value as the estimation image, an appropriate estimation image can be easily determined.

[0053] Furthermore, in this embodiment, the second determination unit 240 determines the content of the estimation process according to the shooting conditions of the target camera. Therefore, even if the shooting conditions differ for each camera 301, for example, the likelihood of appropriately estimating the vehicle position using the estimation image is increased, regardless of the shooting conditions. Also, since a control command is generated using the vehicle position estimated by the estimation process, the likelihood of generating an appropriate control command is increased. Therefore, the likelihood of appropriately remotely controlling the vehicle 100 is increased.

[0054] Furthermore, in this embodiment, the second determination unit 240 determines the program to be used in the estimation process according to the shooting conditions of the target camera. Therefore, by determining the program using the second determination unit 240, the content of the estimation process can be determined according to the shooting conditions of the target camera.

[0055] B. Other embodiments: (B1) In the above embodiment, the evaluation unit 220 may perform image evaluation that takes into account the shooting timing, such as the time of day, time of day, and season, when each captured image Pi was taken. In this case, the evaluation unit 220 can predict the effect of sunlight and its shadows on each captured image Pi based on the shooting timing when each captured image Pi was taken and the position and orientation of each camera 301. The evaluation unit 220 may correct the evaluation score of a captured image Pi so as to reduce the quality of the evaluation result for a captured image Pi that is predicted to be more affected by sunlight and its shadows.

[0056] (B2) In the above embodiment, the difference value is a value representing the brightness difference between the captured image Pi and the reference image. In contrast, the difference value may be, for example, a value representing the brightness difference between the two images, a value representing the hue difference, or a value representing the saturation difference. Alternatively, the difference value may represent two or more of the differences in brightness, brightness, hue, and saturation. In this case, the difference value may be defined as an index value that takes a larger value the greater the brightness difference, brightness difference, hue difference, or saturation difference, or a smaller value. Furthermore, although the evaluation unit 220 performs an evaluation of feature quantities that represent color tone in the image evaluation, this is not required. For example, it may perform an evaluation of various feature quantities that can affect the detection of the vehicle 100 in the captured image Pi, such as an evaluation of blobs, an evaluation of edges, or an evaluation of corners. For example, when the evaluation unit 220 performs an evaluation of edges, it may perform the image evaluation by comparing the angle, coordinates, spacing, and intensity of edges in the captured image Pi and the reference image. In this case, the evaluation unit 220 may also output an evaluation score representing the difference in edge angle, coordinates, spacing, and intensity between the captured image Pi and the reference image. For such edge evaluation, for example, a known edge detection algorithm, a rule-based algorithm built to detect edges in the input image, or a machine learning model pre-trained to detect edges in the input image can be used.

[0057] (B3) In the above embodiment, the evaluation unit 220 compares the features of the captured image Pi with those of a reference image in the image evaluation, but this is not required. For example, the evaluation unit 220 may perform an evaluation of the features of the captured image Pi without comparing the features of the captured image Pi with those of a reference image by inputting the captured image Pi into a machine learning model that has been pre-trained to detect disturbance factors and intrinsic factors of the input image. In this case, the evaluation score can be, for example, the number of different types of influencing factors included in the captured image Pi, the area of ​​the portion of the captured image Pi attributable to the influencing factors, or the degree to which the influencing factors have an impact on the detection of the vehicle 100.

[0058] (B4) In the above embodiment, the camera information Ci may be information representing the shooting conditions of the camera 301, for example, information regarding the detection area included in the captured image Pi taken by the camera 301 may be used. In this case, the camera information Ci may be information representing the orientation in which the camera 301 photographs the vehicle 100, or information representing the detection area that the camera 301 can photograph. By using such camera information Ci, for example, even if the position and orientation of each camera 301 are not fixed with respect to the track SR, the content of the estimation process can be determined according to the shooting conditions of the target camera.

[0059] (B5) In the above embodiment, the second determination unit 240 determines the program used in the estimation process according to the shooting conditions of the target camera, but this is not required. For example, the same program may be used in the estimation process even if the shooting conditions are different. In this case, the second determination unit 240 may determine the content of the estimation process according to the shooting conditions of the target camera by determining the parameters used in the estimation process according to the shooting conditions of the target camera. Alternatively, the content of the estimation process may not be determined according to the shooting conditions of the target camera. In this case, the estimation unit 250 may, for example, estimate the vehicle position by inputting the estimation images into the same machine learning model regardless of the shooting conditions. Also, in this case, the system 10 does not need to include the second determination unit 240.

[0060] (B6) In the above embodiment, the first and second processes include inversion processing, but inversion processing may not be included. In this case, the estimation unit 250 may detect the first part Dp1 and the second part Dp2 in the estimation image that has not been mirror-reversed, regardless of whether it detects the first part Dp1 or the second part Dp2 as the detection location.

[0061] (B7) In the above embodiment, the first part Dp1 and the second part Dp2 do not have to be symmetrically positioned in the vehicle width direction. For example, the left rear corner and the right front corner of the vehicle 100 may be used as the first part Dp1 and the second part Dp2, respectively. Also, the left side and right side of the vehicle 100 do not have to be used as detection points. Furthermore, the number of parts used as detection points may be three or more, or it may be just one.

[0062] (B8) In the above embodiment, instead of determining the captured image Pi with the better image evaluation result as the estimation image, or in addition to this, the first determination unit 230 may determine the captured image Pi with an image evaluation result that is better than a predetermined standard as the estimation image. In this case, if the image evaluation results of two or more captured images Pi are better than the standard, the first determination unit 230 may, for example, randomly select one captured image Pi from these captured images Pi and determine it as the estimation image, or it may determine the captured image Pi that is predicted to have less influence from light and shadow based on the shooting timing described above as the estimation image. Also, if the image evaluation result of each captured image Pi is below the standard, for example, the image acquisition unit 210 may acquire a new captured image Pi from the camera 301. Doing so increases the possibility of determining the estimation image more appropriately. Also, if the image evaluation result of each captured image Pi is below the standard, the command generation unit 260 may generate a control command to brake the vehicle 100. In this way, if the accuracy of estimating the vehicle position of vehicle 100 may decrease, the vehicle 100 can be braked.

[0063] (B9) In the above embodiment, system 10 is configured as a remote control system comprising a vehicle 100, a camera 301, an estimation unit 250, and a command generation unit 260, but it does not have to be configured in this way. For example, system 10 may be configured as a system that determines an estimation image and transmits the determination result to another system comprising a vehicle 100, a camera 301, an estimation unit 250, and a command generation unit 260. In this case, system 10 does not have to comprise some or all of the vehicle 100, camera 301, estimation unit 250, and command generation unit 260. Also in this case, system 10 may be configured to transmit the estimation image to the other system along with the determination result. In this case, the estimation unit 250 provided in the other system may acquire the estimation image transmitted from system 10 during the estimation process.

[0064] (B10) In the above embodiment, the vehicle 100 only needs to have a configuration that allows it to be moved by remote control, and may be in the form of a platform having the configuration described below. Specifically, the vehicle 100 only needs to have at least a vehicle control unit 115 and a communication device 130 in order to perform the three functions of "driving," "turning," and "stopping" by remote control. That is, the vehicle 100 that can be moved by remote control does not need to have at least some of the interior parts such as the driver's seat and dashboard attached, at least some of the exterior parts such as the bumper and fenders attached, and does not need to have a body shell attached. In this case, the remaining parts such as the body shell may be attached to the vehicle 100 before it is shipped from the factory, or the remaining parts such as the body shell may be attached to the vehicle 100 after it has been shipped from the factory without the remaining parts such as the body shell attached to it. The position of the platform can also be determined in the same way as the vehicle 100 in each embodiment.

[0065] This disclosure is not limited to the embodiments described above, and can be implemented in various configurations without departing from its spirit. For example, the technical features in the embodiments corresponding to the technical features in each form described in the summary of the invention can be replaced or combined as appropriate in order to solve some or all of the above-described problems, or to achieve some or all of the above-described effects. Furthermore, if a technical feature is not described as essential in this specification, it can be deleted as appropriate. [Explanation of Symbols]

[0066] 10...System, 100...Vehicle, 110...Vehicle control device, 111...Processor, 112...Memory, 113...Input / Output Interface, 114...Internal Bus, 115...Vehicle Control Unit, 116...Location Information Acquisition Unit, 120...Actuator Group, 130...Communication Device, 140...GNSS Receiver, 200...Control Device, 201...Processor, 202...Memory, 203...Input / Output Interface, 204...Internal Bus, 205...Communication Device, 210...Image Acquisition Unit 220...Evaluation unit, 230...First decision unit, 240...Second decision unit, 250...Estimation unit, 251...Analysis unit, 260...Command generation unit, 271...Corresponding data, 275...Reference data, 280...Estimation program, 281...First analysis program, 282...Second analysis program, 300...Camera group, 301...Camera, 301A...Camera, 301B...Camera, 301C...Camera, 301D...Camera, 301E...Camera, 301F...Camera, 400...Process control device

Claims

1. A system used to estimate the vehicle position of a vehicle that is driven by remote control, An image acquisition unit that acquires multiple images taken by the vehicle, wherein the multiple images are taken by multiple cameras, and the image acquisition unit and the vehicle An evaluation unit performs an evaluation of the feature quantities of the captured image that may affect the detection of the vehicle in the captured image, for each captured image. A first determination unit determines, in accordance with the results of the evaluation, an estimation image to be used in the estimation process for estimating the vehicle position from each of the captured images, The system includes a second determination unit that determines the content of the estimation process according to the shooting conditions of the camera that captured the estimation image, specifically the shooting conditions relating to whether the vehicle is photographed from the left or right side by the camera, The aforementioned estimation image includes at least one of the left portion of the vehicle and the right portion of the vehicle. The second determination unit, depending on the shooting conditions, selectively determines the content of the estimation process to be either a first process that includes detecting the left portion of the estimation image as a predetermined detection location, or a second process that includes detecting the right portion of the estimation image as a detection location. The estimation process is a process of estimating the vehicle position using the detected location, wherein the estimation process is performed by the system.

2. The system according to claim 1, The evaluation unit described above, In the evaluation described above, a value representing the difference between the color tone feature quantity of the captured image and the color tone feature quantity of a pre-prepared reference image is calculated for each captured image. A system that outputs each of the aforementioned values ​​as a result of the evaluation.

3. The system according to claim 1, The second determination unit determines the content of the estimation process by selecting, according to the shooting conditions, the program used in the estimation process to be either a first analysis program for executing the first process or a second analysis program for executing the second process, thereby determining the content of the estimation process.

4. A system according to any one of claims 1 to 3, An estimation unit that performs the estimation process using the estimation image, A system comprising: a command generation unit that generates a control command for remote control using the vehicle position estimated by the estimation process.

Images