System

Abstract

To appropriately execute estimation of a vehicle position using a captured image of the vehicle.SOLUTION: A system for use in estimating a vehicle position for a vehicle that travels by remote control includes: an image acquisition unit which acquires a plurality of captured images of the vehicle, the captured images being captured by a plurality of cameras; an evaluation unit which executes, for each captured image, evaluation on features of the captured images that may affect detecting the vehicle in the captured images; and a first determination unit which determines an estimation image for use in estimation processing for estimating a vehicle position from the captured images, in accordance with results of the evaluation.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to a system. [Background technology]

[0002] Patent Document 1 discloses a technique for running a vehicle by remote control during the vehicle manufacturing process. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2017-538619 Summary of the Invention [Problem to be solved by the invention]

[0004] In remote control of a vehicle, a technique for estimating the vehicle position using an image captured of the vehicle can be used. A technique for appropriately estimating the vehicle position is desired. [Means for solving the problem]

[0005] The present disclosure can be realized in the following forms.

[0006] (1) According to one aspect of the present disclosure, there is provided a system for estimating a vehicle position representing a position of a vehicle traveling by remote control, the system including: an image acquisition unit that acquires a plurality of captured images of the vehicle, the plurality of captured images being captured by a plurality of cameras; an evaluation unit that performs, for each captured image, an evaluation of a feature amount of the captured image that may affect detection of the vehicle in the captured image; and a first determination unit that determines, from each captured image according to a result of the evaluation, an estimation image to be used in an estimation process for estimating the vehicle position. In this embodiment, an estimation image to be used in the estimation process can be determined from the captured images based on the evaluation results of the feature amounts of the multiple captured images captured by the multiple cameras, thereby increasing the possibility of appropriately estimating the vehicle position. (2) In the above embodiment, the evaluation unit may calculate a value representing a difference between a feature amount related to the color tone of the captured image and a feature amount related to the color tone of a reference image prepared in advance, for each captured image, and output each of the values ​​as a result of the evaluation. In this embodiment, since the color tone is highly correlated with factors that may affect the detection of a vehicle in a captured image, the feature amount of each captured image can be effectively evaluated by comparing the feature amounts related to the color tone of the captured image and the reference image. (3) In the above embodiment, a second determination unit may be provided that determines the content of the estimation process according to the image capturing conditions of the camera that captured the estimation image. In this embodiment, the content of the estimation process can be determined according to the image capturing conditions of the camera that captured the estimation image. This increases the possibility that the vehicle position can be estimated with higher accuracy in the estimation process. (4) In the above embodiment, the second determination unit may determine the content of the estimation process by determining a program to be used in the estimation process in accordance with the shooting conditions. In this embodiment, by having the second determination unit determine the program, the content of the estimation process can be determined in accordance with the shooting conditions of a camera that captured the estimation image. (5) In the above embodiment, an estimation unit that executes the estimation process using the estimation image and a command generation unit that generates a control command for the remote control by using the vehicle position estimated by the estimation process. With this embodiment, a control command for remote control can be generated by using an appropriate vehicle position estimated by the estimation process.

[0007] In addition to the above-described system form, the present disclosure can be realized in the form of, for example, a control device, a control method, a computer program for implementing the control method, a non-transitory recording medium on which the computer program is recorded, etc. Furthermore, for example, the above-described system may execute evaluation regarding feature quantities by inputting a captured image into a machine learning model prepared in advance. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a conceptual diagram showing a system configuration. [Diagram 2] 2 is a block diagram showing the configuration of a vehicle and a control device. FIG. [Diagram 3] 13 is a flowchart of a determination process. [Figure 4] FIG. 11 is a diagram illustrating a determination process. [Diagram 5] 13 is a flowchart of an estimation process. [Figure 6] FIG. 11 is an explanatory diagram showing an example of how an image is analyzed in the estimation process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] A. First embodiment: Fig. 1 is a conceptual diagram showing the configuration of a system 10 in the first embodiment. Fig. 2 is a block diagram showing the configuration of a vehicle 100 and a control device 200. The system 10 is configured as a remote control system that causes the vehicle 100 to run by remote control. The system 10 includes one or more vehicles 100, a control device 200 that executes remote control of the vehicle 100, a camera group 300 including a plurality of cameras 301 that capture images Pi captured by the vehicle 100, and a process management device 400 that manages the manufacturing process of the vehicle 100.

[0010] The vehicle 100 in this embodiment is a battery electric vehicle (BEV). Note that the vehicle 100 is not limited to an electric vehicle, and may be, for example, a gasoline vehicle, a hybrid vehicle, or a fuel cell vehicle.

[0011] In this embodiment, remote control of the vehicle 100 is performed in a factory where the vehicle 100 is manufactured. As shown in FIG. 1, the factory in this embodiment includes a first location PL1 and a second location PL2. The first location PL1 is, for example, a location where the vehicle 100 is assembled, and the second location PL2 is, for example, a location where the vehicle 100 is inspected. Any position in the factory is expressed by xyz coordinate values ​​of a reference coordinate system Σr. The reference coordinate system Σr is defined, for example, as a world coordinate system (also called a global coordinate system).

[0012] The first location PL1 and the second location PL2 are connected by a track SR along which the vehicle 100 can travel. The track SR does not have to be a continuous track as shown in FIG. 1, and may be, for example, a track that branches into multiple tracks along the way, or a track where multiple tracks join along the way. A target route along which the vehicle 100 travels by remote control is set on the track SR. The target route in this embodiment includes routes Rt1, Rt2, and Rt3. The target route may be determined according to the type of the vehicle 100 (for example, the model, type, grade, or power source), the inspection results of the vehicle 100, the degree of congestion of the track, the degree of congestion of the next process, or the like. For example, manufacturing information described later may be used to determine such a target route.

[0013] A plurality of cameras 301 are installed around the road SR. Each camera 301 is arranged so that when the vehicle 100 is at any position on the target route, at least one camera 301 can always capture the image of the vehicle 100. The control device 200 can obtain the relative position and orientation of the vehicle 100 with respect to the target route and the traveling direction of the vehicle 100 in real time using the captured images Pi captured by each camera 301. The position, orientation, and traveling direction detected in this way are used to generate a control command for remotely controlling the vehicle 100. In this embodiment, each camera 301 is arranged so that it can capture an image overlooking the road SR from above. In addition, the position of each camera 301 is fixed, and the relative relationship between the reference coordinate system Σr and the device coordinate system of each camera 301 (hereinafter also referred to as the camera coordinate system) is known. A coordinate transformation matrix for mutually transforming the coordinate values ​​of the reference coordinate system Σr and the coordinate values ​​of the device coordinate system of each camera 301 is stored in advance in the control device 200. In addition, in remote control of the vehicle 100, various sensors such as various on-board cameras mounted on the vehicle 100, LiDAR (Light Detection And Ranging), millimeter wave radar, ultrasonic sensors, infrared sensors, etc. may be used as auxiliary sensors.

[0014] The control device 200 in this embodiment is configured as a remote control device that generates the above-mentioned control command and transmits it to the vehicle 100. More specifically, the control device 200 generates a control command for causing the vehicle 100 to travel along a target route, and transmits the control command to the vehicle 100. The vehicle 100 travels according to the received control command. Therefore, the system 10 can move the vehicle 100 from the first location PL1 to the second location PL2 by remote control without using a transport device such as a crane or a conveyor.

[0015] As shown in FIG. 2, the vehicle 100 includes a vehicle control device 110 for controlling each part of the vehicle 100, an actuator group 120 that operates under the control of the vehicle control device 110, a communication device 130 for communicating with the control device 200 by wireless communication, and a GNSS (Global Navigation Satellite System) receiver 140 for acquiring position information of the vehicle 100. In this embodiment, the actuator group 120 includes an actuator of a drive device for accelerating the vehicle 100, an actuator of a steering device for changing the traveling direction of the vehicle 100, and an actuator of a braking device for decelerating the vehicle 100. The drive device includes a battery, a driving motor driven by the power of the battery, and a driving wheel rotated by the driving motor. The actuator of the drive device includes a driving motor. The actuator group 120 may further include an actuator for swinging the wipers of the vehicle 100, an actuator for opening and closing the power windows of the vehicle 100, and the like.

[0016] The vehicle control device 110 is configured by a computer including a processor 111, a memory 112, an input / output interface 113, and an internal bus 114. The processor 111, the memory 112, and the input / output interface 113 are connected to each other via the internal bus 114 so as to be able to communicate bidirectionally. The actuator group 120, the communication device 130, and the GNSS receiver 140 are connected to the input / output interface 113.

[0017] In this embodiment, the processor 111 functions as the vehicle control unit 115 and the position information acquisition unit 116 by executing a program PG1 stored in advance in the memory 112. The vehicle control unit 115 controls the actuator group 120. When a driver is on board the vehicle 100, the vehicle control unit 115 can drive the vehicle 100 by controlling the actuator group 120 in response to an operation by the driver. The vehicle control unit 115 can also drive the vehicle 100 by controlling the actuator group 120 in response to a control command transmitted from the control device 200, regardless of whether a driver is on board the vehicle 100. The position information acquisition unit 116 acquires position information indicating the current location of the vehicle 100 using the GNSS receiver 140. However, the position information acquisition unit 116 and the GNSS receiver 140 can be omitted.

[0018] The control device 200 is configured by a computer including a processor 201, a memory 202, an input / output interface 203, and an internal bus 204. The processor 201, the memory 202, and the input / output interface 203 are connected via the internal bus 204 to enable bidirectional communication. A communication device 205 is connected to the input / output interface 203 for communicating with the vehicle 100, the camera 301, and the process management device 400 by wireless communication.

[0019] In this embodiment, the processor 201 executes a program PG2 prestored in the memory 202 to function as an image acquisition unit 210, an evaluation unit 220, a first determination unit 230, a second determination unit 240, an estimation unit 250, and a command generation unit 260. In addition to the program PG2, the memory 202 stores correspondence data 271, an estimation program 280, and reference data 275, which will be described later. The estimation program 280 in this embodiment is a comprehensive program including a first analysis program 281 and a second analysis program 282, which will be described later. The reference data 275 includes data related to a reference image, which will be described later.

[0020] The image acquisition unit 210 acquires a captured image Pi from the camera 301. More specifically, the image acquisition unit 210 acquires a plurality of captured images Pi captured by the plurality of cameras 301 from each camera 301, as described later. The captured image Pi is preferably a color image, for example, but may be a grayscale image. The camera 301 in this embodiment captures the captured image Pi so as to capture at least one of the first portion Dp1 and the second portion Dp2 shown in FIG. 1. The first portion Dp1 is a left portion located to the left of the center position in the vehicle width direction of the vehicle 100, and the second portion Dp2 is a right portion located to the right of the center position of the vehicle 100. More specifically, the first portion Dp1 is a left rear corner of the vehicle 100, and the second portion Dp2 is a right rear corner of the vehicle 100. That is, in this embodiment, the first portion Dp1 and the second portion Dp2 are portions located approximately symmetrically to each other in the vehicle width direction of the vehicle 100. The first portion Dp1 and the second portion Dp2 correspond to detection portions that can be detected as detection points in the estimation process described below. The detection points are points used to estimate the vehicle position that represents the position of the vehicle 100 in the estimation process.

[0021] In the system 10 in this embodiment, the camera 301A and the camera 301B shown in FIG. 1 photograph the vehicle 100 traveling on the route Rt1. The camera 301C and the camera 301D photograph the vehicle 100 traveling on the route Rt2. The camera 301E and the camera 301F photograph the vehicle 100 traveling on the route Rt3. The cameras 301A, 301C, and 301E photograph the vehicle 100 traveling on the road SR from the left rear side. The cameras 301B, 301D, and 301F photograph the vehicle 100 traveling on the road SR from the right rear side. In this embodiment, "photographing the vehicle 100 from the left rear side" refers to photographing the vehicle 100 from the left rear side so that the first portion Dp1 (left rear corner) is reflected in the photographed image Pi. "Photographing the vehicle 100 from the right rear side" refers to photographing the vehicle 100 from the right rear side so that the second portion Dp2 (right rear corner) appears in the photographed image Pi. Therefore, the photographed image Pi photographed from the left rear side of the vehicle 100 usually includes the first portion Dp1 regardless of the position where the vehicle 100 is traveling. On the other hand, the photographed image Pi may or may not include the second portion Dp2 depending on the position where the vehicle 100 is traveling. Also, when the photographed image Pi photographed from the left rear side of the vehicle 100 includes the first portion Dp1 and the second portion Dp2, the second portion Dp2 appears smaller than the first portion Dp1 in the photographed image Pi. Similarly, the photographed image Pi photographed from the right rear side of the vehicle 100 usually includes the second portion Dp2 regardless of the position where the vehicle 100 is traveling. On the other hand, the captured image Pi may or may not include the first portion Dp1 depending on the position where the vehicle 100 is traveling.

[0022] The evaluation unit 220 shown in FIG. 2 performs image evaluation for each captured image Pi acquired by the image acquisition unit 210. The image evaluation is an evaluation of the feature amount of the captured image Pi. This feature amount is a feature amount that can affect the detection of the vehicle 100 in the captured image Pi, and reflects an influencing factor in the captured image Pi. The influencing factor includes a disturbance factor related to disturbance such as a foreign object, reflected light, or shadow reflected in the captured image Pi, and an internal factor different from the disturbance factor. The internal factor is, for example, an abnormality of the camera 301, or a delay or abnormality in communication between the camera 301 and the control device 200, which causes noise, deterioration of image quality, etc. in the captured image Pi. The foreign object reflected in the captured image Pi is, for example, leaves or dust attached to the lens of the camera 301 that captures the captured image Pi, or a puddle on the road SR, or an image of the vehicle 100 reflected in the puddle. Such an influencing factor can affect the detection of the vehicle 100 in the captured image Pi. For example, the disturbance factor may appear in the captured image Pi so as to overlap with the vehicle 100, or may appear in the captured image Pi with an appearance that may cause the disturbance factor to be erroneously detected as the vehicle 100.

[0023] In the image evaluation in this embodiment, a feature amount related to the color tone of the photographed image Pi is evaluated. "Evaluation of the feature amount related to the color tone of the photographed image Pi" refers to evaluation of the feature amount related to at least one of the brightness, saturation, and hue of the photographed image Pi. For example, the brightness of the photographed image Pi corresponds to the feature amount related to the brightness and hue. When the photographed image Pi includes the above-mentioned disturbance factor or internal factor, the brightness, saturation, and hue of the photographed image Pi change compared to when the photographed image Pi does not include the disturbance factor or internal factor. For example, the brightness and brightness of an area of ​​the photographed image Pi where a shadow or a foreign object attached to the lens of the camera 301 is reflected tends to be lower than the brightness and brightness of the same area where the shadow or foreign object is not reflected. In addition, when the photographed image Pi includes relatively high-intensity light due to sunlight or lighting, the brightness and brightness of the photographed image Pi are locally or overall higher than when the photographed image Pi does not include such light. Also, for example, the color tone of an area of ​​the captured image Pi that contains noise due to internal factors changes relative to the color tone of the same area when no noise is contained. In this way, the feature amount related to the color tone of the captured image Pi correlates well with the influencing factors in the captured image Pi and affects the detection of the vehicle 100 in the captured image Pi. In particular, the feature amount related to the color tone of the captured image Pi correlates well with the disturbance factors in the captured image Pi. In image evaluation, such feature amount correlated with the influencing factors can be used as the feature amount of the evaluation target.

[0024] The evaluation unit 220 in this embodiment performs image evaluation by comparing the feature amount related to the color tone of the photographed image Pi with the feature amount related to the color tone of a reference image prepared in advance for each photographed image Pi. More specifically, in the image evaluation, the evaluation unit 220 calculates a difference value representing the difference in the feature amount related to the color tone between the photographed image Pi and the reference image, and outputs the calculated difference value as a result of the image evaluation. More specifically, in the image evaluation, the evaluation unit 220 calculates, for example, statistics and frequency distributions related to the color tone of the photographed image Pi and the reference image, and outputs a value representing the difference between the statistics and the difference between the frequency distributions of both images as a difference value. The value representing the difference between the frequency distributions is, for example, the sum of the absolute values ​​of the differences between the frequency distributions or the sum of the squared values. Hereinafter, a value output as a result of the image evaluation, such as this difference value, is also referred to as an evaluation score. In this embodiment, the evaluation unit 220 calculates, as this difference value, a difference value representing the difference between the luminance in the photographed image Pi and the luminance in the reference image.

[0025] The reference image is, for example, an image of a scene corresponding to the captured image Pi captured under ideal conditions where the influence of disturbance factors is equal to or less than a predetermined level. In this case, the reference image is preferably an image of the road SR captured in the same orientation as the captured image Pi, and more preferably an image including a vehicle traveling on the road SR. Furthermore, if the reference image includes a vehicle, the vehicle included in the reference image is preferably a vehicle that may be included in the captured image Pi, that is, a vehicle having the same shape and body color as the vehicle 100 to be remotely controlled. The reference image may be created by simulation.

[0026] In the above image evaluation, the evaluation unit 220 can use, for example, a known algorithm for detecting a feature amount related to the color tone of an image, a rule base previously constructed to detect a feature amount related to the color tone of an input image, or various machine learning models previously trained by machine learning to detect a feature amount related to the color tone of an input image. As such machine learning models, a convolutional neural network (hereinafter also referred to as CNN), a deep neural network (hereinafter also referred to as DNN), a support vector machine, a decision tree, or the like can be used. The above-mentioned reference data 275 may include, for example, a reference image prepared in advance, or may include statistics and frequency distributions related to the color tone of the reference image calculated in advance.

[0027] The first determination unit 230 determines an estimation image, which is a captured image Pi used in the estimation process, from each captured image Pi according to the evaluation result by the evaluation unit 220. The first determination unit 230 in this embodiment determines, among each captured image Pi, a captured image Pi with a better image evaluation result as an estimation image. In this embodiment, "better image evaluation result" means that the influence of the influencing factors in the captured image Pi is smaller, and in this embodiment, means that the evaluation score is smaller. This is because the smaller the influence of the external disturbance factors and internal factors in the captured image Pi, the smaller the difference value tends to be. Note that, in other embodiments, for example, the evaluation score may be defined to be larger when the image evaluation result is better.

[0028] The estimation unit 250 shown in FIG. 2 executes an estimation process. The estimation process is a process of detecting the vehicle 100 to be remotely controlled in the estimation image and detecting the vehicle position of the vehicle 100 using the detection result. The estimation process in this embodiment is a process of estimating the vehicle position using a detection point in the estimation image, and includes an analysis process of detecting the detection point in the estimation image by analyzing the estimation image, and a process of estimating the vehicle position using the detected detection point. In this embodiment, the estimation process is executed by the estimation unit 250 executing the estimation program 280. Note that the estimation unit 250 can estimate the position and orientation of the vehicle 100 using, for example, the driving history of the vehicle 100 and position information detected by the GNSS receiver 140. In addition, the driving direction of the vehicle 100 can be estimated using the direction, driving history, control command history, etc. of the vehicle 100.

[0029] The estimation unit 250 in this embodiment has an analysis unit 251 that executes an analysis process. The analysis unit 251 in this embodiment executes the analysis process by executing a first analysis program 281 and a second analysis program 282, which are analysis programs. The analysis programs are programs that realize a function of making a computer that executes the analysis programs analyze a photographed image Pi and detect a detection point in the photographed image Pi. The first analysis program 281 in this embodiment is an analysis program for analyzing a photographed image Pi photographed from the left rear side of the vehicle 100, and is used to detect a first part Dp1 as a detection point. The second analysis program 282 is an analysis program for analyzing a photographed image Pi photographed from the right rear side of the vehicle 100, and is used to detect a second part Dp2 as a detection point.

[0030] The second determination unit 240 determines the contents of the estimation process according to the shooting conditions of the camera 301 that captured the estimation image. Hereinafter, the camera 301 that captured the estimation image is also referred to as the target camera. In this embodiment, the second determination unit 240 specifies the target camera and determines the program to be used in the estimation process according to the camera information Ci of the target camera, thereby changing the contents of the estimation process according to the shooting conditions. More specifically, the second determination unit 240 acquires the camera information Ci of the target camera and determines the detection program by referring to the corresponding data 271 using the acquired camera information Ci. The detection program is an analysis program used to detect the detection point in the analysis process.

[0031] In this specification, "determining a program according to the imaging conditions" refers to determining a processing step of the program according to the imaging conditions. Therefore, the meaning of "determining a program according to the imaging conditions" includes, for example, selecting one analysis program as a program to be determined from a plurality of programs prepared in advance according to the imaging conditions. In this case, the selection of the detection program may be realized, for example, by executing a conditional branch according to the imaging conditions in a comprehensive program including a plurality of programs. In addition, the meaning of "determining a program according to the imaging conditions" includes, for example, preparing a program to be determined by rewriting a part or all of the code included in the program according to the imaging conditions.

[0032] In this embodiment, the position and orientation of each camera 301 are fixed with respect to the running road SR. Therefore, identifying the target camera corresponds to identifying the shooting conditions of the captured image Pi by the target camera. The shooting conditions are conditions related to the detection parts that may be included in the captured image Pi, and include, for example, conditions related to the parts of the vehicle 100 that may be included as detection parts in the captured image Pi and conditions related to the direction in which the captured image Pi is captured. The conditions related to the direction in which the captured image Pi is captured are conditions related to the direction in which the vehicle 100 is captured when the captured image Pi is captured, and are, for example, conditions that determine whether the vehicle 100 is captured from the right side or the left side of the vehicle 100, and conditions that determine whether the vehicle 100 is captured from the front side or the rear side of the vehicle 100. In this embodiment, if the direction in which the captured image Pi is captured is identified, the detection parts that may be included in the captured image Pi are identified. Hereinafter, the shooting conditions of the captured image Pi by the camera 301 are also simply referred to as the "shooting conditions of the camera 301".

[0033] The correspondence data 271 is data that records the correspondence between the camera information Ci and information for determining the contents of the estimation process. The correspondence data 271 in this embodiment is data that records the correspondence between the identification information of each camera 301 and information for determining the program used in the estimation process. In the correspondence data 271 in this embodiment, the identification information of the cameras 301A, 301C, and 301E is associated with a parameter for selecting the first analysis program 281. Also, the identification information of the cameras 301B, 301D, and 301F is associated with a parameter for selecting the second analysis program 282. In another embodiment, the correspondence data 271 may be data that records the correspondence between the identification information of each camera 301, the shooting conditions of each camera 301, and information for determining the contents of the estimation process, for example.

[0034] In this embodiment, each camera 301 is configured to transmit its own identification information as camera information Ci to the control device 200 together with the captured image Pi. This identification information may be used to identify the target camera. In other embodiments, the second determination unit 240 may acquire the camera information Ci from a database pre-stored in the memory 202, an external computer, a recording medium, or the like. In other embodiments, information different from the identification information of the camera 301 may be used as the camera information Ci, and for example, other information capable of identifying the shooting conditions may be used.

[0035] The command generating unit 260 generates a control command for remote control using the position and orientation of the vehicle 100 estimated by the estimating unit 250, and transmits the control command to the vehicle 100. This control command is a command to drive the vehicle 100 according to the target route stored in the memory 202. The control command can be generated as a command including a driving force or a braking force, and a steering angle. Alternatively, the control command may be generated as a command including at least one of the position and orientation of the vehicle 100, and a future driving route.

[0036] The process management device 400 is, for example, configured by a computer, and executes management of the overall manufacturing process of the vehicle 100 in a factory. For example, when one vehicle 100 starts traveling along a target route, individual information indicating an identification number and model number for identifying the vehicle 100 is transmitted from the process management device 400 to the control device 200. This individual information corresponds to manufacturing information used for managing the manufacturing process of the vehicle 100. In addition, the position of the vehicle 100 detected by the control device 200 is also transmitted to the process management device 400. The function of the process management device 400 may be implemented in the same device as the control device 200.

[0037] 3 is a flowchart of the decision process in this embodiment. This decision process is a process for implementing the control method in this embodiment. The decision process is executed by the processor 201 of the control device 200, for example, at predetermined time intervals.

[0038] In S110 of FIG. 3, the image acquisition unit 210 acquires a plurality of captured images Pi. Each captured image Pi acquired in S110 in this embodiment is a captured image Pi captured by a different camera 301. In S120, the evaluation unit 220 performs image evaluation for each captured image Pi acquired in S110. In S130, the first determination unit 230 determines an estimation image from each captured image Pi acquired in S110 according to the evaluation result in S120. In S140, the second determination unit 240 determines a detection program according to the shooting conditions of the target camera, that is, the camera 301 that captured the captured image Pi determined as the estimation image in S130. In S140 in this embodiment, the detection program is determined in this way, and the contents of the estimation process are determined according to the shooting conditions. More specifically, in S140, the contents of the analysis process are determined. The captured images Pi acquired in S110 may be images captured at the same time or at different times. In another embodiment, the captured images Pi acquired in S110 may include two or more captured images Pi captured by the same camera 301.

[0039] FIG. 4 is a diagram for explaining the determination process. FIG. 4 shows a state where the vehicle 100 traveling on the route Rt1 on the road SR is photographed by the camera 301A and the camera 301B. In this case, in S110 of FIG. 3, the photographed image PiA photographed by the camera 301A and the photographed image PiB photographed by the camera 301B are acquired. In S120, image evaluation is performed on the photographed image PiA and on the photographed image PiB, and an evaluation score EA of the photographed image PiA and an evaluation score EB of the photographed image PiB are calculated. In the example of FIG. 4, since the evaluation score EA is lower than the evaluation score EB, in S130, the photographed image PiA is determined as the estimation image. In this case, in S140, the first analysis program 281 is determined as the detection program according to the photographing conditions of the camera 301A. Conversely, if the evaluation score EB is lower than the evaluation score EA, in S130, the captured image PiB is determined as the estimation image, and in S140, the second analysis program 282 is determined as the detection program according to the shooting conditions of the camera 301B. Note that in another embodiment, in S110 of FIG. 3, the image acquisition unit 210 may acquire each captured image Pi captured by each of the three or more cameras 301.

[0040] 5 is a flowchart of the estimation process in this embodiment. In this embodiment, the estimation process is executed every time an estimation image is transmitted to the control device 200 from the responsible camera.

[0041] In S210 of FIG. 5, the estimation unit 250 determines whether or not to use the first analysis program 281 in the analysis of the estimation image determined in S130, based on the result of S140 of FIG. 3. If it is determined in S210 that the first analysis program 281 is to be used, the analysis unit 251 executes a first process realized by the first analysis program 281 in S220 to S230. The first process is a process for detecting the first part Dp1 as the detection part. If it is not determined in S210 that the first analysis program 281 is to be used, the analysis unit 251 executes a second process realized by the second analysis program 282 in S215 to S230. The second process is a process for detecting the second part Dp2 as the detection part. In the following, S220 realized by the first analysis program 281 is also referred to as S220A, and S220 realized by the second analysis program 282 is also referred to as S220B. The same is true for S225 and S230.

[0042] Unlike the first process, the second process in this embodiment includes the inversion process of S215. The inversion process is a process of generating an inverted image by mirror-inverting the estimation image. As a result, the estimation image that is not mirror-inverted is analyzed in S220A to S230A, whereas the inverted image is analyzed in S220B to S230B. The process of mirror-inverting an image in this manner is also called "flip". The first process and the second process are substantially similar except that the second process includes the inversion process. Therefore, in S220B to S230B, it is possible to analyze the inverted image using the same algorithm as in S220A to S230A, that is, in the same procedure. More specifically, when the first portion Dp1 is used as the detection location, in S220A to S230A, the first portion Dp1 is detected as the left rear corner in the non-inverted estimation image. On the other hand, when the second portion Dp2 is used as the detection location, the second portion Dp2, which is actually a right rear corner, is detected as a pseudo left rear corner in the inverted image in S220B to S230B. That is, in this embodiment, the second portion Dp2 is detected in the inverted image as a portion corresponding to the first portion Dp1 in the estimation image that is not mirror-inverted.

[0043] FIG. 6 is an explanatory diagram showing an example of how the image Im1 is analyzed in the estimation process. The image Im1 shown in FIG. 6 is an image captured of the vehicle 100 traveling on the road SR. The image Im1 is expressed in a camera coordinate system having mutually orthogonal Xc and Yc axes as coordinate axes, and the focal point of the camera 301 as the origin. The image Im1 may be an inverted image, or may be an estimation image that is not mirror-inverted. The image Im1 may be subjected to various corrections and pre-processing, such as a distortion correction process for correcting distortion, a rotation process for rotating the image Im1 so that the direction of the movement vector V of the vehicle 100 in the image Im1 faces a predetermined direction, and a crop process for removing unnecessary areas not including the vehicle 100 from the image Im1. For example, an optical flow method is used to detect the magnitude and direction of the movement vector V.

[0044] 5, the analysis unit 251 executes the analysis process. In the analysis process in this embodiment, the analysis unit 251 detects the positioning point PP by calculating the coordinates of the positioning point PP shown in FIG. 6 using the image Im1. The coordinates of the positioning point PP are calculated as coordinates in the vicinity of the detection point in an image coordinate system described later. The positioning point PP detected in this manner represents the detection point in the estimation image.

[0045] In S220, the analysis unit 251 executes a mask process as a detection process for detecting the vehicle 100 in the image Im1. The mask process is a process for detecting the vehicle 100 in the image Im1 and masking a target area including the vehicle 100 in the image Im1 to generate an image Im2 including a mask area Ms in which the target area is masked. In the mask process, the estimation unit 250 generates the image Im2 by inputting the image Im1 to a machine learning model (not shown) that has been machine-learned to mask the vehicle 100 included in the input image. As the machine learning model, for example, a DNN having a CNN structure that realizes semantic segmentation and instance segmentation is used. Note that the machine learning model may be machine-learned using an algorithm other than a neural network, for example.

[0046] In S225, the analysis unit 251 executes a perspective transformation process. The perspective transformation process is a process for generating an image Im3 by performing a perspective transformation on the image Im2. In the perspective transformation process, the analysis unit 251 performs perspective transformation on the image Im2 into a bird's-eye image viewed from above the vehicle 100 (for example, directly above the vehicle 100) approximately perpendicular to the road surface of the road SR, for example, using the position information of the camera 301 and perspective transformation parameters related to internal parameters. The image Im3 is expressed in an image coordinate system. The image coordinate system is a coordinate system having a point on an image plane projected by the perspective transformation as its origin and having mutually orthogonal Xi-axis and Yi-axis as its coordinate axes. The image Im3 includes a mask area Msb corresponding to the mask area Ms transformed by the perspective transformation.

[0047] In S230, the analysis unit 251 executes a positioning point calculation process to calculate the coordinates of the positioning point PP. In the positioning point calculation process in this embodiment, the analysis unit 251 calculates the coordinates of the positioning point PP using the coordinates (Xi1, Yi1) of the first coordinate point P1 and the coordinates (Xi2, Yi2) of the second coordinate point P2.

[0048] The first coordinate point P1 is identified as a coordinate point corresponding to the base coordinate point P0 in the image Im1 among the vertices of the rectangle R1b in the image Im2. The rectangle R1b corresponds to the first circumscribing rectangle R1 transformed by perspective transformation. The first circumscribing rectangle R1 is a rectangle in the image Im2 that circumscribes the mask area Ms and has a long side parallel to the movement vector V. The base coordinate point P0 is a coordinate point representing a vertex of the first circumscribing rectangle R1 that corresponds to the detection location of the vehicle 100. More specifically, the base coordinate point P0 is a coordinate point representing a vertex located to the left and rear of the center of gravity C of the vehicle 100 when the direction of the movement vector V is the forward direction among the vertices of the first circumscribing rectangle R1. The second coordinate point P2 is a coordinate point representing a vertex corresponding to the detection location of the vehicle 100 among the four vertices of the second circumscribing rectangle R2 in the image Im2. More specifically, the second coordinate point P2 is a coordinate representing a vertex of the second circumscribing rectangle R2 that is located to the left and rear of the center of gravity C when the direction of the movement vector V is forward. The second circumscribing rectangle R2 is a rectangle that circumscribes the mask area Msb and has a side parallel to the Xi axis and a side parallel to the Yi axis. In this way, the first coordinate point P1 and the second coordinate point P2 are both coordinate points determined according to the detection location, and therefore have a correlation with each other.

[0049] When the coordinate value Xi1 is greater than the coordinate value Xi2, the estimation unit 250 determines the Xi coordinate value of the positioning point PP to be the coordinate value Xi1. Conversely, when the coordinate value Xi1 is smaller than the coordinate value Xi2, the estimation unit 250 determines the Xi coordinate value to be the coordinate value Xi2. Similarly, when the coordinate value Yi1 is greater than the coordinate value Yi2, the Yi coordinate value of the positioning point PP is determined to be the coordinate value Yi1. When the coordinate value Yi1 is smaller than the coordinate value Yi2, the Yi coordinate value is determined to be the coordinate value Yi2. By determining the Xi coordinate value and the Yi coordinate value in this manner, the positioning point PP is detected.

[0050] At S235, the estimation unit 250 executes a vehicle coordinate calculation process. The vehicle coordinate calculation process is a process of calculating a vehicle coordinate point using the positioning point PP. The vehicle coordinate point is a coordinate point that represents the positioning point PP in the reference coordinate system Σr. At S235, the estimation unit 250 calculates the vehicle coordinate point using, for example, a predetermined estimation formula and the coordinate value of the positioning point PP calculated at S230. The command generation unit 260 generates a control command using the vehicle coordinate point calculated in the estimation process as the vehicle position. The control command thus generated is transmitted to the vehicle 100. That is, in this embodiment, calculating the vehicle coordinate point corresponds to estimating the vehicle position. Note that the estimation method of the vehicle position using the estimation image is not limited to the above. For example, the vehicle 100 in the estimation image may be detected by a process other than the mask process, a detection location in the estimation image may be detected by a process other than the perspective transformation process or the positioning point calculation process, or a coordinate point that represents the vehicle position may be calculated by a process other than the vehicle coordinate calculation process.

[0051] According to the system 10 in the present embodiment described above, the first determination unit 230 determines an estimation image to be used in the estimation process from each captured image Pi in accordance with the result of image evaluation for each captured image Pi acquired by the image acquisition unit 210. Therefore, compared to, for example, a mode in which only one captured image Pi is acquired by the image acquisition unit 210 for the estimation process or a mode in which the estimation process is executed without executing image evaluation, the possibility of appropriately estimating the vehicle position of the remotely controlled vehicle 100 is increased.

[0052] In this embodiment, in the image evaluation, evaluation unit 220 calculates a difference value representing the difference between the feature amount related to the color tone of the captured image Pi and the feature amount related to the color tone of the reference image for each captured image Pi, and outputs each difference value as the result of the image evaluation. Therefore, image evaluation can be performed effectively. In this embodiment, by determining the captured image Pi with the smaller difference value as the estimation image, an appropriate estimation image can be easily determined.

[0053] Furthermore, in this embodiment, second determination unit 240 determines the content of the estimation process according to the shooting conditions of the target camera. Therefore, for example, even if the shooting conditions differ for each camera 301, the possibility that the vehicle position can be appropriately estimated using the estimation image regardless of the shooting conditions is increased. Furthermore, since a control command is generated using the vehicle position estimated by the estimation process, the possibility that an appropriate control command is generated is increased. Therefore, the possibility that vehicle 100 can be appropriately remotely controlled is increased.

[0054] In this embodiment, the second determination unit 240 determines the program to be used in the estimation process according to the shooting conditions of the target camera. Therefore, by determining the program by the second determination unit 240, the content of the estimation process can be determined according to the shooting conditions of the target camera.

[0055] B. Other embodiments: (B1) In the above embodiment, the evaluation unit 220 may perform image evaluation taking into account the timing of each captured image Pi, such as the time, time zone, or season when the image was captured. In this case, the evaluation unit 220 may predict the influence of sunlight and its shadow on each captured image Pi, for example, based on the timing of each captured image Pi and the position and orientation of each camera 301. The evaluation unit 220 may correct the evaluation score of a captured image Pi that is predicted to be more affected by sunlight and its shadow so as to reduce the goodness of the evaluation result of the captured image Pi.

[0056] (B2) In the above embodiment, a value representing the luminance difference between the captured image Pi and the reference image is used as the difference value. In contrast, the difference value may be, for example, a value representing the brightness difference between the two images, a value representing the hue difference, or a value representing the saturation difference. The difference value may also be, for example, a value representing two or more of the brightness difference, the brightness difference, the hue difference, and the saturation difference. In this case, the difference value may be defined as, for example, an index value that takes a larger value or a smaller value as the brightness difference, the brightness difference, the hue difference, or the saturation difference is larger. In addition, the evaluation unit 220 performs an evaluation on a feature amount representing a color tone in the image evaluation, but this is not necessary. For example, an evaluation on a blob, an evaluation on an edge, an evaluation on a corner, or other evaluation on various feature amounts that may affect the detection of the vehicle 100 in the captured image Pi may be performed. For example, when performing an evaluation on an edge, the evaluation unit 220 may perform the image evaluation by comparing the angle, coordinates, interval, and intensity of the edge in the captured image Pi and the reference image. In this case, the evaluation unit 220 may output a value representing the difference in the angle, coordinates, interval, or strength of the edge between the captured image Pi and the reference image as the evaluation score. For such edge evaluation, for example, a known edge detection algorithm, a rule base constructed to detect the edge of the input image, or a machine learning model previously trained by machine learning to detect the edge of the input image can be used.

[0057] (B3) In the above embodiment, the evaluation unit 220 compares the feature amounts of the captured image Pi with those of the reference image in the image evaluation, but this is not necessary. For example, the evaluation unit 220 may perform an evaluation of the feature amounts of the captured image Pi without comparing the feature amounts of the captured image Pi with those of the reference image by inputting the captured image Pi to a machine learning model that has been trained in advance to detect external factors and internal factors of the input image in the image evaluation. In this case, for example, the number of types of influencing factors included in the captured image Pi, the area of ​​the portion of the captured image Pi that is caused by the influencing factors, or the degree of influence of the influencing factors on the detection of the vehicle 100 can be used as the evaluation score.

[0058] (B4) In the above embodiment, information representing the shooting conditions of the camera 301 may be used as the camera information Ci, for example, information relating to a detection portion included in the captured image Pi captured by the camera 301 may be used. In this case, for example, information representing the orientation in which the camera 301 captures the image of the vehicle 100 may be used as the camera information Ci, or information representing a detection portion that may be captured by the camera 301 may be used. By using such camera information Ci, for example, even if the position or orientation of each camera 301 is not fixed with respect to the road SR, the content of the estimation process can be determined according to the shooting conditions of the target camera.

[0059] (B5) In the above embodiment, the second determination unit 240 determines the program used in the estimation process according to the shooting conditions of the target camera, but this is not necessary. For example, even if the shooting conditions are different, the same program may be used for the estimation process. In this case, the second determination unit 240 may determine the content of the estimation process according to the shooting conditions of the target camera by determining the parameters to be used in the estimation process according to the shooting conditions of the target camera. Also, the content of the estimation process may not be determined according to the shooting conditions of the target camera. In this case, the estimation unit 250 may estimate the vehicle position by inputting the estimation image to the same machine learning model regardless of the shooting conditions, for example. Also, in this case, the system 10 may not be provided with the second determination unit 240.

[0060] (B6) In the above embodiment, the first process and the second process include the inversion process, but the inversion process may not be included. In this case, the estimation unit 250 may detect the first part Dp1 or the second part Dp2 in the estimation image that is not mirror-inverted, regardless of whether the detection part is the first part Dp1 or the second part Dp2.

[0061] (B7) In the above embodiment, the first portion Dp1 and the second portion Dp2 do not have to be portions located symmetrically to each other in the vehicle width direction. For example, the left rear corner and the right front corner of the vehicle 100 may be used as the first portion Dp1 and the second portion Dp2, respectively. Furthermore, the left portion and the right portion of the vehicle 100 do not have to be used as the detection location. Furthermore, the number of parts used as the detection location may be three or more, or may be one.

[0062] (B8) In the above embodiment, instead of or in addition to determining the captured image Pi with a better image evaluation result as the estimation image, the first determination unit 230 may determine the captured image Pi with a better image evaluation result than a predetermined standard level as the estimation image. In this case, when the image evaluation results of two or more captured images Pi are better than the standard level, the first determination unit 230 may, for example, determine one captured image Pi randomly selected from these captured images Pi as the estimation image, or may determine a captured image Pi that is predicted to be less affected by light and shadow based on the above-mentioned shooting timing as the estimation image. In addition, when the image evaluation result of each captured image Pi is less than the standard level, for example, the image acquisition unit 210 may newly acquire a captured image Pi from the camera 301. In this way, the possibility of more appropriately determining the estimation image is increased. In addition, when the image evaluation result of each captured image Pi is less than the standard level, the command generation unit 260 may generate a control command to brake the vehicle 100. In this way, the vehicle 100 can be braked when the estimation accuracy of the vehicle position of the vehicle 100 may be degraded.

[0063] (B9) In the above embodiment, the system 10 is configured as a remote control system including the vehicle 100, the camera 301, the estimation unit 250, and the command generation unit 260, but it does not have to be configured in this way. For example, the system 10 may be configured as a system that determines an estimation image and transmits the determination result to another system including the vehicle 100, the camera 301, the estimation unit 250, and the command generation unit 260. In this case, the system 10 may not include some or all of the vehicle 100, the camera 301, the estimation unit 250, and the command generation unit 260. Also, in this case, the system 10 may be configured to transmit the estimation image to the other system together with the determination result, for example. In this case, the estimation unit 250 included in the other system may acquire the estimation image transmitted from the system 10 in the estimation process.

[0064] (B10) In the above embodiment, the vehicle 100 may have a configuration capable of moving by remote control, and may be in the form of a platform having the configuration described below, for example. Specifically, the vehicle 100 may have at least the vehicle control unit 115 and the communication device 130 in order to perform the three functions of "running", "turning" and "stopping" by remote control. That is, the vehicle 100 capable of moving by remote control may not have at least some of the interior parts such as the driver's seat and the dashboard, may not have at least some of the exterior parts such as the bumper and the fenders, and may not have a body shell. In this case, the remaining parts such as the body shell may be attached to the vehicle 100 before the vehicle 100 is shipped from the factory, or the remaining parts such as the body shell may be attached to the vehicle 100 after the vehicle 100 is shipped from the factory in a state in which the remaining parts such as the body shell are not attached to the vehicle 100. Note that the position of the platform may be determined in the same manner as the vehicle 100 in each embodiment.

[0065] The present disclosure is not limited to the above-mentioned embodiment, and can be realized in various configurations without departing from the spirit of the present disclosure. For example, the technical features in the embodiments corresponding to the technical features in each aspect described in the Summary of the Invention column can be appropriately replaced or combined to solve some or all of the above-mentioned problems or to achieve some or all of the above-mentioned effects. Furthermore, if the technical feature is not described as essential in this specification, it can be appropriately deleted. [Explanation of symbols]

[0066] 10...system, 100...vehicle, 110...vehicle control device, 111...processor, 112...memory, 113...input / output interface, 114...internal bus, 115...vehicle control unit, 116...position information acquisition unit, 120...actuator group, 130...communication device, 140...GNSS receiver, 200...control device, 201...processor, 202...memory, 203...input / output interface, 204...internal bus, 205...communication device, 210...image acquisition unit, 220...evaluation unit, 230...first determination unit, 240...second determination unit, 250...estimation unit, 251...analysis unit, 260...command generation unit, 271...correspondence data, 275...reference data, 280...estimation program, 281...first analysis program, 282...second analysis program, 300...camera group, 301...camera, 301A...camera, 301B...camera, 301C...camera, 301D...camera, 301E...camera, 301F...camera, 400...process control device

Claims

1. 1. A system for estimating a vehicle position of a vehicle that is driven by remote control, comprising: an image acquisition unit that acquires a plurality of images of the vehicle, the plurality of images being captured by a plurality of cameras; an evaluation unit that performs an evaluation on a feature amount of the captured image that may affect detection of the vehicle in the captured image for each captured image; a first determination unit that determines, in accordance with a result of the evaluation, an estimation image to be used in an estimation process for estimating the vehicle position, from each of the captured images.

2. 2. The system of claim 1, The evaluation unit is In the evaluation, a value representing a difference between a feature amount relating to a color tone of the photographed image and a feature amount relating to a color tone of a reference image prepared in advance is calculated for each photographed image; The system outputs each of said values ​​as a result of said evaluation.

3. 2. The system of claim 1, a second determination unit that determines content of the estimation process according to a shooting condition of the camera that captured the estimation image.

4. 4. The system of claim 3, The second determination unit determines content of the estimation processing by determining a program to be used in the estimation processing in accordance with the shooting conditions.

5. A system according to any one of claims 1 to 4, comprising: an estimation unit that executes the estimation process using the estimation image; a command generating unit that generates a control command for the remote control using the vehicle position estimated by the estimation process.

Images