System, controller, and control method

Abstract

To provide a technique for increasing the possibility of acquiring an image for detection suitable for remote control.SOLUTION: A system comprises: a vehicle that travels by remote control using a predetermined detection point included in an image for detection obtained by imaging the vehicle; an image acquisition unit that acquires the image for detection; an environmental information acquisition unit that acquires environmental information indicating an environment in which the vehicle travels by the remote control, the environment being likely to affect the image for detection; and a first determination unit that determines at least one of a part of the vehicle to be used as the detection point and an imaging direction of the image for detection according to the acquired environmental information.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to a system, a control device, and a control method. [Background technology]

[0002] Patent Document 1 discloses a technique for running a vehicle by remote control during the vehicle manufacturing process. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2017-538619 Summary of the Invention [Problem to be solved by the invention]

[0004] There is known a technology for remotely controlling a vehicle using an image captured by a camera. However, if the image of the vehicle is simply captured, for example, when the vehicle is exposed to strong sunlight or when the vehicle is turning, it may not be possible to obtain an image suitable for remote control. [Means for solving the problem]

[0005] The present disclosure can be realized in the following forms.

[0006] (1) According to one embodiment of the present disclosure, a system is provided, the system including: a vehicle that travels by remote control using a predetermined detection location included in a detection image captured by the vehicle; an image acquisition unit that acquires the detection image; an environmental information acquisition unit that acquires environmental information that represents an environment in which the vehicle travels by remote control and that may affect the acquired detection image; and a first determination unit that determines at least one of a part of the vehicle to be used as the detection location and a capture direction of the detection image according to the acquired environmental information. In this case, the vehicle part to be used as the detection point and the photographing direction of the detection image can be determined according to the environment in which the vehicle is traveling by remote control, which increases the possibility of obtaining a detection image suitable for remote control of the vehicle. (2) In the above embodiment, the environmental information may include at least one of information indicating a timing at which the vehicle runs under the remote control and information indicating a location at which the vehicle runs under the remote control. In this embodiment, the part of the vehicle used as the detection location and the shooting direction of the detection image can be determined according to the timing and location at which the vehicle runs under the remote control. (3) In the above embodiment, the system includes a plurality of cameras that capture images of the vehicle, and a second determination unit that determines, according to the environmental information, a camera that can capture the detection image so as to satisfy a condition related to at least one of the determined part and the capturing direction, as a camera responsible for capturing the detection image, and the image acquisition unit may acquire the detection image from the camera responsible. With this embodiment, the camera responsible for capturing the detection image can be appropriately determined according to the environmental information. (4) In the above embodiment, the first determination unit may at least determine the portion to be used as the detection portion, and the first determination unit may alternatively determine the portion to be a left part or a right part of the vehicle in accordance with the environmental information. With this embodiment, the left part or the right part of the vehicle can be used as the detection portion in accordance with the environmental information. (5) In the above embodiment, the first determination unit may alternatively determine the part to be a left rear corner or a right rear corner of the vehicle in accordance with the environmental information. In this embodiment, the left rear corner or the right rear corner of the vehicle can be used as the detection part in accordance with the environmental information. (6) In the above embodiment, a third determination unit may be provided that determines the content of the estimation process for estimating the position of the vehicle used in generating the control command for the remote control by using the detection location in the detection image, according to at least one of the determined part and the shooting direction. In this embodiment, the content of the estimation process can be determined according to the part of the vehicle determined as the detection location and the determined shooting direction. Therefore, it is more likely that the position of the vehicle can be estimated with higher accuracy in the estimation process. (7) In the above embodiment, the first determination unit at least determines the part to be used as the detection part, and the first determination unit alternatively determines the part to be one of a first part and a second part that are symmetrically located in a vehicle width direction of the vehicle according to the environmental information, and the third determination unit may determine, when the second part is used as the detection part, to generate an inverted image by mirror-inverting the detection image in the estimation process and to detect the second part in the inverted image, and when the first part is used as the detection part, to detect the first part in the detection image that is not mirror-inverted in the estimation process. With this embodiment, the first part and the second part can be detected in the same procedure except that the detection image is mirror-inverted when the first part and the second part are used as the detection part. (8) In the above embodiment, the third determination unit may determine the content of the estimation process by determining a program to be used in the estimation process in accordance with at least one of the determined part and the shooting direction. In this embodiment, by determining the program by the third determination unit, the content of the estimation process can be determined in accordance with the part of the vehicle determined as the detection location and the determined shooting direction.

[0007] In addition to the above-described system form, the present disclosure can be realized in the form of, for example, a control device, a control method, a computer program for implementing the control method, a non-transitory recording medium on which the computer program is recorded, etc. Furthermore, for example, the above-described system may detect a detection point in a detection image by inputting the detection image to a machine learning model prepared in advance in order to remotely control a vehicle. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a conceptual diagram showing a system configuration. [Diagram 2] 2 is a block diagram showing the configuration of a vehicle and a control device. FIG. [Diagram 3] 13 is a flowchart of a determination process. [Figure 4] FIG. 11 is a first diagram illustrating the decision process in the first embodiment. [Diagram 5] FIG. 2 is a second diagram illustrating the decision process in the first embodiment. [Figure 6] FIG. 11 is a third diagram illustrating the decision process in the first embodiment. [Figure 7] 13 is a flowchart of an estimation process. [Figure 8] FIG. 11 is an explanatory diagram showing an example of how an image is analyzed in the estimation process. [Figure 9] FIG. 11 is a first diagram illustrating a decision process in the second embodiment. [Figure 10] FIG. 11 is a second diagram illustrating the decision process in the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] A. First embodiment: Fig. 1 is a conceptual diagram showing the configuration of a system 10 in the first embodiment. Fig. 2 is a block diagram showing the configuration of a vehicle 100 and a control device 200. The system 10 is configured as a remote control system that causes the vehicle 100 to run by remote control. The system 10 includes one or more vehicles 100, a control device 200 that executes remote control of the vehicle 100, a plurality of cameras 300 that capture images Pi of the vehicle 100, and a process management device 400 that manages the manufacturing process of the vehicle 100.

[0010] The vehicle 100 in this embodiment is a battery electric vehicle (BEV). Note that the vehicle 100 is not limited to an electric vehicle, and may be, for example, a gasoline vehicle, a hybrid vehicle, or a fuel cell vehicle.

[0011] In this embodiment, the vehicle 100 is remotely controlled in a factory where the vehicle 100 is manufactured. As shown in FIG. 1, the factory in this embodiment includes a first location PL1 and a second location PL2. The first location PL1 is, for example, a location where the vehicle 100 is assembled, and the second location PL2 is, for example, a location where the vehicle 100 is inspected. Any position in the factory is expressed by xyz coordinate values ​​of a reference coordinate system Σr. The reference coordinate system Σr is defined, for example, as a world coordinate system (also called a global coordinate system). The first location PL1 and the second location PL2 are connected by a path SR on which the vehicle 100 can travel. The path SR does not have to be a series of paths as shown in FIG. 1, and may be, for example, a path that branches into multiple paths along the way, or a path where multiple paths merge along the way. In this case, the route on which the vehicle 100 runs may be determined according to the type of the vehicle 100 (e.g., vehicle type, model, grade, or power source), the inspection results of the vehicle 100, the degree of congestion on each route, the degree of congestion in the next process, etc. In determining such a route, for example, manufacturing information described later may be used.

[0012] The first location PL1 and the second location PL2 may be provided in the same building, or in different buildings on the same site. The first location PL1 and the second location PL2 may be provided outdoors, not indoors. The first location PL1 and the second location PL2 may be provided distributed across a plurality of sites, for example, in a first factory and a second factory adjacent to each other across a public road or a private road. In this case, the first factory and the second factory are collectively referred to as the factory, and the running path SR may include a part of the public road or a part of the private road.

[0013] A plurality of cameras 300 configured to be able to photograph the vehicle 100 are installed around the road SR. The control device 200 can obtain the relative position and orientation of the vehicle 100 with respect to the target route TR and the traveling direction of the vehicle 100 in real time using the photographed images Pi photographed by each camera 300. The position, orientation, and traveling direction detected in this way are used to generate a control command for remotely controlling the vehicle 100. In this embodiment, each camera 300 is arranged so as to be able to photograph an image overlooking the road SR from above. In addition, the position of each camera 300 is fixed, and the relative relationship between the reference coordinate system Σr and the device coordinate system of each camera 300 (hereinafter also referred to as the camera coordinate system) is known. A coordinate transformation matrix for mutually transforming the coordinate values ​​of the reference coordinate system Σr and the coordinate values ​​of the device coordinate system of each camera 300 is stored in advance in the control device 200. In addition, in remote control of the vehicle 100, various sensors such as various on-board cameras mounted on the vehicle 100, LiDAR (Light Detection And Ranging), millimeter wave radar, ultrasonic sensors, infrared sensors, etc. may be used as auxiliary sensors.

[0014] The control device 200 in this embodiment is configured as a remote control device that generates the above-mentioned control command and transmits it to the vehicle 100. More specifically, the control device 200 generates a control command for causing the vehicle 100 to travel along the target route TR, and transmits the control command to the vehicle 100. The vehicle 100 travels according to the received control command. Therefore, the system 10 can move the vehicle 100 from the first location PL1 to the second location PL2 by remote control without using a transport device such as a crane or a conveyor.

[0015] As shown in FIG. 2, the vehicle 100 includes a vehicle control device 110 for controlling each part of the vehicle 100, an actuator group 120 that operates under the control of the vehicle control device 110, a communication device 130 for communicating with the control device 200 by wireless communication, and a GNSS (Global Navigation Satellite System) receiver 140 for acquiring position information of the vehicle 100. In this embodiment, the actuator group 120 includes an actuator of a drive device for accelerating the vehicle 100, an actuator of a steering device for changing the traveling direction of the vehicle 100, and an actuator of a braking device for decelerating the vehicle 100. The drive device includes a battery, a driving motor driven by the power of the battery, and a driving wheel rotated by the driving motor. The actuator of the drive device includes a driving motor. The actuator group 120 may further include an actuator for swinging the wipers of the vehicle 100, an actuator for opening and closing the power windows of the vehicle 100, and the like.

[0016] The vehicle control device 110 is configured by a computer including a processor 111, a memory 112, an input / output interface 113, and an internal bus 114. The processor 111, the memory 112, and the input / output interface 113 are connected to each other via the internal bus 114 so as to be able to communicate bidirectionally. The actuator group 120, the communication device 130, and the GNSS receiver 140 are connected to the input / output interface 113.

[0017] In this embodiment, the processor 111 functions as the vehicle control unit 115 and the position information acquisition unit 116 by executing a program PG1 stored in advance in the memory 112. The vehicle control unit 115 controls the actuator group 120. When a driver is on board the vehicle 100, the vehicle control unit 115 can drive the vehicle 100 by controlling the actuator group 120 in response to an operation by the driver. The vehicle control unit 115 can also drive the vehicle 100 by controlling the actuator group 120 in response to a control command transmitted from the control device 200, regardless of whether a driver is on board the vehicle 100. The position information acquisition unit 116 acquires position information indicating the current location of the vehicle 100 using the GNSS receiver 140. However, the position information acquisition unit 116 and the GNSS receiver 140 can be omitted.

[0018] The control device 200 is configured by a computer including a processor 201, a memory 202, an input / output interface 203, and an internal bus 204. The processor 201, the memory 202, and the input / output interface 203 are connected via the internal bus 204 to enable bidirectional communication. A communication device 205 is connected to the input / output interface 203 for communicating with the vehicle 100, the camera 300, and the process management device 400 by wireless communication.

[0019] In this embodiment, the processor 201 executes a program PG2 pre-stored in the memory 202 to function as an image acquisition unit 210, an environmental information acquisition unit 220, a first determination unit 230, a second determination unit 240, a third determination unit 245, an estimation unit 250, and a command generation unit 260. In addition to the program PG2, the memory 202 stores condition data 271, camera data 272, and an estimation program 280, which will be described later. The estimation program 280 includes an analysis program, which will be described later.

[0020] The image acquisition unit 210 acquires a captured image Pi from the camera 300. Of the captured images Pi, a captured image Pi that includes a predetermined detection location is also called a detection image. As the detection location, a part of the vehicle 100 is used. The detection location in the detection image is used for remote control. More specifically, the detection location in the detection image is used to estimate the position of the vehicle 100, and the estimated position of the vehicle 100 is used to generate a control command for remote control.

[0021] The environmental information acquisition unit 220 acquires environmental information representing the environment in which the vehicle 100 travels (hereinafter, also referred to as the travel environment) by remote control. The travel environment represented by this environmental information may affect the detection image acquired by the image acquisition unit 210. The environmental information in this embodiment includes travel location information representing the location in which the vehicle 100 travels (hereinafter, also referred to as the travel location) by remote control. The travel location of the vehicle 100 affects the relative position and angle between the vehicle 100 and each camera 300, and therefore may affect the detection image. The travel location information in this embodiment is the current position information of the vehicle 100 on the travel path SR. Note that the "current position information of the vehicle 100" may be information that at least roughly represents the current location of the vehicle 100. The control device 200 and the process management device 400 acquire such current position information by using, for example, the GNSS receiver 140, an area sensor (not shown) provided around the travel path SR, or various sensors (e.g., the camera 300) used for the above-mentioned remote control. Furthermore, by using the current position information of the vehicle 100 and the target route TR, it is possible to identify the location where the vehicle 100 is scheduled to travel in the future, for example, the section where the vehicle 100 is scheduled to travel in the future.

[0022] The first determination unit 230 determines at least one of a detection portion, which is a portion of the vehicle 100 used as a detection location, and a shooting direction of an image for detection, according to the environmental information acquired by the environmental information acquisition unit 220. In this embodiment, the first determination unit 230 determines the detection portion. Hereinafter, the condition regarding at least one of the detection portion and the shooting direction determined by the first determination unit 230 is also referred to as an image condition.

[0023] In this embodiment, the first determination unit 230 determines the detection portion by referring to the condition data 271 according to the environmental information. The condition data 271 in this embodiment is data that records the correspondence between the traveling environment and the portion of the vehicle 100 that can be used as the detection portion. More specifically, the condition data 271 in this embodiment is data that records, for each section included in the road SR, the portion of the vehicle 100 that is used as the detection portion in that section.

[0024] The second determination unit 240 determines the responsible camera, which is the camera 300 responsible for capturing the detection image, according to the environmental information. The responsible camera determined in this manner is the camera 300 capable of capturing an image of the vehicle 100 according to the determined image conditions. In this embodiment, the responsible camera is the camera 300 capable of capturing a captured image Pi including the detection part.

[0025] In this embodiment, the second determination unit 240 determines the responsible camera by referring to the camera data 272 according to the environmental information. The camera data 272 is data that records the correspondence between the driving environment and each camera 300. More specifically, the camera data 272 in this embodiment is data that records, for each section included in the road SR, the identification information of each camera 300 that is responsible for capturing detection images in that section. This camera data 272 is defined so that, in any section included in the road SR, a camera 300 that can capture detection images that satisfy the determined image conditions is determined as the responsible camera.

[0026] The estimation unit 250 executes an estimation process. The estimation process is a process of estimating the position of the vehicle 100 to be remotely controlled by using a detection point in the detection image. The estimation process in this embodiment includes an analysis process and a process of estimating the position of the vehicle 100 by using the result of the analysis process. The analysis process is a process of detecting a detection point in the detection image by analyzing the detection image. In this embodiment, the estimation process is executed by the estimation unit 250 executing the estimation program 280. Note that the estimation unit 250 can estimate the position and orientation of the vehicle 100 by using, for example, the driving history of the vehicle 100 and position information detected by the GNSS receiver 140 mounted on the vehicle 100. In addition, the driving direction of the vehicle 100 can be estimated by using the direction, driving history, control command history, etc. of the vehicle 100.

[0027] The estimation unit 250 in this embodiment has an analysis unit 251 that executes the above-mentioned analysis process. The analysis unit 251 in this embodiment executes the analysis process by executing a first analysis program 281 and a second analysis program 282, which are analysis programs. The analysis programs are programs that realize a function of making a computer that executes the analysis program analyze a detection image and detect a detection point in the detection image.

[0028] The third determination unit 245 determines the contents of the estimation process according to the image conditions, that is, at least one of the determined detection site and the imaging direction. In this embodiment, the third determination unit 245 determines the contents of the estimation process according to the image conditions by determining the program used in the estimation process according to the image conditions. More specifically, the third determination unit 245 determines the detection program according to the image conditions. The detection program is an analysis program used to detect the detection site in the detection image in the analysis process. In this specification, the "determination of the program according to the image conditions" refers to determining the processing steps of the program according to the image conditions. Therefore, the meaning of "determination of the program according to the image conditions" includes, for example, selecting one analysis program according to the image conditions from multiple programs prepared in advance as the program to be determined. In this case, the selection of the detection program may be realized, for example, by executing a conditional branch according to the image conditions in a comprehensive program including multiple programs. In addition, the meaning of "determination of the program according to the image conditions" includes, for example, preparing the program to be determined by rewriting part or all of the code included in the program according to the image conditions.

[0029] The above-mentioned estimation program 280 is a comprehensive program that includes, as analysis programs, a first analysis program 281 and a second analysis program 282. The first analysis program 281 and the second analysis program 282 will be described in detail later.

[0030] The command generating unit 260 generates a control command for remote control using the position and orientation of the vehicle 100 estimated by the estimating unit 250, and transmits the control command to the vehicle 100. This control command is a command to make the vehicle 100 travel according to the target route TR stored in the memory 202. The control command can be generated as a command including a driving force or braking force, and a steering angle. Alternatively, the control command may be generated as a command including at least one of the position and orientation of the vehicle 100, and a future travel route.

[0031] The process management device 400 is, for example, configured by a computer, and executes management of the overall manufacturing process of the vehicle 100 in a factory. For example, when one vehicle 100 starts traveling along the target route TR, individual information indicating an identification number and model number for identifying the vehicle 100 is transmitted from the process management device 400 to the control device 200. This individual information corresponds to manufacturing information used for managing the manufacturing process of the vehicle 100. In addition, the position of the vehicle 100 detected by the control device 200 is also transmitted to the process management device 400. Note that the function of the process management device 400 may be implemented in the same device as the control device 200.

[0032] 3 is a flowchart of the decision process in this embodiment. This decision process is a process for implementing the control method in this embodiment. The decision process is executed by the processor 201 of the control device 200, for example, at predetermined time intervals.

[0033] In S110 of FIG. 3, the environmental information acquisition unit 220 acquires environmental information of the vehicle 100 to be remotely controlled. In S120, the first determination unit 230 determines the image conditions according to the environmental information acquired in S110. As described above, in S120 in this embodiment, a detection part is determined. In S130, the second determination unit 240 determines the camera in charge according to the environmental information acquired in S110. In S140, the third determination unit 245 determines the detection program according to the image conditions determined in S120. In S140 in this embodiment, the detection program is determined in this way, and the contents of the estimation process are determined according to the image conditions. More specifically, in S140, the contents of the analysis process are determined.

[0034] FIG. 4 is a first diagram for explaining the determination process. FIG. 5 is a second diagram for explaining the determination process. FIG. 6 is a third diagram for explaining the determination process. Each of FIG. 4 to FIG. 6 shows an example of the vehicle 100 traveling on the road SR along the target route TR by remote control. More specifically, FIG. 4 shows the vehicle 100A traveling in the section sc1 at time t1. FIG. 5 shows the vehicle 100A traveling in the section sc2 at time t2 after time t1. FIG. 6 shows the vehicle 100A traveling in the section sc3 at time t3 after time t2.

[0035] In the examples of FIG. 4 to FIG. 6, the first portion Dp1 or the second portion Dp2 is alternatively used as the detection portion. In FIG. 4 to FIG. 6, one portion used as the detection portion is indicated by a hatched marker, and the other portion not used as the detection portion is indicated by a white marker. In this embodiment, the first portion Dp1 is a left portion located to the left of the center position in the vehicle width direction of the vehicle 100, and the second portion Dp2 is a right portion located to the right of the center position of the vehicle 100. More specifically, the first portion Dp1 is a left rear corner of the vehicle 100, and the second portion Dp2 is a right rear corner of the vehicle 100. That is, the first portion Dp1 and the second portion Dp2 are portions located approximately symmetrically to each other in the vehicle width direction of the vehicle 100. In the above-mentioned condition data 271, the section sc1 is associated with the first part Dp1, the section sc2 is associated with the second part Dp2, and the section sc3 is associated with the first part Dp1. The first analysis program 281 shown in Fig. 2 is used when the first part Dp1 is used as the detection location. The second analysis program 282 is used when the second part Dp2 is used as the detection location.

[0036] In the present embodiment, in the camera data 272, the section sc1 is associated with the camera 300A, the section sc2 is associated with the camera 300A and the camera 300B, and the section sc3 is associated with the camera 300B. The camera 300A can capture the first part Dp1 in the section sc1, and the second part Dp2 in the section sc2. The camera 300B can capture the second part Dp2 in the section sc2, and the first part Dp1 in the section sc3. When multiple cameras 300 are associated with a certain section, such as the section sc2, it is preferable that the camera 300 capable of more appropriately capturing the detection point is determined as the responsible camera in that section. In this case, for example, the camera 300 that is closer to the vehicle 100 may be determined as the responsible camera, or the responsible camera may be determined so as to more effectively prevent obstacles other than the vehicle 100 from being captured in the detection image. Also, for example, in a certain section, when multiple cameras 300 are capable of capturing detection images according to the determination by the first determination unit 230, the section may be associated with, for example, only the camera 300 that can more appropriately capture the detection location.

[0037] For example, in the example of Fig. 4, in S110 of Fig. 3, travel location information indicating that the current position of the vehicle 100A is section sc1 is acquired as environmental information of the vehicle 100A. Next, in S120, the condition data 271 is referenced according to the travel location information of the vehicle 100A acquired in S110, and the detection site is determined to be the first part Dp1 corresponding to the section sc1. Then, in S130, the camera data 272 is referenced according to the environmental information acquired in S110, and the camera 300A capable of photographing the first part Dp1 corresponding to the section sc1 is determined as the responsible camera. Also, in S140, the first analysis program 281 is determined as the detection program corresponding to the second part Dp2.

[0038] In the example of FIG. 5, in S110 of FIG. 3, travel location information indicating that the travel location of the vehicle 100A is the section sc2 is acquired. In S120, it is determined that the second part Dp2 is used as the detection location corresponding to the section sc2. In S130, a responsible camera is determined corresponding to the section sc2. In this case, as described above, for example, out of the camera 300A and the camera 300B, the camera 300 that can capture the captured image Pi including the second part Dp2 and is closer to the vehicle 100 is determined as the responsible camera. In S140, the second analysis program 282 is determined as the detection program corresponding to the second part Dp2. The example of FIG. 6 is also substantially similar to FIG. 4 and FIG. 5.

[0039] Unlike the present embodiment, when the first portion Dp1 is used as a detection location in the section sc2, for example, due to the vehicle 100 turning right between the sections sc1 and sc2, it may be necessary to arrange a camera capable of photographing the first portion Dp1 of the vehicle 100 traveling in the section sc2 around the road SR. Also, for example, depending on the arrangement of the ceiling, beams, walls, etc. around the road SR, it may be difficult to arrange the camera so as to appropriately photograph the first portion Dp1 of the vehicle 100 traveling in the section sc2. In the present embodiment, it is possible to reduce the number of cameras used for remote control and to suppress the influence on remote control due to restrictions on the installation position of the cameras.

[0040] 7 is a flowchart of the estimation process in this embodiment. In this embodiment, the estimation process is executed every time a detection image is transmitted to the control device 200 from the responsible camera.

[0041] In S205 of FIG. 7, the image acquisition unit 210 acquires a detection image transmitted from the camera in charge. In S210 of FIG. 7, the estimation unit 250 determines whether or not to use the first analysis program 281 for analyzing the detection image based on the result of S140 of FIG. 3. If it is determined in S210 that the first analysis program 281 is to be used, the analysis unit 251 executes the first processing realized by the first analysis program 281 in S220 to S230. If it is not determined in S210 that the first analysis program 281 is to be used, the analysis unit 251 executes the second processing realized by the second analysis program 282 in S215 and S220 to S230. In the following, S220 realized by the first analysis program 281 is also referred to as S220A, and S220 realized by the second analysis program 282 is also referred to as S220B. The same is true for S225 and S230.

[0042] Unlike the first process, the second process includes an inversion process in S215. The inversion process is a process of generating an inverted image by mirror-inverting the detection image. Therefore, while a non-mirror-inverted detection image is analyzed in S220A to S230A, an inverted image is analyzed in S220B to S230B. This process of mirror-inverting an image is also called "flip."

[0043] The first process and the second process are substantially similar except that the second process includes an inversion process. That is, in S220B to S230B, it is possible to analyze the inverted image using the same algorithm as in S220A to S230A, that is, in the same procedure. More specifically, when the first portion Dp1 is used as the detection location, in S220A to S230A, the first portion Dp1 is detected as the left rear corner in the non-inverted detection image. On the other hand, when the second portion Dp2 is used as the detection location, the second portion Dp2, which is actually the right rear corner, is detected as the left rear corner in the inverted image in a pseudo manner in S220B to S230B. That is, in this embodiment, in the inverted image, the second portion Dp2 is detected as a portion corresponding to the first portion Dp1 in the non-mirror-inverted detection image.

[0044] FIG. 8 is an explanatory diagram showing an example of how the image Im1 is analyzed in the estimation process. The image Im1 shown in FIG. 8 is an image captured of the vehicle 100 traveling along the road SR in the traveling direction. The image Im1 is expressed in a camera coordinate system having an Xc axis with the focal point of the camera 300 as the origin and a Yc axis perpendicular to the Xc axis as the coordinate axis. The image Im1 may be an inverted image or a detection image that is not mirror-inverted. The image Im1 may be appropriately subjected to various corrections and pre-processing, such as a distortion correction process for correcting distortion of the image Im1, a rotation process for rotating the image Im1 so that the direction of the movement vector V of the vehicle 100 included in the image Im1 faces a predetermined direction, and a crop process for removing unnecessary areas not including the vehicle 100 from the image Im1. For example, an optical flow method is used to detect the magnitude and direction of the movement vector V of the vehicle 100 included in the captured image Pi.

[0045] 7, the analysis unit 251 executes the analysis process. In the analysis process in this embodiment, the analysis unit 251 detects the positioning point PP by calculating the coordinates of the positioning point PP shown in FIG. 8 using the image Im1. The coordinates of the positioning point PP are calculated as coordinates in the vicinity of the detection point in an image coordinate system described later. The positioning point PP detected in this manner represents the detection point in the detection image.

[0046] In S220, the analysis unit 251 executes a mask process as a detection process for detecting the vehicle 100 in the image Im1. The mask process is a process for detecting the vehicle 100 in the image Im1 and masking a target area including the vehicle 100 in the image Im1 to generate an image Im2 corresponding to the image Im1 in which the target area is masked. Hereinafter, the area masked by the mask process is also referred to as a mask area Ms. In the mask process, the estimation unit 250 generates the image Im2 by inputting the image Im1 to a machine learning model (not shown) that has been machine-learned in advance to mask the vehicle 100 included in the input image. As the machine learning model, for example, a deep neural network (hereinafter also referred to as a DNN) having a structure of a convolutional neural network (hereinafter also referred to as a CNN) that realizes semantic segmentation and instance segmentation is used. Note that the machine learning model is not limited to the above, and may be, for example, a machine learning model that has been trained using an algorithm other than a neural network.

[0047] In S225, the analysis unit 251 executes a perspective transformation process. The perspective transformation process is a process of generating an image Im3 corresponding to the perspective transformed image Im2 by performing a perspective transformation on the image Im2. In the perspective transformation process, the analysis unit 251 performs perspective transformation on the image Im2 into a bird's-eye image viewed from above the vehicle 100 (for example, directly above the vehicle 100) approximately perpendicular to the road surface of the road SR, for example, using the position information of the camera 300 and perspective transformation parameters related to the internal parameters. The image Im3 generated as a bird's-eye image in this way is represented in an image coordinate system different from the camera coordinate system. The image coordinate system is a coordinate system having a point on the image plane projected by the perspective transformation as the origin, and having the Xi axis and the Yi axis perpendicular to the Xi axis as coordinate axes. The image Im3 includes a mask area Msb corresponding to the mask area Ms transformed by the perspective transformation.

[0048] In S230, the analysis unit 251 executes a positioning point calculation process to calculate the coordinates of the positioning point PP. In the positioning point calculation process in this embodiment, the analysis unit 251 calculates the coordinates of the positioning point PP using the coordinates (Xi1, Yi1) of the first coordinate point P1 and the coordinates (Xi2, Yi2) of the second coordinate point P2.

[0049] The first coordinate point P1 is identified as a coordinate point corresponding to the base coordinate point P0 in the image Im1 among the vertices of the rectangle R1b in the image Im2. The rectangle R1b corresponds to the first circumscribing rectangle R1 transformed by perspective transformation. The first circumscribing rectangle R1 is a rectangle in the image Im2 that circumscribes the mask area Ms and has a long side parallel to the movement vector V of the vehicle 100. The base coordinate point P0 is a coordinate point representing a vertex of the first circumscribing rectangle R1 that corresponds to the detection location of the vehicle 100. More specifically, the base coordinate point P0 is a coordinate point representing a vertex located to the left and rear of the center of gravity C of the vehicle 100 among the vertices of the first circumscribing rectangle R1 when the direction of the movement vector V is forward.

[0050] The second coordinate point P2 is a coordinate point representing a vertex corresponding to the detection location of the vehicle 100 among the four vertices of the second circumscribing rectangle R2 in the image Im2. More specifically, the second coordinate point P2 is a coordinate representing a vertex located to the left and rear of the center of gravity C of the vehicle 100 among the vertices of the second circumscribing rectangle R2 when the direction of the movement vector V is forward. The second circumscribing rectangle R2 is a rectangle that circumscribes the mask area Msb and has a side parallel to the Xi axis and a side parallel to the Yi axis. In this way, the first coordinate point P1 and the second coordinate point P2 are both coordinate points determined according to the detection location, and therefore have a correlation with each other.

[0051] When the coordinate value Xi1 of the first coordinate point P1 is greater than the coordinate value Xi2 of the second coordinate point P2, the estimation unit 250 determines the Xi coordinate of the positioning point PP to be the coordinate value Xi2. Conversely, when the coordinate value Xi1 is smaller than the coordinate value Xi2, the estimation unit 250 determines the Xi coordinate of the positioning point PP to be the coordinate value Xi1. Also, when the coordinate value Yi1 of the first coordinate point P1 is greater than the coordinate value Yi2 of the second coordinate point P2, the estimation unit 250 determines the Yi coordinate of the positioning point PP to be the coordinate value Yi1. Conversely, when the coordinate value Yi1 is smaller than the coordinate value Yi2, the estimation unit 250 determines the Yi coordinate of the positioning point PP to be the coordinate value Yi2. In this way, the Xi coordinate and Yi coordinate of the positioning point PP are determined, and the positioning point PP is detected.

[0052] At S235, the estimation unit 250 executes a vehicle coordinate calculation process. The vehicle coordinate calculation process is a process for calculating a vehicle coordinate point using the positioning point PP. The vehicle coordinate point is a coordinate point that represents the positioning point PP in the reference coordinate system Σr. At S235, the estimation unit 250 calculates the vehicle coordinate point using, for example, a predetermined estimation formula and the coordinate value of the positioning point PP calculated at S230. Note that the method of estimating the position of the vehicle 100 using the detection point is not limited to the above. For example, the vehicle 100 in the detection image may be detected by a process other than the mask process, the detection point in the detection image may be detected by a process other than the perspective transformation process or the positioning point calculation process, or the vehicle coordinate point may be calculated by a process other than the vehicle coordinate calculation process.

[0053] The command generator 260 generates a control command using the vehicle coordinate points calculated in the estimation process. The control command thus generated is transmitted to the vehicle 100.

[0054] According to the system 10 in the present embodiment described above, the first determination unit 230 determines at least one of the detection portion and the photographing direction of the detection image in accordance with the environmental information of the vehicle 100. In this way, the detection portion and the photographing direction of the detection image can be determined in accordance with the traveling environment of the vehicle 100. Therefore, the possibility of obtaining a detection image suitable for remote control of the vehicle 100 is increased.

[0055] In this embodiment, the environmental information includes information about the location where the vehicle 100 is traveling. Therefore, the detection portion and the photographing direction of the detection image can be determined by remote control according to the location where the vehicle 100 is traveling.

[0056] In this embodiment, the second determination unit 240 determines, according to the environmental information, the camera 300 capable of capturing the detection image so as to satisfy the determined image condition, among the multiple cameras 300, as the responsible camera. Therefore, the responsible camera can be appropriately determined according to the environmental information.

[0057] In addition, in this embodiment, the first determination unit 230 alternatively determines the detection portion to be the left part or the right part of the vehicle 100 according to the environmental information. In this way, the left part or the right part of the vehicle 100 can be used as the detection portion according to the environmental information. Therefore, by installing the camera 300 on the left side or the right side of the road SR, the detection image including the detection portion can be easily captured by the camera 300. In particular, in this embodiment, the first determination unit 230 alternatively determines the detection portion to be the left rear corner or the right rear corner of the vehicle 100 according to the environmental information. Therefore, by using the camera 300 installed on the left side or the right side of the road SR to capture the vehicle 100 from the rear side of the vehicle 100, the detection image including the detection portion can be easily captured. In addition, in this way, the detection portion can be accurately detected using a circumscribing rectangle in the analysis process.

[0058] In the present embodiment, the third determination unit 245 changes the content of the estimation process according to the determined image conditions. In this way, the content of the estimation process can be changed according to the determined image conditions. Therefore, the possibility that the position of the vehicle 100 can be estimated with higher accuracy in the estimation process is increased.

[0059] In addition, in this embodiment, the third determination unit 245 determines to execute the second process in the estimation process when the second part Dp2 is used as the detection part, and determines to execute the first process in the estimation process when the first part Dp1 is used as the detection part. The first process is a process to detect the first part Dp1 in a detection image that is not mirror-inverted. The second process is a process to detect the second part Dp2 in an inverted image. Therefore, the first part Dp1 and the second part Dp2 can be detected in the same procedure when the first part Dp1 and the second part Dp2 are used as the detection part, except that the detection image is mirror-inverted.

[0060] In the present embodiment, the third determination unit 245 determines the content of the estimation process by determining a program to be used in the estimation process according to the determined image conditions. Therefore, by determining the program by the third determination unit 245, the content of the estimation process can be changed according to the determined image conditions.

[0061] B. Second embodiment: FIG. 9 is a first diagram for explaining the determination process in the second embodiment. FIG. 10 is a second diagram for explaining the determination process in the second embodiment. FIG. 9 and FIG. 10 show an example of the vehicle 100 running along the target route TR on the road SRb by remote control, similarly to FIG. 4 to FIG. 6. In this embodiment, unlike the first embodiment, the environmental information includes running timing information indicating the timing at which the vehicle 100 runs. Also, unlike the first embodiment, the first determination unit 230 in this embodiment determines the shooting direction of the detection image. Parts of the configuration of the system 10 in the second embodiment that are not particularly described are the same as those in the first embodiment.

[0062] FIG. 9 shows the vehicle 100B traveling on the road SRb at time t4. FIG. 10 shows the vehicle 100B traveling on the road SRb at time t5, which is different from time t4. Time t4 is a time when sunlight may be irradiated from the right side of the vehicle 100 when it is fine weather. Therefore, when the vehicle 100 is photographed from the right side of the vehicle 100 at time t4, an appropriate detection image may not be obtained due to the influence of sunlight irradiated on the right side of the vehicle 100. Time t5 is a time when sunlight may be irradiated from the left side of the vehicle 100 when it is fine weather. Therefore, when the vehicle 100 is photographed from the left side of the vehicle 100 at time t5, an appropriate detection image may not be obtained due to the influence of sunlight irradiated on the left side of the vehicle 100. In this way, the travel timing of the vehicle 100 may affect the detection image.

[0063] The driving timing information in this embodiment is information that indicates the current time. In other embodiments, the driving timing information may be, for example, information that indicates a rough time period in which the vehicle 100 drives by remote control. The condition data 271 in this embodiment is data that records the correspondence between the time period and the shooting direction of the detection image. The camera data 272 in this embodiment is data that records the correspondence between the time period and each camera 300.

[0064] In this embodiment, in the condition data 271, a time period including time t4 but not including time t5 is associated with the first shooting direction d1, and a time period including time t5 but not including time t4 is associated with the second shooting direction d2. The first shooting direction d1 is a shooting direction in which the vehicle 100 is shot from the left rear side. The second shooting direction d2 is a shooting direction in which the vehicle 100 is shot from the right rear side. In addition, in the camera data 272, a time period including time t4 but not including time t5 is associated with the camera 300C, and a time period including time t5 but not including time t4 is associated with the camera 300D. The camera 300C is disposed so that the vehicle 100 can be shot from the left rear side. The camera 300D is disposed so that the vehicle 100 can be shot from the right rear side. In addition, in this embodiment, the first analysis program 281 is used when the shooting direction is the first shooting direction d1. In addition, the second analysis program 282 is used when the shooting direction is the second shooting direction d2.

[0065] In the example of FIG. 9, in S110 of FIG. 3, traveling timing information indicating that the current time is time t4 is acquired as environmental information of the vehicle 100B. In S120, the condition data 271 is referenced according to the traveling timing information of the vehicle 100B acquired in S110, and the shooting direction is determined to be the first shooting direction d1 corresponding to the time t4. In S130, the camera data 272 is referenced according to the environmental information acquired in S110, and the camera 300C capable of shooting the vehicle 100B from the first shooting direction d1 is determined as the responsible camera corresponding to the time t4. The example of FIG. 10 is also substantially similar to that of FIG. 9.

[0066] According to the second embodiment described above, the environmental information includes the travel timing information of the vehicle 100. In this way, the detection portion and the photographing direction of the detection image can be determined according to the timing when the vehicle 100 travels by remote control. For example, in the example of FIG. 9 and FIG. 10 described above, at time t4 when sunlight may be irradiated from the right side of the vehicle 100, the vehicle 100 can be photographed from the left side to obtain the detection image. Also, at time t5 when sunlight may be irradiated from the left side of the vehicle 100, the vehicle 100 can be photographed from the right side to obtain the detection image. Therefore, the influence of sunlight on the detection image can be suppressed. In this way, according to this embodiment, the possibility of obtaining a detection image suitable for remote control of the vehicle 100 is increased.

[0067] In another embodiment, the environmental information may include both the travel timing information and the travel location information. In this case, the first determination unit 230 may use the condition data 271 that records the correspondence between the travel timing information and the travel location information, and the parts of the vehicle 100 that can be used as the detection parts and the image capture direction, in determining the detection parts and the image capture direction. In this case, the condition data 271 may be, for example, data that records the correspondence between the section in which the vehicle 100 travels and the detection parts and the image capture direction in that section for each time period.

[0068] C. Other embodiments: (C1) In the above embodiment, the first determination section 230 may determine both the detection portion and the imaging direction of the detection image.

[0069] (C2) In the above embodiment, the environmental information may include various information representing the driving environment of the vehicle 100 instead of or in addition to the driving timing information and the driving location information. For example, the environmental information may include weather information representing the weather in an area including the road SR. In this case, the first determination unit 230 may use the condition data 271 that records the correspondence between the weather and the parts of the vehicle 100 that can be used as the detection parts and the shooting direction when determining the detection parts and the shooting direction. In this case, the first determination unit 230 can, for example, make the detection parts, the shooting direction, and the responsible camera different between sunny weather and rainy weather by referring to the condition data 271 according to the weather information. In addition, when the environmental information includes the driving location information and the weather information, for example, the detection parts, the shooting direction, and the responsible camera can be determined by considering the likelihood of puddles occurring in each section of the road SR in rainy weather. In this way, for example, a decrease in the accuracy of estimating the position of the vehicle 100 caused by the image of the vehicle 100 reflected by a puddle being included in the detection image can be suppressed. In addition, the environmental information may be information indicating the presence or absence of obstacles around the vehicle 100, and the environmental temperature and humidity of the location where the vehicle 100 is traveling, obtained, for example, by a sensor mounted on the vehicle 100 to be remotely controlled or by various sensors installed around the running path SR.

[0070] (C3) In the above embodiment, the camera in charge is determined from the multiple cameras 300 in accordance with the environmental information, but it does not have to be determined in this manner. For example, the angle and position of the camera 300 may be configured to be changeable, and the angle and position of one camera 300 may be changed so that the determined detection portion can be photographed or the vehicle 100 can be photographed in the determined photographing direction in accordance with the environmental information. In this case, the number of cameras 300 provided in the system 10 may be only one. Also, in this case, the system 10 may not have the second determination unit 240.

[0071] (C4) In the above embodiment, for example, the first portion Dp1 may be the right side of the vehicle 100, and the second portion Dp2 may be the left side. In addition, a pair of portions located symmetrically with each other in the vehicle width direction, such as the first portion Dp1 and the second portion Dp2, may not be used as the detection location. For example, the left rear corner and the right front corner of the vehicle 100 may be used as the detection location. In addition, the left side or right side of the vehicle 100 may not be used as the detection location, and a portion of the vehicle 100 other than the left side or right side may be used. In addition, three or more portions may be used as the detection location. In addition, when the detection portion is not determined by the first determination unit 230, the number of portions used as the detection location may be only one.

[0072] (C5) In the above embodiment, the third determination unit 245 may not determine the program used in the estimation process according to the determined image conditions. In this case, for example, the program used in the estimation process may be determined according to the environmental conditions. Also, for example, even if the image conditions are different, the same program may be used in the estimation process. In this case, the estimation unit 250 and the analysis unit 251 may change the contents of the estimation process and the analysis process by changing the parameters used in the estimation process and the analysis process according to the image conditions. Also, the third determination unit 245 may not determine the contents of the estimation process according to the image conditions. In this case, the estimation unit 250 may estimate the position of the vehicle 100 by inputting the detection image into the same machine learning model regardless of the image conditions. Also, in this case, the system 10 may not include the third determination unit 245.

[0073] (C6) In the above embodiment, the third determination unit 245 determines to execute the second process in the estimation process when the second part Dp2 is used as the detection location. In contrast, the third determination unit 245 may not determine to execute the second process in the above case. In this case, the analysis unit 251 may detect the second part Dp2 by actually detecting the second part Dp2 in a detection image that is not mirror-inverted, for example.

[0074] (C7) In the above embodiment, the vehicle 100 may have a configuration capable of moving by remote control, and may be in the form of a platform having the configuration described below, for example. Specifically, the vehicle 100 may have at least the vehicle control unit 115 and the communication device 130 in order to perform the three functions of "running", "turning", and "stopping" by remote control. That is, the vehicle 100 capable of moving by remote control may not have at least some of the interior parts such as the driver's seat and the dashboard, may not have at least some of the exterior parts such as the bumper and the fenders, and may not have a body shell. In this case, the remaining parts such as the body shell may be attached to the vehicle 100 before the vehicle 100 is shipped from the factory, or the remaining parts such as the body shell may be attached to the vehicle 100 after the vehicle 100 is shipped from the factory in a state in which the remaining parts such as the body shell are not attached to the vehicle 100. Note that the position of the platform may be determined in the same manner as the vehicle 100 in each embodiment.

[0075] (C8) In the above embodiment, the system 10 is configured as a remote control system including the camera 300, the estimation unit 250, and the command generation unit 260, but it does not have to be configured in this way. For example, the system 10 may be configured as a system that determines at least one of the detection portion and the shooting direction of the detection image in accordance with environmental information, and transmits the determination result to another system that includes the camera 300, the estimation unit 250, and the command generation unit 260. In this case, the system 10 does not have to include, for example, some or all of the camera 300, the estimation unit 250, and the command generation unit 260.

[0076] The present disclosure is not limited to the above-mentioned embodiment, and can be realized in various configurations without departing from the spirit of the present disclosure. For example, the technical features in the embodiments corresponding to the technical features in each aspect described in the Summary of the Invention column can be appropriately replaced or combined to solve some or all of the above-mentioned problems or to achieve some or all of the above-mentioned effects. Furthermore, if the technical feature is not described as essential in this specification, it can be appropriately deleted. [Explanation of symbols]

[0077] 10... system, 100, 100A, 100B... vehicle, 110... vehicle control device, 111... processor, 112... memory, 113... input / output interface, 114... internal bus, 115... vehicle control unit, 116... position information acquisition unit, 120... actuator group, 130... communication device, 140... GNSS receiver, 200... control device, 201... processor, 202... memory, 203... input / output interface, 204... internal bus bus, 205... communication device, 210... image acquisition unit, 220... environmental information acquisition unit, 230... first determination unit, 240... second determination unit, 245... third determination unit, 250... estimation unit, 260... command generation unit, 271... condition data, 272... camera data, 280... estimation program, 281... first analysis program, 282... second analysis program, 300, 300A, 300B, 300C, 300D... cameras, 400... process control device

Claims

1. A vehicle that runs by remote control using a predetermined detection location included in a detection image captured by the vehicle; an image acquisition unit for acquiring the detection image; an environmental information acquisition unit that acquires environmental information representing an environment in which the vehicle travels by remote control and which may affect the acquired detection image; A system comprising: a first determination unit that determines at least one of a part of the vehicle to be used as the detection location and a shooting direction of the detection image according to the acquired environmental information.

2. 2. The system of claim 1, The system, wherein the environmental information includes at least one of information representing a timing at which the vehicle will travel under the remote control and information representing a location at which the vehicle will travel under the remote control.

3. 2. The system of claim 1, A plurality of cameras for photographing the vehicle; a second determination unit that determines, according to the environmental information, a camera that can capture the detection image so as to satisfy a condition related to at least one of the determined part and the imaging direction, as a camera responsible for capturing the detection image; The image acquisition unit acquires the detection image from the assigned camera.

4. 2. The system of claim 1, The first determination unit determines at least the portion to be used as the detection portion, The first determination unit alternatively determines the part to be a left part or a right part of the vehicle depending on the environmental information.

5. 5. The system of claim 4, The system wherein the first determination unit alternatively determines the portion to be a left rear corner or a right rear corner of the vehicle in accordance with the environmental information.

6. A system according to any one of claims 1 to 5, comprising: the system further comprising a third determination unit that determines the content of an estimation process that uses the detection location in the detection image to estimate the position of the vehicle to be used for generating a control command for the remote control, in accordance with at least one of the determined part and the shooting direction.

7. 7. The system of claim 6, The first determination unit determines at least the portion to be used as the detection portion, the first determination unit determines, in accordance with the environmental information, the part to be one of a first portion and a second portion that are symmetrically positioned with respect to each other in a vehicle width direction of the vehicle; The third determination unit, determining, when the second portion is used as the detection location, to generate an inverted image by mirror-inverting the image for detection in the estimation process, and to execute a process of detecting the second portion in the inverted image; determining, when the first portion is used as the detection portion, to execute a process of detecting the first portion in the detection image that is not mirror-inverted in the estimation process; system.

8. 7. The system of claim 6, The third determination unit determines content of the estimation process by determining a program to be used in the estimation process according to at least one of the determined part and the imaging direction.

9. an image acquisition unit that acquires a detection image obtained by capturing a detection image of a vehicle, the detection image including a predetermined detection portion that is used for remote control of the vehicle; an environmental information acquisition unit that acquires environmental information representing an environment in which the vehicle travels by remote control and which may affect the acquired detection image; A control device comprising: a first determination unit that determines at least one of a part of the vehicle to be used as the detection location and a shooting direction of the detection image according to the acquired environmental information.

10. acquiring a detection image of a vehicle, the detection image being a photograph of the vehicle, the detection image including a predetermined detection location being used for remote control of the vehicle; acquiring environmental information representing an environment in which the vehicle is traveling by remote control and which may affect the acquired detection image; A control method comprising a step of determining at least one of a part of the vehicle to be used as the detection location and a shooting direction of the detection image according to the acquired environmental information.

Images