Remote control device, remote control system, and remote control method
Through the remote control device and sensor system, the vehicle driving movement is optimized using backoff control values and prediction algorithms, and the problem of low position judgment accuracy in overlapping multiple vehicles is solved, achieving more accurate position tracking and safe driving.
Patent Information
- Application Number
- CN202410593928.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-05-26
- Filing Date
- 2024-05-14
- Publication Date
- 2024-11-26
AI Technical Summary
When using cameras or lidar to track vehicle positions, it is difficult to accurately track multiple vehicles when overlapping, resulting in a reduced accuracy of position judgment.
Design a remote control device, obtain information of multiple moving bodies through sensors set in different places, detect and produce backoff control values, ensure that the positional relationship between the moving bodies does not overlap, and optimize the target distance using prediction algorithms and vehicle appearance and shape information. Adjust the driving movement to reduce the probability of overlap.
Improve the accuracy of position judgment when multiple vehicles overlap in sensor information, ensure more accurate position tracking, and reduce the risk of conflict between vehicles.
Smart Images

Figure CN119024826A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to a remote control device, a remote control system and a remote control method. Background Art
[0002] In the past, there are known vehicles that can travel automatically by remote control (Patent Document 1). In this technology, a monitoring system such as a camera or a laser radar installed in a place different from the vehicle is used to track the movement of the vehicle. In this way, in the event of a malfunction of the vehicle, the movement of the vehicle can be remotely controlled before the movement of the vehicle becomes obstructed.
[0003] Prior art literature
[0004] Patent Document 1: Japanese Patent Application No. 2017-538619 Summary of the invention
[0005] Problems to be solved by the invention
[0006] When tracking the position of a vehicle using a captured image obtained by a camera installed at a different location from the vehicle, if two or more vehicles overlap in the captured image, the position of each vehicle may not be accurately tracked. Similarly, when tracking the position of a vehicle using laser radar information obtained by a laser radar installed at a different location from the vehicle, if two or more vehicles overlap in the detection range of the laser radar, the position of each vehicle may not be accurately tracked. Such issues are not limited to vehicles, but are common even when a moving object is used as an object.
[0007] Means for solving problems
[0008] The present disclosure can be implemented as the following aspects.
[0009] (1) According to the first aspect of the present disclosure, a remote control device is provided. The remote control device for remotely controlling the driving action of at least one of a plurality of moving bodies comprises: an acquisition unit for acquiring sensor information representing at least a part of the plurality of moving bodies acquired by a sensor installed in a place different from the plurality of moving bodies; a detection unit for detecting the moving body included in the sensor information; a control value generating unit for generating a control value for specifying the driving action of at least a part of the moving bodies in the plurality of moving bodies using the sensor information; and a sending unit for sending the control value to a control target moving body that is a control target of the driving action among the plurality of moving bodies, wherein the control value generating unit generates a control value for the driving action of at least a part of the moving bodies in the plurality of moving bodies. In at least one of the first case where two or more of the plurality of moving bodies in the sensor information exist overlappingly and the second case where it is predicted before the sensor information is obtained that the two or more moving bodies will exist overlappingly when the sensor information is obtained, in order to form a positional relationship in which the detection target moving body that is the detection target based on the sensor information among the two or more moving bodies can be detected by the detection unit without overlapping with other moving bodies among the plurality of moving bodies, a retraction control value is prepared as the control value that specifies the driving action of at least one of the two or more moving bodies. According to this mode, in at least one of the first case and the second case, a retraction control value can be prepared for forming a positional relationship in which the detection target moving body among the two or more moving bodies can be detected without overlapping with other moving bodies. And the prepared retraction control value can be sent to the control target moving body. Thus, when the position of the moving body is tracked using sensor information representing the moving body obtained by a sensor set at a place different from the moving body, the possibility of the two or more moving bodies existing overlappingly in the sensor information can be reduced. Therefore, the position of the moving body can be tracked more accurately using the sensor information.
[0010] (2) In the above-mentioned aspect, the first situation may be a situation where the first moving body and the second moving body among the plurality of moving bodies overlap in the sensor information, the second situation may be a situation where it is predicted before the sensor information is obtained that the first moving body and the second moving body will overlap when the sensor information is obtained, the remote control device may further include an actual distance calculation unit that calculates the actual distance between the first moving body and the second moving body, and the retraction control value may be the control value that specifies the travel action of at least one of the first moving body and the second moving body so that the actual distance becomes a target distance longer than the actual distance at a time point when at least one of the first situation and the second situation occurs. According to this aspect, the actual distance between the first moving body and the second moving body that overlap or are predicted to overlap in the sensor information can be calculated. Furthermore, in order to make the actual distance between the first moving body and the second moving body become a target distance longer than the actual distance at a time point when at least one of the first situation and the second situation occurs, a retraction control value that specifies the travel action of at least one of the first moving body and the second moving body may be prepared. Thus, when the positions of the first moving object and the second moving object are tracked using the sensor information, the possibility that the first moving object and the second moving object exist overlapping on the sensor information can be more reliably reduced.
[0011] (3) In the above-mentioned method, the target distance may be set according to the appearance shape of the first moving body and the appearance shape of the second moving body, and the larger the appearance shape of the first moving body and the appearance shape of the second moving body, the longer the target distance, and the smaller the appearance shape of the first moving body and the appearance shape of the second moving body, the shorter the target distance. According to this method, the target distance can be set according to the appearance shape of the first moving body and the appearance shape of the second moving body. In this way, it is possible to set the target distance corresponding to the appearance shape of the moving body that is correlated with the area occupied by the moving body in the sensor information. Thereby, it is possible to suppress the decrease in the calculation accuracy of the position of the moving body according to the type of the moving body. In addition, it is possible to suppress the position of the moving body from becoming untraceable according to the type of the moving body.
[0012] (4) In the above-mentioned method, the second moving body may be the moving body that is traveling in the same forward direction as the first moving body behind the first moving body, and the retreat control value may include at least one of an acceleration control value that increases the acceleration of the first moving body in the forward direction, an acceleration control value that decreases the acceleration of the second moving body in the forward direction, a direction control value that causes the second moving body to retreat in a direction opposite to the forward direction, and a stop control value that stops the traveling action of the second moving body. According to this method, when the second moving body is traveling in the same forward direction as the first moving body behind the first moving body, a retreat control value including an acceleration control value that increases the acceleration of the first moving body in the forward direction can be generated. Thus, by controlling the acceleration of the first moving body, the actual distance between the first moving body and the second moving body can be changed to the target distance. In addition, according to this method, a retreat control value including an acceleration control value that decreases the acceleration of the second moving body in the forward direction can be generated. Thus, by controlling the acceleration of the second moving body, the actual distance between the first moving body and the second moving body can be changed to the target distance. In addition, according to this method, a retreat control value including a direction control value for causing the second moving body to retreat in a direction opposite to the forward direction can be created. Thus, by causing the second moving body to retreat in a backward direction opposite to the forward direction, the actual distance between the first moving body and the second moving body can be changed to the target distance. In addition, according to this method, a retreat control value including a stop control value for stopping the travel action of the second moving body can be created. Thus, by stopping the second moving body, the actual distance between the first moving body and the second moving body can be changed to the target distance.
[0013] (5) In the above-mentioned method, the above-mentioned retreat control value may also include a deflection angle control value that specifies the deflection angle of the above-mentioned control object moving body. According to this method, a retreat control value that includes a deflection angle control value that specifies the deflection angle of the moving body in addition to at least one of the acceleration control value, the direction control value, and the stop control value can be generated. In this way, in the process of changing the actual distance between the first moving body and the second moving body to the target distance, the position deviation relative to the predetermined target driving route can be corrected.
[0014] (6) In the above method, the retraction control value may include a deflection angle control value that specifies the deflection angle of the control target moving body in order to move any one of the first moving body and the second moving body in the width direction of the travel path. According to this method, a retraction control value including a deflection angle control value for moving any one of the first moving body and the second moving body in the width direction of the travel path can be prepared. Thus, by moving any one of the first moving body and the second moving body in the width direction of the travel path, the actual distance between the first moving body and the second moving body can be changed to the target distance.
[0015] (7) In the above-mentioned method, the above-mentioned retreat control value may also include an acceleration control value that specifies the acceleration of the above-mentioned control object moving body. According to this method, it is possible to generate a retreat control value that includes an acceleration control value in addition to a deflection angle control value for moving either the above-mentioned first moving body or the above-mentioned second moving body in the width direction of the travel path. In this way, it is possible to adjust the moving speed when moving either the first moving body or the second moving body in the width direction of the travel path.
[0016] (8) In the above-mentioned method, the second moving body may be the moving body that is traveling in the same forward direction as the first moving body behind the first moving body, and the retraction control value is the control value for causing any one of the first moving body and the second moving body to move in the width direction of the travel path and cause the second moving body to exceed the first moving body, and includes an acceleration control value that specifies the acceleration of the moving body to be controlled and a deflection angle control value that specifies the deflection angle of the moving body to be controlled. According to this method, when the second moving body is traveling in the same forward direction as the first moving body behind the first moving body, the following retraction control value can be prepared. In this case, the retraction control value including the acceleration control value and the deflection angle control value for causing any one of the first moving body and the second moving body to move in the width direction of the travel path and cause the second moving body to exceed the first moving body can be prepared. Thus, by causing the second moving body to exceed the first moving body, the actual distance between the first moving body and the second moving body can be changed to the target distance.
[0017] (9) In the above method, the retraction control value may include a track control value that specifies a travel trajectory by arranging the target travel positions of the control object moving body at predetermined intervals in a time series order. According to this method, a retraction control value that includes a track control value that specifies a travel trajectory of the control object moving body by arranging the target travel positions of the control object moving body at predetermined intervals in a time series order can be prepared. In this way, the travel trajectory of the control object moving body can be specified without specifying detailed travel actions such as acceleration and deflection angle.
[0018] (10) In the above-mentioned aspect, the remote control device may include a determination unit that determines whether the second situation is met based on the types of the two or more moving objects that are predicted to be included in the sensor information when the sensor information is acquired. According to this aspect, it is possible to determine whether the second situation is met based on the types of the two or more moving objects that are predicted to be included in the sensor information when the sensor information is acquired. In this way, it is possible to more reliably determine whether the second situation is met.
[0019] (11) In the above-mentioned aspect, the remote control device may include a determination unit that determines whether the second situation is met based on the appearance shapes of the two or more moving objects that differ according to the type of the moving objects and are predicted to be included in the sensor information when the sensor information is acquired. According to this aspect, it is possible to determine whether the second situation is met based on the appearance shapes of the two or more moving objects that differ according to the type of the moving objects and are predicted to be included in the sensor information when the sensor information is acquired. As a result, it is possible to more reliably determine whether the second situation is met.
[0020] (12) According to the second mode of the present disclosure, a remote control system is provided. The remote control system comprises: a plurality of mobile bodies; a sensor, which is arranged at a location different from the plurality of mobile bodies and obtains sensor information representing at least a part of the plurality of mobile bodies; and the remote control device described in the above mode. According to this mode, in at least one of the first case and the second case, a backoff control value can be generated for forming a positional relationship in which a detection target mobile body among two or more mobile bodies can be detected without overlapping with other mobile bodies. And, the generated backoff control value can be sent to the control target mobile body. Thus, when the position of the mobile body is tracked using sensor information representing the mobile body obtained by a sensor arranged at a location different from the mobile body, the possibility of two or more mobile bodies existing overlappingly in the sensor information can be reduced. Therefore, the position of the mobile body can be tracked more accurately using the sensor information.
[0021] (13) According to the third aspect of the present disclosure, a remote control method is provided. The remote control method for remotely controlling the driving action of at least one of a plurality of moving bodies comprises the following steps: an acquisition step of acquiring sensor information representing at least a part of the plurality of moving bodies acquired by a sensor installed in a location different from the plurality of moving bodies; a detection step of detecting the moving body included in the sensor information; a control value creation step of using the sensor information to create a control value that specifies the driving action of at least a part of the moving bodies among the plurality of moving bodies; and a sending step of sending the control value to a control target moving body that is a control target of the driving action among the plurality of moving bodies, wherein in the control value creation step In at least one of the first case where two or more of the plurality of moving bodies overlap in the sensor information and the second case where it is predicted before the sensor information is obtained that the two or more moving bodies will overlap when the sensor information is obtained, a retraction control value is prepared as the control value for specifying the driving action of at least one of the two or more moving bodies in order to form a positional relationship in which the detection target moving body that is the detection target based on the sensor information can be detected in the detection process without overlapping with other moving bodies in the plurality of moving bodies. According to this mode, in at least one of the first case and the second case, a retraction control value for forming a positional relationship in which the detection target moving body can be detected without overlapping with other moving bodies in the two or more moving bodies can be prepared. And the prepared retraction control value can be sent to the control target moving body. Thus, when the position of the moving body is tracked using sensor information representing the moving body obtained by a sensor set at a place different from the moving body, the possibility of two or more moving bodies overlapping in the sensor information can be reduced. Therefore, the position of the moving body can be tracked more accurately using the sensor information.
[0022] The present disclosure can be implemented in various ways other than the above-mentioned remote control device, remote control system and remote control method. For example, it can be implemented in the form of a manufacturing method of a remote control device, a control method of a remote control device and a remote control system, a computer program for implementing the control method, a non-transitory recording medium recording the computer program, etc. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] Figure 1 It is a diagram showing a schematic configuration of a remote control system.
[0024] Figure 2 It is a diagram showing a schematic configuration of a remote control device.
[0025] Figure 3It is a diagram showing details of a CPU mounted on the remote control device.
[0026] Figure 4A This is a diagram showing an example of a method of setting a target distance.
[0027] Figure 4B This is a flowchart showing a processing procedure of the travel control of the vehicle in the first embodiment.
[0028] Figure 5 This is a flowchart showing a method of calculating the position of a detection target vehicle using a captured image.
[0029] Figure 6 It shows that the Figure 5 Schematic diagram showing examples of various images when the position calculation method is shown.
[0030] Figure 7 This is a diagram for explaining the details of the coordinate point calculation process.
[0031] Figure 8 This is a diagram for explaining a method of obtaining basic coordinate points.
[0032] Fig. 9 This is the first diagram for explaining the details of the position conversion process.
[0033] Fig.10 This is a second diagram for explaining the details of the position conversion process.
[0034] Fig.11 1 is a first flowchart showing the remote control method in the first embodiment.
[0035] Fig.12 1 is a second flowchart showing the remote control method in the first embodiment.
[0036] Fig.13 3 is a third flowchart showing the remote control method in the first embodiment.
[0037] Fig.14 This is a diagram for explaining a method of determining the classification of a vehicle.
[0038] Fig.15 1 is a first flowchart showing the remote control method in the second embodiment.
[0039] Fig.16 1 is a second flowchart showing the remote control method in the second embodiment.
[0040] Fig.17 3 is a third flowchart showing the remote control method in the second embodiment.
[0041] Fig.181 is a first flowchart showing the remote control method in the third embodiment.
[0042] Fig.19 1 is a second flowchart showing the remote control method in the third embodiment.
[0043] Fig. 20 is a third flow chart showing the remote control method in the third embodiment.
[0044] Fig.21 1 is a first flowchart showing the remote control method in the fourth embodiment.
[0045] Fig. 22 1 is a second flowchart showing the remote control method in the fourth embodiment.
[0046] Fig.23 It is a diagram showing details of a CPU mounted in the remote control device in the fifth embodiment.
[0047] Fig.24 This is a diagram showing a method of calculating the position of a detection target vehicle using external laser radar information.
[0048] Fig.25 1 is a first flowchart showing the remote control method in the sixth embodiment.
[0049] Fig.26 1 is a second flowchart showing the remote control method in the sixth embodiment. DETAILED DESCRIPTION
[0050] A. First Implementation Method:
[0051] A-1. Structure of remote control system:
[0052] Figure 1 1 is a diagram showing a schematic structure of a remote control system 1. The remote control system 1 includes: one or more vehicles 10 as mobile bodies, a remote control device 5 as a server for remotely controlling the driving action of at least one vehicle 10 among the multiple vehicles 10, and one or more sensors (hereinafter referred to as external sensors 9) installed in a location different from the multiple vehicles 10.
[0053] In the present disclosure, "mobile body" means an object that can move, such as a vehicle or an electric vertical take-off and landing aircraft (so-called flying car). The vehicle can be a vehicle that runs on wheels or a vehicle that runs on an infinite track, such as a passenger car, a truck, a bus, a two-wheeled vehicle, a four-wheeled vehicle, a chariot, a construction vehicle, etc. Vehicles include electric vehicles (BEV: Battery Electric Vehicle), gasoline vehicles, hybrid vehicles and fuel cell vehicles. In the case where the mobile body is other than a vehicle, the expressions "vehicle" and "car" in the present disclosure can be appropriately replaced with "mobile body", and the expression "running" can be appropriately replaced with "moving".
[0054] The vehicle 10 is configured to be able to travel by unmanned driving. "Unmanned driving" means driving that does not rely on driving operations of passengers. Driving operations mean operations related to at least one of "driving", "turning" and "stopping" of the vehicle 10. Unmanned driving is achieved by automatic or manual remote control using a device located outside the vehicle 10 or by autonomous control of the vehicle 10. Passengers who do not perform driving operations may also be on board the vehicle 10 that travels by unmanned driving. Passengers who do not perform driving operations include, for example, people who simply sit in the seats of the vehicle 10, and people who perform operations different from driving operations such as assembly, inspection, and switch operations while riding in the vehicle 10. In addition, driving that relies on driving operations of passengers is sometimes referred to as "manned driving".
[0055] In this specification, “remote control” includes “full remote control” in which all actions of the vehicle 10 are completely determined from outside the vehicle 10, and “partial remote control” in which part of the actions of the vehicle 10 are determined from outside the vehicle 10. In addition, “autonomous control” includes “full autonomous control” in which the vehicle 10 autonomously controls its own actions without receiving any information from a device outside the vehicle 10, and “partial autonomous control” in which the vehicle 10 autonomously controls its own actions using information received from a device outside the vehicle 10.
[0056] The external sensor 9 obtains information indicating at least one of the plurality of vehicles 10 (hereinafter referred to as sensor information). Specifically, the external sensor 9 obtains bird's-eye view information indicating the state of the vehicle 10 and the surrounding area of the vehicle 10. In the present embodiment, the external sensor 9 is a camera 90. The camera 90 obtains a captured image by capturing a detection range RG including the vehicle 10 from the outside of the vehicle 10. The camera 90 transmits the captured image to the remote control device 5 together with the camera identification information identifying the plurality of camera devices 90 and the time when the captured image is obtained. The captured image transmitted to the remote control device 5 is used to calculate the position of the vehicle 10. The captured image is a two-dimensional image formed by a collection of pixels arranged in the XcYc plane of the camera coordinate system. The camera coordinate system is a coordinate system having a coordinate axis represented by an Xc axis and a Yc axis orthogonal to the Xc axis and taking the focus of the camera 90 as the origin. The captured image includes at least two-dimensional data of the vehicle 10 as the target vehicle whose position is calculated. The captured image is preferably a color image, but may also be a gray image. The imaging device 90 is, for example, a camera including an imaging element such as a CCD image sensor or a CMOS image sensor and an optical system.
[0057] In the present embodiment, the camera 90 obtains an image of the travel path 6 and the vehicle 10 traveling on the travel path 6 from above. In order to use one or more camera devices 90 to capture the entire travel path 6, the installation position and the number of the camera devices 90 are determined in consideration of the detection range RG (angle of view) of the camera device 90 and the like. Specifically, each camera device 90 is installed so that the detection range RG of the first camera device 901, i.e., the first detection range RG1, and the detection range RG of the second camera device 902, i.e., the second detection range RG2, overlap each other. Furthermore, each camera device 90 is installed at a position where a measurement point 10e which is set in advance for a specific part of the vehicle 10 traveling on the travel path 6 can be captured. In the present embodiment, the measurement point 10e is the rear end (hereinafter referred to as the left rear end) of the left side of the vehicle 10. In addition, the measurement point 10e may be other than the left rear end of the vehicle 10. In addition, the camera device 90 is not limited to the information from above the vehicle 10, and may also obtain information from the front, rear, side, etc. of the vehicle 10.
[0058] The vehicle 10 has a manned driving mode and a remote unmanned driving mode. In the manned driving mode, the driving conditions of the vehicle 10 are generated by the driver riding in the vehicle 10 operating the input devices such as the steering wheel and accelerator provided in the vehicle 10. Thus, the vehicle 10 travels according to the generated driving conditions. The driving conditions are conditions that define the driving action of the vehicle 10. The driving conditions include, for example, the driving route, position, driving speed, acceleration, and deflection angle of the wheels of the vehicle 10. In the remote unmanned driving mode, the vehicle 10 receives the driving conditions of the vehicle 10 from the outside of the vehicle 10 and travels based on the received driving conditions. The remote unmanned driving mode has a remote manual driving mode and a remote automatic driving mode. In the remote manual driving mode, the driving conditions of the vehicle 10 are generated by the operator operating the operator input device provided in a place different from the vehicle 10. Thus, the vehicle 10 receives the driving conditions generated by the operator input device and travels according to the received driving conditions. In addition, in the remote automatic driving mode, the remote control device 5 provided in a place different from the vehicle 10 creates a control value that defines the driving action of the vehicle 10 and sends it to the vehicle 10. Thus, the vehicle 10 receives the control value and performs automatic driving according to the received control value.
[0059] The vehicle 10 is driven in a remote automatic driving mode, for example, in a factory that produces the vehicle 10 by executing a plurality of production processes. The factory is not limited to being in one building, one site, or one address, but may be located across multiple buildings, multiple sites, or multiple addresses. In this case, the vehicle 10 is not limited to being driven on private roads, but may be driven on public roads. In addition, the vehicle 10 may be driven in a remote automatic driving mode outside the factory.
[0060] The vehicle 10 is, for example, an electric vehicle, a hybrid vehicle, a fuel cell vehicle, a gasoline vehicle, or a diesel vehicle. The vehicle 10 may be a household vehicle such as a passenger car, or a commercial vehicle such as a truck, a bus, or a construction vehicle. The vehicle 10 may be at least one of a finished product, a semi-finished product, and a work in progress.
[0061] The vehicle 10 includes: a driving device 110 for accelerating the vehicle 10, a steering device 120 for changing the traveling direction of the vehicle 10, and a braking device 130 for decelerating the vehicle 10. The vehicle 10 also includes a vehicle communication unit 140 and a vehicle control device 150 for communicating with external devices using wireless communication or the like. The external devices are other devices other than the vehicle 10, such as the remote control device 5 and the camera 90, and other vehicles 10. The vehicle communication unit 140 is, for example, a wireless communication device. The vehicle communication unit 140 communicates with external devices connected to the network N, for example, via an access point in a factory. The vehicle control device 150 includes: a CPU, a storage unit, and an input / output interface. The CPU, the storage unit, and the input / output interface are connected to each other via an internal bus and an interface circuit. The input / output interface communicates with internal devices such as the driving device 110 mounted on the vehicle 10. The input / output interface is connected to the vehicle communication unit 140 in a manner that allows communication. In addition, the structure of the vehicle 10 is not limited to the above. In addition, at least a part of the functions of the vehicle control device 150 may be realized as one function of the remote control device 5 or the external sensor 9 .
[0062] Figure 2 : is a diagram showing a schematic structure of a remote control device 5 serving as a server. The remote control device 5 uses sensor information to control the driving action of the vehicle 10. In the present embodiment, the remote control device 5 uses sensor information, i.e., a captured image, obtained by a camera 90 serving as an external sensor 9 to calculate the position of a detection target vehicle 10a included in the captured image. The detection target vehicle 10a is a vehicle 10 that is a detection target based on the sensor information. The detection target vehicle 10a may be set in advance, or may be appropriately set based on at least one of data generated in the process of calculating the position of the vehicle 10 and a calculation result of the position of the vehicle 10. In the present embodiment, the detection target vehicle 10a is a vehicle 10 at least part of which is present in a predetermined recognition target area. The recognition target area is, for example, the area described later. Figure 6 The area with grid lines 61 is shown. The remote control device 5 adopts the position of the predetermined measuring point 10e of the detection target vehicle 10a as the position of the detection target vehicle 10a. And, the remote control device 5 uses the calculated driving conditions such as the position of the detection target vehicle 10a to create a control value that specifies the driving action of the control target vehicle 10b. The control target vehicle 10b is a vehicle 10 included in the sensor information and is a vehicle 10 that is the control target of the driving action. The control target vehicle 10b and the detection target vehicle 10a may be the same or different. And, the remote control device 5 sends the created control value to the control target vehicle 10b. Thereby, the remote control device 5 controls the driving action of the control target vehicle 10b included in the captured image.
[0063] The remote control device 5 includes a remote communication unit 51, a storage unit 53, and a CPU 52. The remote communication unit 51, the storage unit 53, and the CPU 52 are connected to each other via an internal bus and an interface circuit. The remote communication unit 51 connects the vehicle control device 150 and the camera 90 to the remote control device 5 in a communicative manner. The remote communication unit 51 is, for example, a wireless communication device.
[0064] The storage unit 53 of the remote control device 5 stores various information including various programs for controlling the operation of the remote control device 5, a position detection model Md, distortion correction parameters Pa1, perspective transformation parameters Pa2, a camera database D1, and a vehicle model database D2. The storage unit 53 includes, for example, a RAM, a ROM, and a hard disk drive (HDD).
[0065] The position detection model Md is a machine learning model that has been learned in order to determine the position of the detection target vehicle 10a included in the captured image. In the present embodiment, the position detection model Md is a machine learning model that has been pre-learned by masking the vehicle 10 in the input image by inputting a captured image and any one of various images after various processing is performed on the captured image. As an algorithm of the position detection model Md, for example, a deep neural network (hereinafter referred to as DNN) having a structure of a convolutional neural network (hereinafter referred to as CNN) that implements semantic segmentation and instance segmentation is used. As an example of a DNN used as an algorithm of the position detection model Md, a DNN such as YOLACT++ that performs instance segmentation can be cited. In addition, the structure of the position detection model Md is not limited to the above. The position detection model Md may also be a machine learning model that has been learned using an algorithm other than a neural network.
[0066] The distortion correction parameter Pa1 is a parameter used when correcting the distortion of the captured image. The perspective transformation parameter Pa2 is a parameter used when performing perspective transformation on the first mask image. The first mask image is an image in which a mask area is added to the vehicle 10 in the image by masking the vehicle 10 included in the image input to the position detection model Md. The correction parameter Pa1 and the perspective transformation parameter Pa2 will be described in detail later.
[0067] The camera database D1 is a database that shows the imaging parameters calculated based on the installation position of the imaging device 90 in the global coordinate system for each imaging device 90. In this embodiment, the imaging parameters are the height H (described later) of the imaging device 90 from the road surface 60. Fig. 9), is the height H relative to the road surface 60 where the detection target vehicle 10a is located. The camera database D1 is, for example, a database that associates camera identification information with shooting parameters of the shooting device 90 determined by the camera identification information. In addition, the structure of the camera database D1 is not limited to the above. In the camera database D1, the detection range RG, the installation position, the installation angle, etc. of the shooting device 90 may also be associated with the camera identification information.
[0068] The vehicle model database D2 is a database containing shape information. The shape information is information showing the appearance shape of the vehicle 10 for each category of the vehicle 10. The shape information is, for example, information that establishes an association between vehicle model identification information for identifying multiple categories of the vehicle 10 and the appearance shape of the vehicle 10 of the category determined by the vehicle model identification information. The appearance shape of the vehicle 10 is different for each category of the vehicle 10 due to, for example, the body shape of the vehicle 10 and the vehicle grade (also referred to as "vehicle size") determined by the total length, width and height of the vehicle 10. In addition, the structure of the vehicle model database D2 is not limited to the above. At least a part of the various information stored in the storage unit 53 may also be stored in the storage unit of the vehicle control device 150, the camera 90, etc.
[0069] Figure 3 1 is a diagram showing details of the CPU 52 mounted in the remote control device 5. The CPU 52 expands various programs stored in the storage unit 53 to function as an acquisition unit 521, a position calculation unit 522, a position conversion unit 523, an actual distance calculation unit 524, a determination unit 525, a control value creation unit 526, and a transmission unit 527.
[0070] The acquisition unit 521 acquires various information. For example, the acquisition unit 521 acquires sensor information acquired by the external sensor 9. In the present embodiment, the acquisition unit 521 acquires a captured image acquired by the imaging device 90 as the external sensor 9 as the sensor information. In addition, the acquisition unit 521 acquires the imaging parameters of the imaging device 90 as the acquisition source of the captured image to be analyzed, for example, by referring to the camera database D1 stored in the storage unit 53.
[0071] The position calculation unit 522 calculates the image coordinate point using the captured image. The image coordinate point is a coordinate point indicating the position of the detection target vehicle 10a in the image coordinate system described later. The position calculation unit 522 includes: a distortion correction unit 522a, a rotation processing unit 522b, a cropping processing unit 522c, a detection unit 522d, a deletion unit 522e, a perspective transformation unit 522f, and a coordinate point calculation unit 522g. The distortion correction unit 522a generates a corrected image in which the distortion of the captured image is corrected. The rotation processing unit 522b generates a rotated image in which the corrected image is rotated so that the direction of the vector indicating the moving direction of the detection target vehicle 10a (hereinafter, referred to as the moving vector) becomes a predetermined direction. The cropping processing unit 522c deletes the area (hereinafter, referred to as the unnecessary area) other than the area (hereinafter, referred to as the necessary area) including the vehicle area corresponding to the detection target vehicle 10a and the predetermined surrounding area of the detection target vehicle 10a from the rotated image. Thus, the cropping processing unit 522c generates a processed image in which the required area is cropped from the rotated image. In the present embodiment, when the detection object vehicle 10a moves a distance exceeding a predetermined threshold, the cropping processing unit 522c deletes the moved area corresponding to the distance moved by the detection object vehicle 10a as an unnecessary area from the rotated image. Thus, the cropping processing unit 522c generates a processed image in which the unmoved area including the detection object vehicle 10a is cropped from the rotated image as the required area. The detection unit 522d uses the position detection model Md to detect the vehicle included in the processed image and generates a first mask image. When a plurality of vehicles 10 are included in the first mask image, the deletion unit 522e deletes the mask area of other vehicles 10 (hereinafter referred to as non-target vehicles 10f) other than the detection object vehicle 10a from the first mask image. The non-target vehicle 10f is, for example, a vehicle 10 included in the captured image or predicted to be included in the captured image, and the measuring point 10e cannot be detected because other vehicles 10 overlap in the captured image so as to cover the measuring point 10e. In addition, the non-target vehicle 10f may be, for example, a vehicle 10 that exists outside the recognition target area. The perspective transformation unit 522f generates a second mask image by perspective transformation of the first mask image. The coordinate point calculation unit 522g calculates the image coordinate point by correcting the first coordinate point using the second coordinate point. The first coordinate point is a coordinate point in the image coordinate system of a specified vertex of a first circumscribed rectangle set for the mask area in the first mask image. The second coordinate point is a coordinate point in the image coordinate system of a vertex that indicates the same position as the first coordinate point among the vertices of the second circumscribed rectangle set for the mask area in the second mask image. In addition, the structure of the position calculation unit 522 is not limited to the above.
[0072] The position conversion unit 523 converts the image coordinate point into a vehicle coordinate point using the imaging parameter acquired by the acquisition unit 521 and the image coordinate point calculated by the position calculation unit 522. The vehicle coordinate point is a coordinate point indicating the position of the detection target vehicle 10a in the global coordinate system.
[0073] The actual distance calculation unit 524 calculates the actual distance (hereinafter referred to as the actual distance L1) between the first vehicle 101 and the second vehicle 102, which are vehicles 10 included in the sensor information. The actual distance calculation unit 524 calculates the actual distance L1 between the first vehicle 101 and the second vehicle 102 using, for example, information obtained by on-board sensors mounted on the first vehicle 101 and the second vehicle 102. The on-board sensors are, for example, on-board cameras, on-board radars, and on-board laser radars. Figure 1 In the example shown, the second vehicle 102 travels in the same lane as the first vehicle 101 and in the same direction of travel behind the first vehicle 101, but the present invention is not limited to the above. The second vehicle 102 may be a vehicle 10 traveling in front of the first vehicle 101, or may be a vehicle 10 traveling in a different lane from the first vehicle 101. In addition, the method for calculating the actual distance L1 between the first vehicle 101 and the second vehicle 102 is not limited to the above. The actual distance calculation unit 524 may also calculate the actual distance L1 between the first vehicle 101 and the second vehicle 102 by analyzing an image such as a second mask image, for example. In addition, the actual distance calculation unit 524 may also calculate the actual distance L1 between the first vehicle 101 and the second vehicle 102 using, for example, the vehicle coordinate points, the travel speed, and the elapsed time from the time when the captured image used to calculate the vehicle coordinate points is obtained.
[0074] The judgment unit 525 judges whether at least one of the first situation and the second situation is met. The first situation is a situation where two or more vehicles 10 among a plurality of vehicles overlap in the sensor information. The judgment unit 525 judges whether the first situation is met by, for example, analyzing the sensor information acquired by the acquisition unit 521. In the case where the sensor information is a captured image as in the present embodiment, the judgment unit 525 judges whether two or more vehicles 10 overlap, for example, using an overlap detection model. The overlap detection model is a machine learning model that has been learned in advance in a manner that can judge whether two or more vehicles 10 overlap in the input image by inputting a captured image and any one of various images after various processing is performed on the captured image. As an algorithm of the overlap detection model, for example, a DNN having a CNN structure that realizes instance segmentation is used. The overlap detection model can also be implemented as a function of the position detection model Md. In addition, the judgment unit 525 can also use other methods different from the above-mentioned method to judge whether the first situation is met. The judgment unit 525 can also use at least one of the data generated in the process of calculating the position of the vehicle 10 and the calculation result of the position of the vehicle 10 to judge whether the first situation is met.
[0075] The second situation is a situation where it is predicted that two or more vehicles 10 will overlap when the sensor information is assumed to be obtained before the sensor information is obtained. The judgment unit 525 determines whether the second situation is met, for example, based on the categories of the two or more vehicles 10 that are predicted to be included in the sensor information when the sensor information is obtained. Specifically, first, the judgment unit 525 determines whether it is predicted that two or more vehicles 10 will be included in the sensor information when the sensor information is obtained, for example, based on driving conditions such as the driving speed of each vehicle 10 and the position at a predetermined time point. Then, when it is predicted that two or more vehicles 10 will be included in the sensor information, the judgment unit 525 determines the vehicle type identification information indicating the category of each of the two or more vehicles 10 that are predicted to be included in the sensor information. At this time, the judgment unit 525 uses, for example, predetermined information to determine the category of each of the two or more vehicles 10 that are predicted to be included in the sensor information. The predetermined information is information indicating which category of vehicle 10 is scheduled to travel at which position on the predetermined target driving route at which time point. Then, the judgment unit 525 obtains the shape information of the vehicle 10 of the category determined by the vehicle type identification information. At this time, the projection area of each vehicle 10 from the shooting direction of the camera 90, that is, the area occupied by each vehicle 10 on the captured image Im1, is related to the appearance shape of the vehicle 10. For example, when the sensor information is obtained, the greater the vehicle grade, the higher the possibility that more than one vehicle 10 exists in the sensor information in an overlapping manner, and the smaller the vehicle grade, the lower the possibility that more than two vehicles 10 exist in the sensor information in an overlapping manner. Thus, the judgment unit 525 uses the difference in appearance shape corresponding to the category of the vehicle 10 to determine whether the second situation is met. Specifically, the judgment unit 525 determines that the second situation is met, for example, when any one of the total values of the total length, vehicle width and vehicle height of each vehicle 10 is greater than a predetermined vehicle grade threshold. On the other hand, when the total values of the total length, vehicle width and vehicle height of each vehicle 10 are all less than the vehicle grade threshold, the judgment unit 525 determines that the second situation is not met.
[0076] In addition, the judgment unit 525 may also use a method different from the above-mentioned method to determine whether the second situation is met. For example, the judgment unit 525 may also determine that the second situation is met when the sum of the total values of the total length, vehicle width and vehicle height of each vehicle 10, that is, the cumulative value is above a predetermined cumulative threshold. In this case, the judgment unit 525 determines that the second situation is not met when the cumulative value is less than the cumulative threshold. Even in this way, it is possible to determine whether the second situation is met. In addition, the judgment unit 525 may also use information related to the camera 90, such as the detection range RG, setting position and setting angle of the camera 90, in addition to the above-mentioned information to determine whether the second situation is met. In this way, it is possible to more accurately determine whether the second situation is met.
[0077] In addition, the judgment unit 525 may determine the category of each of the two or more vehicles 10 predicted to be included in the sensor information using a method different from the above-mentioned method. For example, the judgment unit 525 may determine the category of each of the two or more vehicles 10 predicted to be included in the sensor information using sensor information that has been obtained by an external sensor 9 located at the rear side of the object sensor in the direction of travel of the vehicle 10. The object sensor is an external sensor 9 that is predetermined to obtain sensor information of the object of judgment of whether the second situation is met. In this way, the category of the vehicle 10 can be determined using past sensor information of two or more vehicles 10 that contain sensor information predicted to be included in the object of judgment of whether the second situation is met. In addition, the judgment unit 525 is not an essential function in the CPU 52. The control value generation unit 526 may also identify whether at least one of the first situation and the second situation is met, for example, by receiving input from the user via the input operation unit.
[0078] The control value generator 526 generates a travel control signal, i.e., a control value, that specifies the travel action of at least one of the plurality of vehicles 10 using the sensor information. The control value generator 526 generates a retreat control value in order to form a positional relationship in which the detection target vehicle 10a among the two or more vehicles 10 can be detected by the detector 522d without overlapping with other vehicles 10 other than the detection target vehicle 10a among the plurality of vehicles 10 in at least one of the first case and the second case. The retreat control value is a control value that specifies the travel action of at least one of the two or more vehicles 10 in order to form a positional relationship in which the detection target vehicle 10a can be detected by the detector 522d without overlapping with other vehicles 10. In other words, the retreat control value is a control value for causing the external sensor 9 to obtain sensor information in a state in which the detection target vehicle 10a does not overlap with other vehicles 10. In the present embodiment, the control value generator 526 identifies whether at least one of the first case and the second case is met based on the judgment result of the judgment unit 525.
[0079] The avoidance control value in the first case is a control value that specifies the travel action of at least one of the first vehicle 101 and the second vehicle 102 so that the actual distance L1 between the first vehicle 101 and the second vehicle 102 that overlap in the sensor information becomes a target distance L2 that is longer than the actual distance L1 at the time point when the first case is determined to have occurred. The avoidance control value in the second case is a control value that specifies the travel action of at least one of the first vehicle 101 and the second vehicle 102 so that the actual distance L1 between the first vehicle 101 and the second vehicle 102 that are predicted to overlap in the sensor information when the sensor information is acquired becomes a target distance L2 that is longer than the actual distance L1 at the time point when the second case is determined to have occurred.
[0080] The target distance L2 is set, for example, based on the actual distance L1 between the first vehicle 101 and the second vehicle 102 that exist in an overlapping manner or are predicted to exist in an overlapping manner in the sensor information, the appearance shape of the first vehicle 101, and the appearance shape of the second vehicle 102. The larger the appearance shape of the first vehicle 101 and the appearance shape of the second vehicle 102, the longer the target distance L2. On the other hand, the smaller the appearance shape of the first vehicle 101 and the appearance shape of the second vehicle 102, the shorter the target distance L2. The following describes a case where the control value production unit 526 sets the target distance L2, but the target distance L2 may also be set by a unit other than the control value production unit 526. The control value production unit 526 may also recognize the target distance L2 set by the user, for example, by receiving input from the user via the input operation unit.
[0081] Figure 4A FIG. 1 is a diagram showing an example of a method for setting the target distance L2. Figure 4A The first vehicle 101 and the second vehicle 102 are vehicles 10 that overlap in sensor information or are predicted to overlap when sensor information is acquired. The control value generator 526 sets the target distance L2 using the actual distance L1 between the first vehicle 101 and the second vehicle 102, the camera database D1, and the vehicle model database D2.
[0082] The control value generating unit 526 acquires the coordinate value indicating the position of the imaging device 90 in the global coordinate system as information indicating the installation position of the imaging device 90 as the acquisition source of the captured image Im1. In the first case, the control value generating unit 526 refers to the camera database D1 to acquire the height H of the imaging device 90 as the acquisition source of the captured image Im1 from the road surface 60. The control value generating unit 526 acquires the height H of the imaging device 90 as the acquisition source of the captured image Im1 from the road surface 60 by acquiring, for example, a shooting parameter associated with the camera identification information transmitted together with the captured image Im1 from the camera database D1. In the second case, the control value generating unit 526 acquires the height H of the imaging device 90 as the acquisition source of the captured image Im1 from the road surface 60 for the imaging device 90 that is scheduled to acquire the sensor information predicting that the first vehicle 101 and the second vehicle 102 will overlap. In addition, the control value generating unit 526 acquires the height hg of the rear end portion (hereinafter referred to as the upper rear end portion 10g) on the roof side of the first vehicle 101 traveling on the front side of the second vehicle 102 in the traveling direction from the road surface 60.
[0083] Next, the control value generating unit 526 calculates the position of the upper rear end portion 10g of the first vehicle 101 using, for example, the relative positional relationship between the upper rear end portion 10g of the first vehicle 101 and the measuring point 10e and the position of the measuring point 10e of the first vehicle 101. Figures 5 to 10 The position of the first vehicle 101 calculated by the position calculation method shown is used as the position of the measuring point 10e of the first vehicle 101. The control value generating unit 526 may use the estimated position of the first vehicle 101 estimated using the travel speed and travel time of the first vehicle 101 as the position of the measuring point 10e of the first vehicle 101.
[0084] Next, the control value generating unit 526 calculates the required distance L3 using the similarity relationship between the height H of the camera 90 from the road surface 60 and the height hg of the upper rear end 10g of the first vehicle 101 from the road surface 60. The required distance L3 is the minimum inter-vehicle distance between the first vehicle 101 and the second vehicle 102 required to form a positional relationship in which the detection target vehicle 10a can be detected by the detection unit 522d without overlapping with other vehicles 10. When the required distance L3 is calculated using the similarity relationship between the height H of the camera 90 from the road surface 60 and the height hg of the upper rear end 10g of the first vehicle 101 from the road surface 60, the control value generating unit 526 performs the following processing, for example. The control value generating unit 526 calculates the interval distance Lt between the first intersection point Q1, which is the intersection point of the first perpendicular line J1 and the road surface 60, and the second intersection point Q2, which is the intersection point of the second perpendicular line J2 and the road surface 60, along the traveling direction of the first vehicle 101. The first perpendicular line J1 is a perpendicular line from the front end 10i of the vehicle 10 to the road surface 60. The second perpendicular line J2 is a perpendicular line from the camera 90 to the road surface 60. Moreover, the control value generating unit 526 obtains the total length Lf of the vehicle 10 that is close to the distance between the first intersection Q1 and the third intersection Q3. The third intersection Q3 is the intersection of the third perpendicular line J3 and the road surface 60. The third perpendicular line J3 is a perpendicular line from the upper rear end 10g of the vehicle 10 to the road surface 60. At this time, as shown in the following formula (1), the similarity ratio of the height hg of the upper rear end 10g of the vehicle 10 from the road surface 60 and the height H of the camera 90 from the road surface 60 is equal to the similarity ratio of the required distance L3 and the total distance Lp obtained by adding the distance Lt between the first intersection Q1 and the second intersection Q2 and the total length Lf of the vehicle 10 to the required distance L3.
[0085] hg:H=L3:Lp=L3:(Lt+Lf+L3) Formula (1)
[0086] Thus, the control value generating unit 526 calculates the required distance L3 by substituting the calculated or acquired values into the above-mentioned equation (1).
[0087] Next, the control value making unit 526 obtains the actual distance L1 between the first vehicle 101 and the second vehicle 102. And, when the actual distance L1 between the first vehicle 101 and the second vehicle 102 is less than the required distance L3, the control value making unit 526 sets the calculated required distance L3 as the target distance L2 to make the retreat control value. On the other hand, when the actual distance L1 between the first vehicle 101 and the second vehicle 102 is greater than the required distance L3, the control value making unit 526 sets, for example, an arbitrary distance greater than the actual distance L1 as the target distance L2. In this way, it is possible to set an appropriate target distance L2 according to the driving condition of the vehicle 10. In addition, when the actual distance L1 between the first vehicle 101 and the second vehicle 102 is greater than the required distance L3, the control value making unit 526 may set an arbitrary distance greater than the actual distance L1 as the target distance L2. In this way, it is possible to further reduce the possibility that the first vehicle 101 and the second vehicle 102 exist overlapping in the sensor information.
[0088] The method for setting the target distance L2 is not limited to the above. The control value generating unit 526 may also set the target distance L2 using information related to the imaging device 90 that is the source of the image Im1, such as the detection range RG, the installation position, and the installation angle of the imaging device 90. In this way, a more appropriate target distance L2 can be set.
[0089] The avoidance control value includes, for example, a deflection angle control value that specifies the deflection angle of the control object vehicle 10b. In this way, the actual distance L1 between the first vehicle 101 and the second vehicle 102 can be changed to the target distance L2 by moving either one of the first vehicle 101 and the second vehicle 102 in the width direction of the travel path 6. Moving the vehicle 10 in the width direction of the travel path 6, for example, refers to moving the vehicle 10 to at least one of the shoulder and the side road set beside the travel path 6 and moving the vehicle 10 to other lanes adjacent to the driving lane. In this way, it is possible to avoid other vehicles 10 other than the detection object vehicle 10a from becoming an obstacle when detecting the detection object vehicle 10a. In addition, when the first vehicle 101 and the second vehicle 102 are respectively traveling along the target travel route, in the process of changing the actual distance L1 between the first vehicle 101 and the second vehicle 102 to the target distance L2, the position deviation relative to the target travel route can be corrected.
[0090] In the case where the second vehicle 102 is traveling in the same forward direction as the first vehicle 101 behind the first vehicle 101, the avoidance control value may also include the following control value. In this case, the acceleration control value for specifying the acceleration of the control object vehicle 10b may also be included. In this way, the actual distance L1 between the first vehicle 101 and the second vehicle 102 can be changed to the target distance L2 by the second vehicle 102 exceeding the first vehicle 101. Thus, for example, in the case where the first vehicle 101 traveling on the front side is desired to be used as the detection object vehicle 10a, the second vehicle 102 traveling on the rear side can be made to exceed the first vehicle 101 to avoid the second vehicle 102 from becoming an obstacle when detecting the first vehicle 101. In addition, the moving speed of either the first vehicle 101 or the second vehicle 102 when moving in the width direction of the travel road 6 can be adjusted.
[0091] Moreover, the avoidance control value may also replace the above-mentioned control value and include, for example, the following control value. In this case, the avoidance control value may include, for example, at least one of an acceleration control value that increases the acceleration in the forward direction of the first vehicle 101, an acceleration control value that decreases the acceleration in the forward direction of the second vehicle 102, a direction control value that causes the second vehicle 102 to retreat in a direction opposite to the forward direction, and a stop control value that stops the driving action of the second vehicle 102. In this way, the distance between the first vehicle 101 and the second vehicle 102 in the direction along the forward direction can be made greater than the time point at which at least one of the first situation and the second situation is reached. Thus, the actual distance L1 between the first vehicle 101 and the second vehicle 102 can be changed to the target distance L2. Therefore, it is possible to avoid other vehicles 10 other than the detection target vehicle 10a from becoming an obstacle when detecting the detection target vehicle 10a.
[0092] In addition, the types of control values included in the back-off control value as the driving control signal are not limited to the above. The back-off control value may also be a control value including any one of the destination control value and the track control value instead of the acceleration control value, the deflection angle control value, the direction control value and the stop control value. The destination control value is a control value that specifies the position of the control object vehicle 10b at the time point of ending the control based on the back-off control value. Thus, the destination control value has a target arrival time and a target arrival place of the control object vehicle 10b. By making a back-off control value including the destination control value, the control object vehicle 10b can be moved to the target arrival place at a desired time without specifying detailed driving actions such as acceleration and deflection angle. The track control value is a control value that specifies the driving trajectory by arranging the target driving positions of the control object vehicle 10b at predetermined times in a time series order. By making a back-off control value including the track control value, the driving trajectory of the control object vehicle 10b can be specified without specifying detailed driving actions such as acceleration and deflection angle.
[0093] The control value making unit 526 may also make a control value other than the backoff control value as a driving control signal. The control value making unit 526 may, for example, make a reference control value and a correction control value. The reference control value is a control value for causing the vehicle 10 to travel along a predetermined target driving route. The correction control value is a control value for correcting the relative position of the vehicle 10 relative to the target driving route. In addition, in the case where the vehicle control device 150 receives either the reference control value or the correction control value and controls the movement of the vehicle 10 according to the received control value, the control value making unit 526 may also make the backoff control value as a control value replacing the reference control value and the correction control value. In this way, when the vehicle 10 is traveling based on a control value other than the backoff control value at a time point when at least one of the first situation and the second situation occurs, the movement of the traveling vehicle 10 can be changed to a driving movement corresponding to the backoff control value.
[0094] Figure 3 The transmitter 527 shown transmits various information. For example, the transmitter 527 transmits the control value generated by the control value generator 526 to the control target vehicle 10b. In addition, the structure of the CPU 52 is not limited to the above. At least part of the function of the CPU 52 can also be implemented as a function of the vehicle control device 150 or the camera 90.
[0095] Figure 4B: is a flowchart showing the processing steps of the driving control of the vehicle 10 in the first embodiment. In step S901, the server 5 obtains the vehicle position information using the detection result output from the sensor located outside the vehicle 10, that is, the external sensor. The vehicle position information is the position information that becomes the basis for generating the driving control signal. In the present embodiment, the vehicle position information includes the position and orientation of the vehicle 10 in the reference coordinate system of the factory. In the present embodiment, the reference coordinate system of the factory is the global coordinate system, and any position in the factory can be represented by the coordinates of X, Y, and Z in the global coordinate system. In the present embodiment, the external sensor is a camera installed in the factory, and the captured image is output from the external sensor as the detection result. That is, in step S901, the server 5 obtains the vehicle position information using the captured image obtained from the camera as the external sensor.
[0096] Specifically, in step S901, the server 5 detects the shape of the vehicle 10 from the captured image, calculates the coordinates of the measuring point of the vehicle 10 in the coordinate system of the captured image, that is, the local coordinate system, and transforms the calculated coordinates into coordinates in the global coordinate system, thereby acquiring the position of the vehicle 10. The shape of the vehicle 10 included in the captured image can be detected by, for example, inputting the captured image to a detection model that makes use of artificial intelligence. The detection model is prepared, for example, in the system 1 or outside the system 1, and is pre-stored in the memory of the server 5. As the detection model, for example, a machine learning model that has been learned in a manner that realizes either semantic segmentation or instance segmentation can be cited. As the machine learning model, for example, a convolutional neural network (hereinafter referred to as CNN) that has been learned by supervised learning using a learning data set can be used. The learning data set, for example, includes a plurality of training images including the vehicle 10 and labels indicating whether each region in the training image is a region representing the vehicle 10 or a region other than the vehicle 10. When learning the CNN, it is preferable to update the parameters of the CNN by back propagation (error back propagation method) in a manner that reduces the error between the output result of the detection model and the label. The server 5 can acquire the orientation of the vehicle 10 by estimating the orientation of the vehicle 10 based on the orientation of the movement vector of the vehicle 10 calculated from the position change of the feature point of the vehicle 10 between frames of the captured image, for example, using the optical flow method.
[0097] In step S902, the server 5 determines the target location to which the vehicle 10 should go next. In the present embodiment, the target location is represented by the coordinates of X, Y, and Z in the global coordinate system. The path that the vehicle 10 should travel, i.e., the reference path, is pre-stored in the memory of the server 5. The path is represented by a node representing a departure point, a node representing a passing point, a node representing a destination, and a road segment connecting each node. The server 5 uses the vehicle position information and the reference path to determine the target location to which the vehicle 10 should go next. The server 5 determines the target location on the reference path that is ahead of the current location of the vehicle 10.
[0098] In step S903, the server 5 generates a driving control signal for driving the vehicle 10 toward the determined target position. In the present embodiment, the driving control signal includes the acceleration and steering angle of the vehicle 10 as parameters. The server 5 calculates the driving speed of the vehicle 10 according to the change of the position of the vehicle 10, and compares the calculated driving speed with the target speed. In general, the server 5 determines the acceleration in a manner to accelerate the vehicle 10 when the driving speed is lower than the target speed, and determines the acceleration in a manner to decelerate the vehicle 10 when the driving speed is higher than the target speed. In addition, when the vehicle 10 is on the reference path, the server 5 determines the steering angle and acceleration in a manner to prevent the vehicle 10 from deviating from the reference path, and when the vehicle 10 is not on the reference path, in other words, when the vehicle 10 deviates from the reference path, determines the steering angle and acceleration in a manner to return the vehicle 10 to the reference path. In other embodiments, the driving control signal may also include the speed of the vehicle 10 as a parameter instead of the acceleration of the vehicle 10 or on this basis.
[0099] In step S904, the server 5 transmits the generated travel control signal to the vehicle 10. The server 5 repeatedly acquires vehicle position information, determines a target position, generates a travel control signal, transmits the travel control signal, and the like in a predetermined cycle.
[0100] In step S905, the vehicle 10 receives the travel control signal transmitted from the server 5. In step S906, the vehicle 10 controls the actuator of the vehicle 10 using the received travel control signal to drive the vehicle 10 at the acceleration and steering angle indicated by the travel control signal. The vehicle 10 repeatedly receives the travel control signal and controls the actuator of the vehicle 10 in a predetermined cycle. According to the system 1 in this embodiment, the vehicle 10 can be driven by remote control, and the vehicle 10 can be moved without using a transport device such as a crane or a conveyor.
[0101] A-2. Method for calculating the position of the detection target vehicle:
[0102] Figure 5This is a flowchart showing an example of a method of calculating the position of the detection target vehicle 10 a using one captured image acquired by one imaging device 90 . Figure 5 The method shown is repeatedly executed at predetermined time intervals after the detection target vehicle 10a starts traveling in the remote automatic driving mode, for example. Figure 5 To be described later Fig.10 First, a method for calculating the position of the detection target vehicle 10a when neither the first situation nor the second situation is met will be described. Figure 5 To be described later Fig.10 In the position calculation method shown in FIG. 1 , the second vehicle 102 is the detection target vehicle 10a. The first vehicle 101 is the non-detection target vehicle 10f. Figure 6 In the illustrated example, the first vehicle 101 is determined to be an out-of-target vehicle 10 f because it is outside the recognition target area.
[0103] Figure 6 It shows that the Figure 5 Schematic diagram of various image examples when the position calculation method is shown. Figure 6 In the Figure 5 The step numbers for each step in Figures 5 to 10 In the position calculation method shown, a case where the first vehicle 101 and the second vehicle 102 move (travel) in the same direction of travel on a road 60 on which mesh-shaped grid lines 61 are drawn along an Xg axis parallel to the travel directions of the first vehicle 101 and the second vehicle 102 and a Yg axis orthogonal to the Xg axis is taken as an example for explanation. The Xg axis and the Yg axis are coordinate axes in the global coordinate system. In addition, in other embodiments, the grid lines 61 may be omitted.
[0104] like Figure 5 As shown in FIG. 1 , in the position calculation method, first, an image acquisition process (step S1) is performed. Figure 6 As shown, the image acquisition step is a step of acquiring a captured image Im1 including the detection target vehicle 10 a . In the image acquisition step, the acquisition unit 521 acquires the captured image Im1 acquired by the imaging device 90 .
[0105] like Figure 5 As shown, the position calculation step is executed after the image acquisition step. The position calculation step is a step of calculating the image coordinate point about the detection target vehicle 10a using the captured image Im1 acquired in the image acquisition step.
[0106] In the position calculation process, first, a distortion correction process is performed (step S2). Figure 6As shown, the distortion correction process is a process of correcting the distortion of the captured image Im1. In the distortion correction process, the distortion correction unit 522a generates a corrected image Im2 by correcting the distortion of the captured image Im1. Specifically, the distortion correction unit 522a corrects the distortion of the captured image Im1 using, for example, a distortion correction parameter Pa1 pre-stored in the storage unit 53. The distortion correction parameter Pa1 is, for example, a parameter related to the position information of the grid line 61 obtained by calibration. In addition, the distortion correction method is not limited to the above. The distortion correction parameter Pa1 may also be any parameter other than the above.
[0107] like Figure 5 As shown in FIG. 1 , after the distortion correction step, a rotation processing step (step S3) is performed. Figure 6 As shown, the rotation processing step is a step of rotating the correction image Im2 in such a manner that the direction of the motion vector of the detection target vehicle 10a included in the correction image Im2 is oriented toward a predetermined direction. In the rotation processing step, the rotation processing unit 522b rotates the correction image Im2 in such a manner that the direction of the motion vector of the detection target vehicle 10a included in the correction image Im2 is oriented toward a predetermined direction. Thus, the rotation processing unit 522b generates a rotated image Im3. Specifically, the rotation processing unit 522b rotates the correction image Im2 with the center of gravity of the detection target vehicle 10a in the correction image Im2 as the rotation center, for example, in such a manner that the direction of the motion vector of the detection target vehicle 10a on the screen of the display device displaying the correction image Im2 is oriented upward. The movement of the feature point (for example, the center of gravity) of the detection target vehicle 10a can be expressed as the direction of the motion vector by, for example, the optical flow method. The magnitude of the motion vector of the detection target vehicle 10a and the direction of the motion vector are estimated, for example, based on the change in the position between image frames of the feature point appropriately set on the correction image Im2. In addition, the rotation processing method is not limited to the above.
[0108] like Figure 5 As shown in FIG. 1 , after the rotation process, the cutting process (step S4) is performed. Figure 6As shown, the cropping process is a process of generating a processed image Im4 including a required area from the rotated image Im3. In the present embodiment, in the cropping process, when the detection object vehicle 10a moves a distance exceeding a predetermined threshold value, the cropping processing unit 522c deletes the moved area A2 corresponding to the distance moved by the detection object vehicle 10a as an unnecessary area from the rotated image Im3. Thus, the cropping processing unit 522c generates a processed image Im4 in which the unmoved area A1 including the detection object vehicle 10a is cropped out from the rotated image Im3 as a required area. At this time, the cropping processing unit 522c recognizes the distance moved by the detection object vehicle 10a, for example, by the estimated size of the movement vector of the detection object vehicle 10a, thereby estimating the moved area A2. In addition, in the position calculation process, it does not matter which of the rotation process and the cropping process is performed first. In addition, the cropping processing method is not limited to the above.
[0109] like Figure 5 As shown in FIG. 1 , after the cutting process, the detection process (step S5) is performed. Figure 6 As shown, the detection process is a process of detecting the outer shape (contour) of the vehicle 10 included in the processed image Im4 using the position detection model Md. In the detection process, the detection unit 522d inputs the processed image Im4 to the position detection model Md. Thus, the detection unit 522d detects the first vehicle 101 included in the processed image Im4 and the second vehicle 102 as the detection object vehicle 10a. In addition, the detection unit 522d generates a first mask image Im5 to which a first mask area M1 is added by masking the first vehicle 101 in the processed image Im4 and a second mask area M2 is added by masking the second vehicle 102 in the processed image Im4. In addition, the detection method of the outer shape of the vehicle 10 is not limited to the above.
[0110] like Figure 5 As shown in FIG. 1 , after the detection process, the deletion process (step S6) is performed. Figure 6 As shown in FIG. 1 , the deletion step is a step of deleting the mask region M1 of the other vehicles 10 other than the detection target vehicle 10a, that is, the non-target vehicle 10f, from the first mask image Im5. Figure 6In the example shown, in the deletion process, the deletion unit 522e deletes the first mask area M1 from the first mask image Im5. As a result, the deletion unit 522e generates a first mask image Im6 that includes only the mask area M2 related to the detection target vehicle 10a. Hereinafter, the first mask image Im6 after deleting the mask area M1 related to the vehicle 10 other than the detection target vehicle 10a, that is, the first mask image Im6 that includes only the mask area M2 related to the detection target vehicle 10a is referred to as "first mask image Im6". In addition, the deletion method is not limited to the above. In addition, in the case where only one vehicle 10 is included in the captured image Im1 and in the case where the positions of two or more vehicles 10 included in the captured image Im1 are calculated when neither the first situation nor the second situation is met, the deletion process is omitted. In the following, the first mask image Im6 in the case of executing the deletion process represents the first mask image Im6 generated in the deletion process, that is, the first mask image Im6 after deleting the mask area M1 related to the vehicle 10 other than the detection target vehicle 10a. The first mask image Im5 when the deleting step is omitted indicates the first mask image Im5 generated in the detecting step.
[0111] like Figure 5 As shown in FIG. 1 , after the deletion process, the perspective transformation process (step S7) is performed. Figure 6 As shown, the perspective transformation process is a process of performing perspective transformation on the first mask image Im6. In the perspective transformation process, the perspective transformation unit 522f generates the second mask image Im7 by performing perspective transformation on the first mask image Im6. Specifically, the perspective transformation unit 522f, for example, uses the perspective transformation parameter Pa2 pre-stored in the storage unit 53 to perspectively transform the first mask image Im6 into a bird's-eye view image when observed from a viewpoint above the detection target vehicle 10a that is approximately perpendicular to the road surface 60 (for example, directly above the detection target vehicle 10a). The perspective transformation parameter Pa2 is, for example, a parameter related to the position information and internal parameters of the camera 90 obtained by calibration. Thus, the perspective transformation unit 522f generates the second mask image Im7 represented by the image coordinate system from the first mask image Im6 represented by the camera coordinate system. The image coordinate system is a coordinate system that takes a point of the image plane projected by perspective transformation as the origin and has coordinate axes represented by the Xi axis and the Yi axis orthogonal to the Xi axis. In addition, the perspective transformation method is not limited to the above. The perspective transformation parameter Pa2 may be any parameter other than the above.
[0112] like Figure 5 As shown, after the perspective transformation step, a coordinate point calculation step (step S8) is executed. Figure 7 1 is a diagram for explaining the details of the coordinate point calculation process. The coordinate point calculation process is a process of calculating the image coordinate point P3 for the detection target vehicle 10a.
[0113] In the coordinate point calculation step, the coordinate point calculation unit 522g obtains the base coordinate point P0 from the first circumscribed rectangle R1 set for the mask region M2 in the first mask image Im6 which is an image before perspective transformation. Figure 8 : is a diagram for explaining a method of obtaining a basic coordinate point P0. In order to obtain the basic coordinate point P0, the coordinate point calculation unit 522g sets a basic circumscribed rectangle R0 for the mask area M2 in the first mask image Im6. Next, the coordinate point calculation unit 522g rotates the first mask image Im6 by a required amount with the center of gravity C of the mask area M2 as the rotation center so that the direction of the movement vector V of the detection target vehicle 10a corresponding to the mask area M2 in the first mask image Im6 is oriented in a predetermined direction. The predetermined direction is, for example, the upward direction on the screen of the display device. Next, the coordinate point calculation unit 522g sets a first circumscribed rectangle R1 for the mask area M2 of the rotated first mask image Im6 so that the long side is parallel to the direction of the movement vector V. Next, the coordinate point calculation unit 522g reversely rotates the first mask image Im6 to which the first circumscribed rectangle R1 is attached by the above-mentioned amount with the center of gravity C of the mask area M2 as the rotation center. Thus, the coordinate point calculation unit 522g obtains the coordinate point of one of the four vertices of the first circumscribed rectangle R1 that has the coordinate closest to the measuring point 10e of the detection target vehicle 10a as the base coordinate point P0. Figure 7 As shown, the coordinate point calculation unit 522g performs perspective transformation on the first mask image Im6 after reverse rotation, that is, the first mask image Im6 after obtaining the basic coordinate point P0. As a result, the coordinate point calculation unit 522g obtains the first coordinate point P1 as the coordinate point corresponding to the basic coordinate point P0 in the first circumscribed rectangle R1 deformed by the perspective transformation.
[0114] Furthermore, the coordinate point calculation unit 522g sets a second circumscribed rectangle R2 for the mask area M2 in the second mask image Im7 after perspective transformation of the first mask image Im6. Then, the coordinate point calculation unit 522g obtains, as the second coordinate point P2, a vertex indicating the same position as the first coordinate point P1 among the vertices of the second circumscribed rectangle R2. Specifically, the coordinate point calculation unit 522g obtains, as the second coordinate point P2, a vertex having the coordinate closest to the measuring point 10e of the detection target vehicle 10a among the four vertices of the second circumscribed rectangle R2. That is, the first coordinate point P1 and the second coordinate point P2 are coordinate points indicating the same position, and therefore have a correlation relationship with each other.
[0115] Moreover, the coordinate point calculation unit 522g corrects the coordinates (Xi1, Yi1) of the first coordinate point P1 by the coordinates (Xi2, Yi2) of the second coordinate point P2 based on the size relationship between the coordinate values of the first coordinate point P1 and the coordinate values of the second coordinate point P2. When the coordinate value Xi1 in the Xi direction of the first coordinate point P1 is larger than the coordinate value Xi2 in the Xi direction of the second coordinate point P2 (Xi1>Xi2), the coordinate point calculation unit 522g replaces the coordinate value Xi1 in the Xi direction of the first coordinate point P1 with the coordinate value Xi2 in the Xi direction of the second coordinate point P2. When the coordinate value Yi1 in the Yi direction of the first coordinate point P1 is larger than the coordinate value Yi2 in the Yi direction of the second coordinate point P2 (Yi1>Yi2), the coordinate point calculation unit 522g replaces the coordinate value Yi1 in the Yi direction of the first coordinate point P1 with the coordinate value Yi2 in the Yi direction of the second coordinate point P2. Figure 7 In the example shown, the coordinate value Xi1 of the first coordinate point P1 in the Xi direction is greater than the coordinate value Xi2 of the second coordinate point P2 in the Xi direction. In addition, the coordinate value Yi1 of the first coordinate point P1 in the Yi direction is smaller than the coordinate value Yi2 of the second coordinate point P2 in the Yi direction. As a result, the image coordinate point P3 has the coordinates (Xi2, Yi1). In this way, the coordinate point calculation unit 522g calculates the image coordinate point P3 representing the position of the detection object vehicle 10a in the image coordinate system by correcting the first coordinate point P1 using the second coordinate point P2. In addition, the calculation method of the image coordinate point P3 is not limited to the above.
[0116] like Figure 5 As shown, after the position calculation process, a position conversion process (step S9) is performed. The position conversion process is a process of converting the image coordinate point P3 into a vehicle coordinate point to calculate a vehicle coordinate point indicating the position of the detection target vehicle 10a in the global coordinate system. In the position conversion process, the position conversion unit 523 converts the image coordinate point P3 into a vehicle coordinate point using the shooting parameters acquired by the acquisition unit 521 and predetermined vehicle parameters. The vehicle parameter is a parameter related to the distance from the road surface 60 to the measuring point 10e of the detection target vehicle 10a. In the present embodiment, the vehicle parameter is a height h (described later) of the measuring point 10e of the detection target vehicle 10a from the road surface 60. Fig. 9 ). The vehicle parameter may be a parameter related to the distance from the measuring point 10e of a vehicle 10 of a predetermined type. In other words, the vehicle parameter may be a variable corresponding to the type of the vehicle 10, or may be a predetermined constant.
[0117] The position conversion unit 523 converts the image coordinate point P3 into the vehicle coordinate point using the equations (2) to (4) described later, which use the vehicle coordinate point as the target variable and include the image coordinate point P3, the shooting parameter, and the vehicle parameter as the explanatory variables. In this case, the position conversion unit 523 substitutes the coordinate value of the image coordinate point P3 calculated by the position calculation unit 522 into the relational expressions represented by the equations (2) to (4). The position conversion unit 523 substitutes the shooting parameter acquired by the acquisition unit 521, that is, the value of the shooting parameter corresponding to the imaging device 90 that acquired the captured image Im1, and the value of the vehicle parameter into the relational expressions represented by the equations (2) to (4).
[0118] Fig. 9 This is the first figure for explaining the details of the position conversion process. Fig. 9 , a state in which the detection target vehicle 10a is viewed from the left side is illustrated. Fig.10 FIG. 2 is a second diagram for explaining the details of the position conversion process. Fig.10 , a state when the detection target vehicle 10a is viewed from the roof side is illustrated. Fig. 9 and Fig.10 The global coordinate system is a coordinate system having a fixed coordinate point Pf representing an arbitrary reference position of the road surface 60 as an origin and having coordinate axes represented by an Xg axis and a Yg axis orthogonal to the Xg axis. The photographing coordinate point Pc is a coordinate point representing the position of the photographing device 90 that has obtained the photographing image Im1 used in calculating the image coordinate point P3 in the global coordinate system. The fixed coordinate point Pf and the photographing coordinate point Pc are stored in advance in the storage unit 53.
[0119] like Fig. 9 As shown in FIG. 1 , the observation distance on the XgYg plane between the position of the imaging device 90 and the position of the detection target vehicle 10a (image coordinate point P3) is set to Do. The observation error is set to ΔD. The height [m] of the imaging device 90 from the road surface 60 as an imaging parameter is set to H. The height [m] of the measuring point 10e of the detection target vehicle 10a from the road surface 60 as a vehicle parameter is set to h. In this case, the observation error ΔD can be expressed by the following formula (2).
[0120] ΔD=h / H×Do Formula (2)
[0121] That is, the larger the observation distance Do is, the larger the observation error ΔD is.
[0122] Next, when the actual distance (hereinafter referred to as the first distance) between the position of the imaging device 90 and the position of the measuring point 10e of the detection target vehicle 10a is defined as D, the first distance D can be expressed by the following equation (3).
[0123] D=Do×(1-h / H) Formula (3)
[0124] That is, the first distance D is determined by the observation distance Do, the height H of the imaging device 90 as an imaging parameter, and the height h of the measurement point 10e of the detection target vehicle 10a as a vehicle parameter.
[0125] And, if Fig.10 As shown, when the estimated distance between the reference position and the position of the detection target vehicle 10a is set to Dp and the actual distance between the reference position and the detection target vehicle 10a (hereinafter referred to as the second distance) is set to Dt, the second distance Dt can be expressed by the following formula (4).
[0126] Dt=Dp×(1-h / H) Formula (4)
[0127] Here, the estimated distance Dp can be calculated using the actual distance (hereinafter referred to as the third distance Dc) obtained from the fixed coordinate point Pf and the photographed coordinate point Pc, the image coordinate point P3, and the fixed coordinate point Pf. Therefore, the position conversion unit 523 can calculate the vehicle coordinate point Pv using the second distance Dt and the fixed coordinate point Pf obtained by correcting the estimated distance Dp using the above formula (4). At this time, the calculated vehicle coordinate point Pv is a coordinate point indicating the position of the detection target vehicle 10a in the global coordinate system, and is therefore equivalent to the position of the detection target vehicle 10a in the actual space.
[0128] A-3. Remote control method:
[0129] Fig.11 1 is a first flowchart showing the remote control method in the first embodiment. Fig.12 1 is a second flowchart showing the remote control method in the first embodiment. Fig.13 3 is a third flowchart showing the remote control method in the first embodiment. Figures 11 to 13 The remote control method shown is repeatedly executed at predetermined intervals after the vehicle 10 starts driving in the remote automatic driving mode, for example. Figures 11 to 13 In, about Figure 5 The same steps as those shown are denoted by the same reference numerals and the description thereof is omitted.
[0130] like Fig.11 As shown, the judgment unit 525 judges whether the second situation is met (step S0). If the judgment unit 525 judges that the second situation is not met (step S0: No), the remote control device 5 sequentially performs the acquisition process, the distortion correction process, the rotation process, the cropping process and the detection process.
[0131] In the process of calculating the position of the vehicle 10, the judgment unit 525 uses the first mask image Im5, which is data generated by the detection unit 522d, to judge whether the captured image Im1 contains two or more vehicles 10 (step S10). When the judgment unit 525 judges that the captured image Im1 contains only one vehicle 10 (step S10: No), the remote control device 5 sets the vehicle 10 contained in the captured image Im1 as the detection target vehicle 10a. Then, as Figure 5 As shown, the remote control device 5 sequentially executes the perspective transformation step, the coordinate point calculation step, and the position transformation step to calculate the vehicle coordinate point Pv indicating the position of the detection target vehicle 10a. The calculated vehicle coordinate point Pv is used to create a control value such as a correction control value.
[0132] like Fig.11 As shown, when the determination unit 525 determines that two or more vehicles 10 are included in the captured image Im1 (step S10 : Yes), the determination unit 525 determines whether the first situation is met (step S20 ).
[0133] When the determination unit 525 determines that the first condition is not met (step S20: No), the determination unit 525 determines that each vehicle 10 included in the captured image Im1 is a detection target vehicle 10a. Fig.12 As shown, the remote control device 5 sequentially performs the deletion process, the perspective transformation process, the coordinate point calculation process, and the position transformation process on each detection target vehicle 10a. Fig.12 , the processing flow in the case where the captured image Im1 includes the first vehicle 101 and the second vehicle 102 is illustrated. Thus, the remote control device 5 calculates the vehicle coordinate point Pv indicating the position of two or more vehicles 10 included in the captured image Im1 for each detection target vehicle 10a. The calculated vehicle coordinate point Pv is used, for example, to create a control value such as a correction control value.
[0134] On the other hand, Fig.11 As shown, when it is determined that the first situation is met (step S20: yes), as Fig.13 As shown, the determination unit 525 determines whether to classify two or more vehicles 10 included in the captured image Im1 as either the detection target vehicle 10 a or the non-detection target vehicle 10 f (step S31 ).
[0135] Fig.14 10a and 10f are diagrams for explaining an example of a method for determining whether a vehicle is classified as a detection target vehicle 10a or a non-target vehicle 10f. Fig.14 The following describes a method for determining when both the first vehicle 101 and the second vehicle 102 are present in the recognition target area. Fig.14In FIG. 6 , the grid lines 61 are omitted. Fig.14 , a case where the second vehicle 102 overlaps and exists in the captured image Im1 so as to cover the measuring point 10e of the first vehicle 101 is illustrated. That is, in the captured image Im1, the first vehicle 101 is located on the inner side of the second vehicle 102. In this case, the determination unit 525 determines whether the first vehicle 101 and the second vehicle 102 included in the captured image Im1 are respectively classified as the detection target vehicle 10a and the non-target vehicle 10f, based on whether the measuring point 10e can be detected in the captured image Im1. Specifically, the determination unit 525 determines that the second vehicle 102 for which the measuring point 10e can be detected is the detection target vehicle 10a. On the other hand, the determination unit 525 determines that the first vehicle 101 for which the measuring point 10e cannot be detected because another vehicle 10 overlaps and covers the measuring point 10e in the captured image Im1 is the non-target vehicle 10f.
[0136] like Fig.13 As shown, the remote control device 5 does not perform the deletion process, perspective transformation process, coordinate point calculation process and position transformation process on the non-target vehicle 10f, and creates a stop control value as a retreat control value (step S75). The sending unit 527 sends the retreat control value (here, the stop control value) as the control value created by the control value creating unit 526 to the vehicle as the control target (here, the non-target vehicle 10f) (step S80).
[0137] On the other hand, the remote control device 5 sequentially performs a deletion process, a perspective transformation process, a coordinate point calculation process, and a position transformation process for each detection object vehicle 10a. Thus, the remote control device 5 calculates a vehicle coordinate point Pv representing the position of the detection object vehicle 10a in two or more vehicles 10 included in the captured image Im1 for each detection object vehicle 10a. Then, the judgment unit 525 selects a vehicle 10 as a control object vehicle 10b from the detection object vehicles 10a (step S60). The actual distance calculation unit 524 calculates the actual distance L1 between the detection object vehicle 10a selected by the judgment unit 525 as the control object vehicle 10b and the vehicle 10 traveling without including other vehicles 10 between the detection object vehicle 10a (step S61). The control value making unit 526 sets the target distance L2 (step S62). Then, the control value making unit 526 makes a retreat control value for the detection object vehicle 10a selected by the judgment unit 525 as the control object vehicle 10b (step S71). The sending unit 527 sends the retreat control value, which is the control value generated by the control value generating unit 526, to the vehicle to be controlled (here, the control target vehicle 10b) (step S80). In the first case, if there is only one vehicle 10 determined to be the detection target vehicle 10a, step S60 is omitted.
[0138] like Fig.11 As shown, when the determination unit 525 determines that the second situation is met (step S0: Yes), the determination unit 525 determines that two or more vehicles 10 that are predicted to be included in the captured image Im1 are detection target vehicles 10a. Fig.13 As shown, the judgment unit 525 selects the vehicle 10 as the control target vehicle 10b from the detection target vehicles 10a (step S60). Then, the control value making unit 526 makes the retreat control value for the detection target vehicle 10a selected as the control target vehicle 10b by the judgment unit 525 (step S71). The sending unit 527 sends the retreat control value made by the control value making unit 526 to the control target vehicle 10b (step S80). In addition, when the judgment unit 525 determines that the second situation is met, steps S61 and S62 may be executed in addition to step S60, or steps S61 and S62 may not be executed.
[0139] According to the first embodiment, in at least one of the first and second situations, a retreat control value can be generated to form a positional relationship in which the detection target vehicle 10a among two or more vehicles 10 can be detected by the detection unit 522d without overlapping with other vehicles 10. And the generated retreat control value can be sent to the control target vehicle 10b. Thus, when the position of the vehicle 10 is tracked using the captured image Im1 obtained by the imaging device 90 set at a place different from the vehicle 10, the possibility of two or more vehicles 10 overlapping on the captured image Im1 can be reduced. Therefore, the position of the vehicle 10 can be tracked more accurately using the captured image Im1.
[0140] In addition, according to the first embodiment, it is possible to predict whether two or more vehicles 10 will overlap when the captured image Im1 is acquired before acquiring the captured image Im1 as sensor information. And, when it is predicted that two or more vehicles 10 will overlap when the captured image Im1 is acquired, a retreat control value can be created and sent to the control target vehicle 10b. In this way, it is possible to control the driving action of the vehicle 10 before a state where two or more vehicles 10 overlap on the captured image Im1 is reached. Thus, when the captured image Im1 is used to track the position of the vehicle 10, the possibility of two or more vehicles 10 overlapping on the captured image Im1 can be further reduced. Thus, it is possible to prevent the position of the vehicle 10 from becoming untraceable.
[0141] In addition, according to the first embodiment, the actual distance L1 between the first vehicle 101 and the second vehicle 102, which exist in an overlapping manner in the captured image Im1 or are predicted to exist in an overlapping manner when the captured image Im1 is acquired, can be calculated. And, the retreat control value for specifying the travel action of at least one of the first vehicle 101 and the second vehicle 102 can be created so that the actual distance L1 between the first vehicle 101 and the second vehicle 102 is longer than the actual distance L1 at the time point when it is determined that at least one of the first situation and the second situation has occurred. In this way, when the positions of the first vehicle 101 and the second vehicle 102 are tracked using the captured image Im1, the possibility that the first vehicle 101 and the second vehicle 102 exist in an overlapping manner in the captured image Im1 can be more reliably reduced.
[0142] In addition, according to the first embodiment, the target distance L2 can be set according to the appearance shape of the first vehicle 101 and the appearance shape of the second vehicle 102. In this way, the target distance L2 corresponding to the appearance shape of the vehicle 10 that is related to the area occupied by the vehicle 10 on the captured image Im1 can be set. As a result, the calculation accuracy and computability of the position of the vehicle 10 can be suppressed from decreasing according to the type of the vehicle 10.
[0143] According to the first embodiment, a stop control value can be created for stopping a vehicle 10 whose position cannot be calculated due to two or more vehicles 10 overlapping in the captured image Im1. Thus, the vehicle 10 whose position cannot be calculated can be stopped safely.
[0144] According to the first embodiment, whether the second situation is met can be determined based on the types of two or more vehicles 10 predicted to be included in the captured image Im1 when the captured image Im1 is acquired.
[0145] In addition, according to the first embodiment, it is possible to determine whether the second situation is met based on the appearance shapes of two or more vehicles 10 that are predicted to be included in the captured image Im1 when the captured image Im1 is acquired, which differ depending on the type of the vehicle 10. In this way, it is possible to determine whether the second situation is met based on the appearance shapes of the vehicles 10 that are related to the area occupied by the vehicles 10 on the captured image Im1. As a result, it is possible to more reliably determine whether the second situation is met.
[0146] According to the first embodiment, it is possible to classify two or more vehicles 10 included in the captured image Im1 as either the detection target vehicle 10a or the non-detection target vehicle 10f based on the positional relationship of the two or more vehicles 10 on the captured image Im1.
[0147] According to the first embodiment, a vehicle 10 that cannot detect the measuring point 10e because it overlaps with another vehicle 10 so as to cover the measuring point 10e can be set as an out-of-target vehicle 10f. Furthermore, the deletion step, perspective transformation step, coordinate point calculation step, and position transformation step for calculating the position of the out-of-target vehicle 10f can be omitted. Thus, the processing load in the remote control method can be reduced.
[0148] B. Second Implementation Method:
[0149] Fig.15 1 is a first flowchart showing the remote control method in the second embodiment. Fig.16 is a second flow chart showing the remote control method in the second embodiment. Fig.16 , a processing flow is illustrated when the captured image Im1 includes the first vehicle 101 and the second vehicle 102 . Fig.17 3 is a third flowchart showing the remote control method in the second embodiment. Figures 15 to 17 The remote control method shown is repeatedly executed at predetermined time intervals after the vehicle 10 starts traveling in the remote automatic driving mode, for example.
[0150] In this embodiment, the judgment unit 525 judges whether the two or more vehicles 10 included in the captured image Im1 are classified into the detection target vehicle 10a and the non-target vehicle 10f, respectively, based on whether the image coordinate point P3 is successfully calculated. As a result, part of the function of the judgment unit 525 and part of the processing of the remote control method are different from those in the first embodiment. Regarding other structures, they are the same as the first embodiment. Regarding the steps that are the same as the steps in the first embodiment and the structures that are the same as the first embodiment, the same figure marks are marked and the description is omitted.
[0151] like Fig.15 and Fig.16 As shown in FIG. 1 , when the judgment unit 525 judges that the second condition is not met but the first condition is met (step S0: No, step S20: Yes), the remote control device 5 performs the following processing. In this case, the remote control device 5 sequentially performs the deletion process, the perspective transformation process, and the coordinate point calculation process for each of the two or more vehicles 10 included in the captured image Im1. At this time, for the vehicle 10 whose measuring point 10e cannot be detected because other vehicles 10 overlap in a manner of covering the measuring point 10e, the image coordinate point P3 cannot be calculated, and, for example, an error code is generated. Therefore, the judgment unit 525 judges whether the two or more vehicles 10 included in the captured image Im1 are classified into the detection target vehicle 10a and the non-target vehicle 10f, respectively, according to whether the image coordinate point P3 is successfully calculated. Specifically, the judgment unit 525 judges whether the image coordinate point P3 is successfully calculated (step S32). Furthermore, when the judgment unit 525 judges that the image coordinate point P3 of the first vehicle 101 has been successfully calculated (step S32: Yes), the first vehicle 101 is judged to be the detection target vehicle 10a (step S33). On the other hand, when the judgment unit 525 judges that the image coordinate point P3 of the first vehicle 101 has not been calculated (step S32: No), the first vehicle 101 is judged to be an out-of-target vehicle 10f (step S34). Similarly, when the judgment unit 525 judges that the image coordinate point P3 of the second vehicle 102 has been successfully calculated (step S32: Yes), the second vehicle 102 is judged to be the detection target vehicle 10a (step S33). On the other hand, when the judgment unit 525 judges that the image coordinate point P3 of the second vehicle 102 has not been calculated (step S32: No), the second vehicle 102 is judged to be an out-of-target vehicle 10f (step S34). Then, execution is performed based on the judgment results shown in steps S32 to S34. Fig.17 The steps shown.
[0152] According to the second embodiment, it is possible to determine whether the two or more vehicles 10 included in the captured image Im1 are classified as the detection target vehicle 10a or the non-detection target vehicle 10f, depending on whether the image coordinate point P3 is successfully calculated.
[0153] C. Third Implementation Method:
[0154] Fig.18 1 is a first flowchart showing the remote control method in the third embodiment. Fig.19 is a second flow chart showing the remote control method in the third embodiment. Fig.19 , a processing flow is illustrated when the captured image Im1 includes the first vehicle 101 and the second vehicle 102 . Fig. 20 is a third flow chart showing the remote control method in the third embodiment. Figures 18 to 20 The remote control method shown is repeatedly executed at predetermined time intervals after the vehicle 10 starts traveling in the remote automatic driving mode, for example.
[0155] In this embodiment, the judgment unit 525 judges whether the two or more vehicles 10 included in the captured image Im1 are classified into the detection target vehicle 10a and the non-target vehicle 10f, respectively, based on whether the vehicle coordinate point Pv is successfully calculated. As a result, part of the function of the judgment unit 525 and part of the processing of the remote control method are different from those in the first embodiment. Regarding other structures, they are the same as those in the first embodiment. Regarding the steps that are the same as the steps in the first embodiment and the structures that are the same as those in the first embodiment, the same figure marks are marked and the description is omitted.
[0156] like Fig.18 and Fig.19As shown, when the judgment unit 525 judges that the second situation is not met but the first situation is met (step S0: No, step S20: Yes), the remote control device 5 performs the following processing. In this case, the remote control device 5 sequentially performs the deletion process, the perspective transformation process, the coordinate point calculation process, and the position transformation process for each of the two or more vehicles 10 included in the captured image Im1. At this time, for the vehicle 10 whose measuring point 10e cannot be detected because other vehicles 10 overlap in a manner of covering the measuring point 10e, the vehicle coordinate point Pv cannot be calculated, and, for example, an error code is generated. Therefore, the judgment unit 525 judges whether the two or more vehicles 10 included in the captured image Im1 are classified into one of the detection target vehicle 10a and the non-target vehicle 10f according to whether the vehicle coordinate point Pv is successfully calculated. Specifically, the judgment unit 525 judges whether the vehicle coordinate point Pv is successfully calculated (step S35). Furthermore, when the judgment unit 525 judges that the vehicle coordinate point Pv of the first vehicle 101 has been successfully calculated (step S35: Yes), the first vehicle 101 is judged to be the detection target vehicle 10a (step S36). On the other hand, when the judgment unit 525 judges that the vehicle coordinate point Pv of the first vehicle 101 has not been calculated (step S35: No), the first vehicle 101 is judged to be an out-of-target vehicle 10f (step S37). Similarly, when the judgment unit 525 judges that the vehicle coordinate point Pv of the second vehicle 102 has been successfully calculated (step S35: Yes), the second vehicle 102 is judged to be the detection target vehicle 10a (step S36). On the other hand, when the judgment unit 525 judges that the vehicle coordinate point Pv of the second vehicle 102 has not been calculated (step S35: No), the second vehicle 102 is judged to be an out-of-target vehicle 10f (step S37). Then, execution is performed according to the judgment results shown in steps S35 to S37. Fig. 20 each step.
[0157] According to the third embodiment, it is possible to determine whether the two or more vehicles 10 included in the captured image Im1 are classified as the detection target vehicle 10 a or the non-detection target vehicle 10 f , depending on whether the vehicle coordinate point Pv has been successfully calculated.
[0158] D. Fourth Implementation Method:
[0159] Fig.21 1 is a first flowchart showing the remote control method in the fourth embodiment. Fig. 22 1 is a second flowchart showing the remote control method in the fourth embodiment. Fig.21 and Fig. 22 The remote control method shown is repeatedly executed at predetermined time intervals after the vehicle 10 starts traveling in the remote automatic driving mode, for example.
[0160] In this embodiment, in addition to the captured image Im1, the judgment unit 525 also uses the vehicle coordinate point Pv calculated in the position transformation process to judge whether the first situation is met. Therefore, the functions of a part of the judgment unit 525 and a part of the processing of the remote control method are different from those in the first embodiment. Regarding other structures, they are the same as those in the first embodiment. Regarding the steps that are the same as the steps in the first embodiment and the structures that are the same as those in the first embodiment, the same figure marks are marked and the description is omitted.
[0161] like Fig.21 As shown, when the judgment unit 525 judges that the second condition is not met (step S0: No), the remote control device 5 executes each process from step S1 to step S9. When the captured image Im1 contains two or more vehicles 10 (step S10: Yes), the judgment unit 525 uses the vehicle coordinate point Pv calculated in the position conversion process in addition to the captured image Im1 to judge whether the first condition is met (step S20). Then, the remote control device 5 executes the process according to the judgment result in step S20. Fig. 22 each step.
[0162] According to the fourth embodiment, whether or not the first situation is met can be determined using the vehicle coordinate point Pv indicating the position of the vehicle 10 calculated using the captured image Im1.
[0163] E. Fifth Implementation Method:
[0164] Fig.23 1 is a diagram showing details of the CPU 52a mounted on the remote control device 5 in the fifth embodiment. In the present embodiment, the external sensor 9 is an external laser radar (LiDAR: Light Detection And Ranging) installed in a place different from the vehicle 10. Therefore, part of the structure of the remote control device 5 and the method of calculating the position of the vehicle 10 are different from those in the first embodiment. The other structures are the same as those in the first embodiment. The same reference numerals are used for the same structures as those in the first embodiment and the description is omitted.
[0165] The external laser radar is a laser radar that detects the detection target vehicle 10a from the outside of the detection target vehicle 10a. The external laser radar is a sensor that detects the distance and angle between the external laser radar and the object, the shape of the object, etc. by irradiating laser light to a predetermined detection range RF and detecting the reflected light reflected by the object such as the detection target vehicle 10a. The external laser radar sends the acquired external laser radar information together with the laser radar identification information for identifying multiple external laser radars and the acquisition time of the external laser radar information to the remote control device 5. In order to detect the entire driving road 6 using more than one external laser radar, the installation position and the number of installations of the external laser radar are determined in consideration of the detection range of each external laser radar and the objects (obstacles) existing in the surrounding area of the driving road 6.
[0166] CPU52a Replacement Figure 3 The CPU 52 shown in FIG. 1 is provided with a position calculation unit 522 and a position conversion unit 523. Fig.23 The position generating unit 528 shown in FIG. 5 is used to generate the position of the detection target vehicle 10a using the external laser radar information. The external laser radar information is sensor information obtained by the external laser radar. In this embodiment, the position generating unit 528 uses the coordinate value of the measuring point 10e of the detection target vehicle 10a in the global coordinate system as the position of the detection target vehicle 10a.
[0167] Fig.24 1 is a diagram showing an example of a method for calculating the position of the detection target vehicle 10a using external laser radar information. Fig.24 In the example shown, a reference object 8 capable of reflecting the laser light sent by the external laser radar 95 is shown. The reference object 8 is, for example, a column (pillar) on which a reflector for reflecting the laser light is mounted. A plurality of reference objects 8 may be provided at different positions. The coordinate values (absolute positions) of one or more reference objects 8 in the global coordinate system are known. Information related to the absolute position of the reference object 8 is, for example, pre-stored in the storage unit 53 of the remote control device 5.
[0168] In order to calculate the position of the detection target vehicle 10a, the position generating unit 528 calculates the position of the external laser radar 95 using the absolute position of the reference object 8 and the first detection vector Vs1. The first detection vector Vs1 is a vector from the external laser radar 95 to the reference object 8. The first detection vector Vs1 has a distance Li1 between the external laser radar 95 and the reference object 8 and a direction from the external laser radar 95 to the reference object 8. Fig.24, the direction from the external laser radar 95 to the reference object 8 is represented by the angle θ1 formed by the optical axis reference Ls of the external laser radar 95 and the first detection vector Vs1. The optical axis reference Ls of the external laser radar 95 is equivalent to the center line of the detection range RF, which is the irradiation range of the laser light emitted from the projector of the external laser radar 95. The position generating unit 528 uses trigonometric functions to calculate the difference ΔX1 in the coordinate value in the X direction between the external laser radar 95 and a specific reference object 8, that is, the specific reference object 8, and the difference ΔY1 in the coordinate value in the Y direction between the external laser radar 95 and the specific reference object 8. Then, the position generating unit 528 considers the angle difference θ1 between the optical axis reference Ls of the external laser radar 95 and the first detection vector Vs1, and makes the difference ΔX1 in the X direction and the difference ΔY1 in the Y direction respectively reflected in the coordinate value representing the absolute position of the specific reference object 8. Thus, the position generating unit 528 calculates the position of the external laser radar 95 in the global coordinate system.
[0169] Next, the position generating unit 528 calculates the position of the detection target vehicle 10a using the calculated position of the external laser radar 95 and the second detection vector Vs2. The second detection vector Vs2 is a vector from the external laser radar 95 to the detection target vehicle 10a. The second detection vector Vs2 has a distance Li2 between the external laser radar 95 and the detection target vehicle 10a and a direction from the external laser radar 95 to the detection target vehicle 10a. Fig.24 , the direction from the external laser radar 95 to the detection target vehicle 10a is represented by the angle θ2 formed by the optical axis reference Ls of the external laser radar 95 and the second detection vector Vs2. The position generating unit 528 uses trigonometric functions to calculate the difference ΔX2 in the coordinate values of the external laser radar 95 and the detection target vehicle 10a in the X direction and the difference ΔY2 in the coordinate values of the external laser radar 95 and the detection target vehicle 10a in the Y direction. Then, the position generating unit 528 considers the angle difference θ2 between the optical axis reference Ls of the external laser radar 95 and the second detection vector Vs2, and makes the difference ΔX2 in the X direction and the difference ΔY2 in the Y direction respectively reflected in the coordinate values representing the position of the external laser radar 95. Thus, the position generating unit 528 calculates the coordinate value of the specific part 10j of the detection target vehicle 10a of the external laser radar 95 in the detection target vehicle 10a in the global coordinate system. Then, the position generating unit 528 calculates the coordinate value of the measuring point 10e of the detection target vehicle 10a in the global coordinate system based on the calculated coordinate value of the specific part 10j in the global coordinate system, based on the relative positional relationship between the specific part 10j and the measuring point 10e of the detection target vehicle 10a. In addition, the method of calculating the position of the detection target vehicle 10a using the external laser radar information is not limited to the above. In addition, when calculating the position of the detection target vehicle 10a using the external laser radar information, it is not necessary to use the information related to the reference object 8.
[0170] According to the fifth embodiment, the position of the detection target vehicle 10a can be calculated using the external laser radar information obtained by the external laser radar 95. In addition, in at least one of the first and second situations, the position of the detection target vehicle 10a calculated based on the external laser radar information can be used to create a retreat control value and send it to the control target vehicle 10b. As a result, when the position of the vehicle 10 is tracked using the external laser radar information, the possibility of two or more vehicles 10 overlapping in the external laser radar information can be reduced. Therefore, the position of the vehicle 10 can be tracked more accurately using the external laser radar information.
[0171] F. Sixth embodiment:
[0172] Fig.25 1 is a first flowchart showing the remote control method in the sixth embodiment. Fig.26 1 is a second flowchart showing the remote control method in the sixth embodiment. By using a plurality of external sensors 9 to detect a vehicle 10, it is possible to improve the calculation accuracy when calculating the position of the vehicle 10 or more accurately detect the vehicle 10 traveling in the boundary area of the detection ranges RG and RF of the adjacent plurality of external sensors 9. Therefore, in this embodiment, a remote control method for calculating the position of a vehicle 10 using a plurality of external laser radar information obtained by a plurality of external sensors 9 at the same acquisition time point is described. In this embodiment, the detection target vehicle 10a is preset. Fig.25 and Fig.26 The remote control method shown is repeatedly executed at predetermined intervals after the vehicle 10 starts driving in the remote automatic driving mode, for example. Fig.25 and Fig.26 In the present invention, the same steps as those in the first embodiment are denoted by the same reference numerals and description thereof is omitted.
[0173] like Fig.26As shown, when the detection target vehicle 10a can be detected by any external sensor 9 that includes the detection target vehicle 10a in the detection range RG, RF (step S91: No, step S93: No), the actual distance calculation unit 524 calculates the actual distance L1 between the detection target vehicle 10a and the vehicle 10 that is traveling without including other vehicles 10 between the detection target vehicle 10a (step S61). Then, the control value preparation unit 526 sets the target distance L2 (step S62) and prepares the retreat control value (step S71). In contrast, when the detection target vehicle 10a cannot be detected by any external sensor 9 that includes the detection target vehicle 10a in the detection range RG, RF (step S93: Yes), the control value preparation unit 526 prepares the stop control value for safely stopping the detection target vehicle 10a (step S75). The transmission unit 527 transmits the control value prepared by the control value preparation unit 526 to the control target vehicle 10b (step S80). In addition, when the vehicle 10 can be detected by all the external sensors 9 that include the vehicle 10 in the detection ranges RG and RF (step S91 : Yes), the remote control method is terminated.
[0174] According to the sixth embodiment, when the position of a vehicle 10 is calculated using a plurality of external laser radar information acquired by a plurality of external sensors 9 at the same acquisition time point, the possibility of two or more vehicles 10 overlapping in the sensor information can be reduced. Thus, the position of the vehicle 10 can be more accurately calculated using the sensor information.
[0175] In addition, according to the sixth embodiment, the position of one detection target vehicle 10a can be calculated using a plurality of external sensors 9. In this way, when the detection target vehicle 10a can be detected by any external sensor 9 that includes the detection target vehicle 10a in the detection range RG, RF, it is not necessary to stop the detection target vehicle 10a from the perspective of ensuring safety. As a result, the driving in the remote automatic driving mode can be continued. Therefore, the possibility of delaying the movement of the vehicle 10 can be reduced.
[0176] G. Other implementation methods:
[0177] G-1. Other implementation methods 1:
[0178] The control value generation unit 526 may also set the target distance L2 using the actual distance L1 between the first vehicle 101 and the second vehicle 102 that exist in an overlapping manner or are predicted to exist in an overlapping manner in the sensor information, the sensor information, and the vehicle model database D2. In this case, the vehicle model database D2 also includes information related to the required distance L3. In this case, the required distance L3 is a constant corresponding to the combination of the category of the first vehicle 101 and the category of the second vehicle 102. When the target distance L2 is set according to the predetermined required distance L3, the control value generation unit 526 obtains the vehicle model identification information indicating the category of the first vehicle 101 and the vehicle model identification information indicating the category of the second vehicle 102. Then, the control value generation unit 526 refers to the vehicle model database D2 and obtains the required distance L3 corresponding to the combination of the category of the first vehicle 101 and the category of the second vehicle 102 determined by the vehicle model identification information. When the actual distance L1 between the first vehicle 101 and the second vehicle 102, which overlap or are predicted to overlap in the sensor information, is less than the required distance L3, the control value creation unit 526 sets the acquired required distance L3 as the target distance L2 to create the avoidance control value. On the other hand, when the actual distance L1 between the first vehicle 101 and the second vehicle 102 is greater than the required distance L3, the control value creation unit 526 sets an arbitrary distance greater than the actual distance L1 as the target distance L2. In this way, the target distance L2 can be easily set.
[0179] G-2. Other implementation methods 2:
[0180] The determination unit 525 may determine whether the second condition is met based on only the type of the vehicle 10 without acquiring the shape information. Even in such a method, it is possible to determine whether the second condition is met.
[0181] G-3. Other implementation methods 3:
[0182] The determination unit 525 may determine whether the second situation is met only by using the actual distance L1 between the first vehicle 101 and the second vehicle 102 that is predicted to overlap in the sensor information when the sensor information is acquired. In this case, the determination unit 525 determines that the second situation is met, for example, when the actual distance L1 between the first vehicle 101 and the second vehicle 102 is less than a predetermined distance threshold. On the other hand, the determination unit 525 determines that the second situation is not met when the actual distance L1 between the first vehicle 101 and the second vehicle 102 is greater than the distance threshold. In this way, it is possible to easily determine whether the second situation is met.
[0183] G-4. Other implementation methods 4:
[0184] The judgment unit 525 may also use the actual distance L1 between the first vehicle 101 and the second vehicle 102 calculated at multiple different time points to determine whether the second situation is met. In this case, the judgment unit 525, for example, arranges the actual distances L1 calculated at multiple different time points in a chronological order. And, the judgment unit 525 determines that the second situation is met when the actual distance L1 is closer to zero as the time point for determining whether the second situation is met is closer in the chronological order. If this is the case, by grasping the time series changes of the actual distance L1 between the first vehicle 101 and the second vehicle 102 calculated at multiple different time points, it is possible to determine whether the second situation is met.
[0185] G-5. Other implementation methods 5:
[0186] The judgment unit 525 may also use the actual distance L1 between the first vehicle 101 and the second vehicle 102 calculated at different multiple time points to judge whether the second situation is met. In this case, the judgment unit 525, for example, arranges the difference between the actual distance L1 at the first time point and the actual distance L1 at the previous calculated time point of the first time point, that is, the first difference, in a time series order. The first difference is a value calculated by subtracting the actual distance L1 calculated at the previous calculated time point of the first time point from the actual distance L1 calculated at the first time point. And, the judgment unit 525 judges that the second situation is met when the difference in the actual distance L1 is smaller as the time point closer to the time point for judging whether the second situation is met in the time series order. If this is the case, by grasping the time series change of the difference in the actual distance L1 between the first vehicle 101 and the second vehicle 102 calculated at different multiple time points, it is possible to judge whether the second situation is met.
[0187] G-6. Other implementation methods 6:
[0188] The judgment unit 525 may also use the actual distance L1 between the first vehicle 101 and the second vehicle 102 calculated at multiple different time points to determine whether the second situation is met. In this case, the judgment unit 525, for example, calculates a function representing a timing diagram generated by arranging the actual distances L1 calculated at multiple different time points in a time series order and connecting the actual distances L1 at each time point. In addition, the judgment unit 525 calculates the slope in the timing diagram by differentiating the function. In addition, the judgment unit 525 determines whether the second situation is met by an absolute comparison of the slope in the timing diagram with a predetermined differential threshold. If this is the case, the time series change of the difference in the actual distance L1 between the first vehicle 101 and the second vehicle 102 calculated at multiple different time points can be quantitatively represented using a differential value.
[0189] G-7. Other implementation methods 7:
[0190] The judgment unit 525 may also use the actual distance L1 between the first vehicle 101 and the second vehicle 102 calculated at different multiple time points to judge whether the second situation is met. In this case, the judgment unit 525 calculates the second difference, for example. The second difference is a value calculated by subtracting the actual distance L1 calculated at the previous calculation time point closest to the judgment time point from the actual distance L1 calculated at the calculation time point closest to the judgment time point. And, the judgment unit 525 judges that the second situation is met when the second difference is less than a predetermined difference threshold. On the other hand, when the second difference is greater than a predetermined difference threshold, it is judged that the second situation is not met. In this way, it is possible to judge whether the second situation is met by comparing the difference of the actual distance L1 between the first vehicle 101 and the second vehicle 102 calculated at different multiple time points with the predetermined difference threshold by absolute comparison.
[0191] G-8. Other implementation methods 8:
[0192] In the above-mentioned embodiments, the vehicle 10 may be provided with a structure capable of being moved by remote control, for example, it may be provided with a platform having the structure described below. Specifically, in order to perform the three functions of "driving", "turning" and "stopping" by remote control, the vehicle 10 may be provided with at least a driving device 110, a steering device 120, a braking device 130, a vehicle communication unit 140 and a vehicle control device 150. That is, the vehicle 10 capable of being moved by remote control may not be equipped with at least a part of the interior parts such as a driver's seat and an instrument panel, may not be equipped with at least a part of the exterior parts such as a bumper and a fender, and may not be equipped with a body shell. In this case, the remaining parts such as the body shell may be assembled to the vehicle 10 during the period until the vehicle 10 is shipped from the factory, or the remaining parts such as the body shell may be assembled to the vehicle 10 after the vehicle 10 is shipped from the factory in a state where the remaining parts such as the body shell are not assembled to the vehicle 10. In addition, for the platform, the position can also be calculated in the same way as the vehicle 10 in each embodiment.
[0193] F: Other ways:
[0194] (F1) In the above-mentioned embodiments, the external sensor is not limited to a camera, and may be, for example, a distance measuring device. The distance measuring device is, for example, LiDAR (Light Detection And Ranging). In this case, the detection result output by the external sensor may be three-dimensional dot matrix data representing the vehicle 10. In this case, the server 5 and the vehicle 10 may obtain the vehicle position information by template matching using the three-dimensional dot matrix data as the detection result and the reference dot matrix data prepared in advance.
[0195] (F2) In the above embodiment in which the vehicle 10 can travel by autonomous control, the vehicle 10 may be equipped with an internal sensor, and the detection result outputted from the internal sensor may be used for at least one of the generation of the route and the generation of the travel control signal. For example, the vehicle 10 may obtain the detection result of the internal sensor, and reflect the detection result of the internal sensor in the route when generating the route. The vehicle 10 may obtain the detection result of the internal sensor, and reflect the detection result of the internal sensor in the travel control signal when generating the travel control signal.
[0196] (F3) In the above-mentioned embodiment in which the vehicle 10 can travel by autonomous control, the vehicle 10 uses the detection results of the external sensor to obtain the vehicle position information. In contrast, it is also possible that the vehicle 10 is equipped with an internal sensor, the vehicle 10 uses the detection results of the internal sensor to obtain the vehicle position information, determines the target position to which the vehicle 10 should go next, generates a path from the current position of the vehicle 10 represented by the obtained vehicle position information to the target position, generates a travel control signal for traveling on the generated path, and uses the generated travel control signal to control the actuator of the vehicle 10. In this case, the vehicle 10 can travel without using the detection results of the external sensor at all. In addition, the vehicle 10 can also obtain the target arrival time and congestion information from the outside of the vehicle 10, so that the target arrival time and congestion information are reflected in at least one of the path and the travel control signal. In addition, it is also possible that the functional structure of the system 1 is all set in the vehicle 10. That is, the processing implemented by the system 1 in the present disclosure can also be implemented by the vehicle 10 alone.
[0197] (F4) In the first embodiment described above, the server 5 automatically generates a travel control signal to be sent to the vehicle 10. In contrast, the server 5 may generate a travel control signal to be sent to the vehicle 10 in accordance with an operation of an external operator located outside the vehicle 10. For example, the external operator may operate an operating device including a display for displaying a captured image output from an external sensor, a steering wheel for remotely operating the vehicle 10, an accelerator pedal, a brake pedal, and a communication device for communicating with the server 5 via wired communication or wireless communication, and the server 5 may generate a travel control signal corresponding to the operation applied to the operating device.
[0198] (F5) In each of the above embodiments, the vehicle 10 may be provided with a structure capable of moving by unmanned driving, for example, it may be provided with a platform having the structure described below. Specifically, in order to perform the three functions of "driving", "turning" and "stopping" by unmanned driving, the vehicle 10 may be provided with at least a control device and a drive device, a steering device and an actuator such as a braking device that control the driving of the vehicle 10. In the case where the vehicle 10 obtains information from the outside for unmanned driving, the vehicle 10 may also be provided with a communication device. That is, the vehicle 10 capable of moving by unmanned driving may not be equipped with at least a part of the interior parts such as the driver's seat and the instrument panel, may not be equipped with at least a part of the exterior parts such as the bumper and the fender, and may not be equipped with the body shell. In this case, the remaining parts such as the body shell may be assembled to the vehicle 10 until the vehicle 10 is shipped from the factory, or the remaining parts such as the body shell may be assembled to the vehicle 10 after the vehicle 10 is shipped from the factory in a state where the remaining parts such as the body shell are not assembled to the vehicle 10. Each component can be assembled from any direction such as the upper side, lower side, front side, rear side, right side or left side of the vehicle 10, can be assembled from the same direction or from different directions. In addition, the platform can also be positioned in the same manner as the vehicle 10 in the first embodiment.
[0199] (F6) The vehicle 10 can also be manufactured by combining a plurality of modules. A module means a unit composed of one or more components gathered according to the structure and function of the vehicle 10. For example, the platform of the vehicle 10 can also be manufactured by combining a front module constituting the front part of the platform, a central module constituting the central part of the platform, and a rear module constituting the rear part of the platform. In addition, the number of modules constituting the platform is not limited to three, and can also be two or less or four or more. In addition, the parts of the vehicle 10 that are different from the platform can also be modularized in addition to or instead of the platform. In addition, various modules can include any exterior parts such as bumpers and grilles, and any interior parts such as seats and consoles. In addition, not limited to the vehicle 10, any form of mobile body can be manufactured by combining a plurality of modules. Such a module can be manufactured, for example, by joining a plurality of parts by welding or fixings, or by molding at least a portion of the module as a single component by casting. The molding technique of molding at least a portion of the module as a single component is also called integrated die casting or large integrated die casting. By using integrated die casting, various parts of the mobile body that were previously formed by joining a plurality of parts can be formed as a single component. For example, the front module, the central module, and the rear module mentioned above may also be manufactured by integrated die casting.
[0200] (F7) The transportation of the vehicle 10 by the driving of the unmanned vehicle 10 is also referred to as "autonomous transportation". In addition, the structure for realizing autonomous transportation is also referred to as "vehicle remote control autonomous driving transportation system". In addition, the production method of producing the vehicle 10 by autonomous transportation is also referred to as "autonomous production". In autonomous production, for example, in a factory that manufactures the vehicle 10, at least a part of the transportation of the vehicle 10 is realized by autonomous transportation.
[0201] The present disclosure is not limited to the above-mentioned embodiments, and can be implemented in various structures within the scope of its main purpose. For example, the technical features of the embodiments corresponding to the technical features in each method described in the invention content section can be appropriately replaced or combined in order to solve part or all of the above-mentioned problems or to achieve part or all of the above-mentioned effects. In addition, as long as the technical feature is not described as a necessary technical feature in this specification, it can be appropriately deleted.
[0202] Description of Reference Numerals
[0203] 1…remote control system, 5…remote control device, 6…driving road, 8…reference object, 9…external sensor, 10…vehicle, 10a…detection target vehicle, 10b…control target vehicle, 10e…measurement point, 10f…non-target vehicle, 10g…upper rear end, 10i…front end, 10j…specific part, 51…remote communication unit, 52, 52a…CPU, 53…storage unit, 60…road surface, 61…grid line, 90…photographing device, 95…external laser radar, 101…first vehicle, 102…second vehicle, 110…driving device, 120…steering device, 130…braking device, 140…vehicle communication unit, 150…vehicle control device, 521 …acquisition unit, 522…position calculation unit, 522a…distortion correction unit, 522b…rotation processing unit, 522c…cropping processing unit, 522d…detection unit, 522e…deletion unit, 522f…perspective transformation unit, 522g…coordinate point calculation unit, 523…position transformation unit, 524…actual distance calculation unit, 525…judgment unit, 526…control value preparation unit, 527…transmission unit, 528…position generation unit, 901…first camera, 902…second camera, A1…unmoved area, A2…moved area, C…center of gravity, ΔD…observation error, D…first distance, D1…camera database, D2…vehicle type database, Dc…third distance, Do…observation distance, Dp…estimated distance, Dt…second distance, h…height of the measuring point of the detection target vehicle from the road surface, H…height of the imaging device from the road surface, hg…height of the upper rear end of the first vehicle from the road surface, Im1…captured image, Im2…corrected image, Im3…rotated image, Im4…processed image, Im5, Im6…first mask image, Im7…second mask image, J1…first perpendicular line, J2…second perpendicular line, J3…third perpendicular line, L1…actual distance, L2…target distance, L3…required distance, Lf…total length, Li1…distance between the external laser radar and the reference object, Li2…distance between the external laser radar and the detection target vehicle, Lp…total distance , Ls…optical axis reference, Lt…interval distance, M1…first mask area, M2…second mask area, Md…position detection model, N…network, P0…basic coordinate point, P1…first coordinate point, P2…second coordinate point, P3…image coordinate point, Pa1…distortion correction parameter, Pa2…perspective transformation parameter, Pc…shooting coordinate point, Pf…fixed coordinate point, Pv…vehicle coordinate point, Q1…first intersection point, Q2…second intersection point, Q3…third intersection point, R0…basic circumscribed rectangle, R1…first circumscribed rectangle, R2…second circumscribed rectangle, RF, RG, RG1, RG2…detection range, V…movement vector, Vs1…first detection vector, Vs2…second detection vector.
Claims
1. A remote control device for remotely controlling the driving action of at least one of a plurality of moving objects, The remote control device comprises: an acquisition unit that acquires sensor information indicating at least a part of the plurality of moving objects, which is acquired by a sensor installed at a location different from the plurality of moving objects; a detection unit that detects the moving object included in the sensor information; a control value generating unit that generates a control value that specifies the travel action of at least a part of the plurality of moving bodies using the sensor information; and a transmitting unit configured to transmit the control value to a control target moving body among the plurality of moving bodies that is a control target of the driving action, The control value generating unit generates a retreat control value as the control value for specifying the driving action of at least one of the two or more mobile bodies in order to form a positional relationship in which a detection target mobile body among the two or more mobile bodies that becomes a detection target based on the sensor information can be detected by the detection unit without overlapping with other mobile bodies among the multiple mobile bodies in at least one of the first situation in which two or more mobile bodies among the multiple mobile bodies exist overlappingly in the sensor information and the second situation in which it is predicted before the sensor information is obtained that the two or more mobile bodies will exist overlappingly when the sensor information is obtained.
2. The remote control device according to claim 1, wherein: The first case is a case where a first moving object and a second moving object among the plurality of moving objects are overlapped in the sensor information, The second case is a case where it is predicted before the sensor information is acquired that the first moving object and the second moving object will overlap when the sensor information is acquired. The remote control device further includes an actual distance calculation unit for calculating an actual distance between the first moving body and the second moving body. The retraction control value is the control value that specifies the travel action of at least one of the first and second moving bodies in such a way that the actual distance becomes a target distance longer than the actual distance at a time point when at least one of the first and second situations occurs.
3. The remote control device according to claim 2, wherein: The target distance is set according to the appearance shape of the first moving body and the appearance shape of the second moving body. The target distance becomes longer as the outer shape of the first moving body and the outer shape of the second moving body become larger, and the target distance becomes shorter as the outer shape of the first moving body and the outer shape of the second moving body become smaller.
4. The remote control device according to claim 2, wherein: The second moving body is the moving body that travels behind the first moving body in the same advancing direction as the first moving body, The retreat control value includes at least one of an acceleration control value that increases the acceleration of the first moving body in the forward direction, an acceleration control value that reduces the acceleration of the second moving body in the forward direction, a direction control value that causes the second moving body to retreat in a direction opposite to the forward direction, and a stop control value that stops the driving action of the second moving body.
5. The remote control device according to claim 4, wherein: The retraction control value further includes a deflection angle control value that specifies a deflection angle of the control target moving body.
6. The remote control device according to claim 2, wherein: The retraction control value includes a deflection angle control value that defines a deflection angle of the control target moving body in order to move one of the first moving body and the second moving body in the width direction of the travel path.
7. The remote control device according to claim 6, wherein: The retraction control value further includes an acceleration control value that specifies the acceleration of the control target moving body.
8. The remote control device according to claim 2, wherein: The second moving body is the moving body that travels behind the first moving body in the same advancing direction as the first moving body, The retreat control value is a control value used to make any one of the first and second moving bodies move in the width direction of the travel path and make the second moving body exceed the first moving body, and includes an acceleration control value that specifies the acceleration of the control object moving body and a deflection angle control value that specifies the deflection angle of the control object moving body.
9. The remote control device according to claim 2, wherein: The retreat control value includes a trajectory control value that specifies a travel trajectory by arranging the target travel positions of the control target moving body at predetermined time intervals in a time series order.
10. The remote control device according to claim 1, wherein: The remote control device includes a determination unit that determines whether the second situation applies based on the types of the two or more moving objects that are predicted to be included in the sensor information when the sensor information is acquired.
11. The remote control device according to claim 1, wherein: The remote control device includes a determination unit that determines whether the second condition is satisfied based on the appearance shapes of the two or more moving objects that differ depending on the type of the moving object and are predicted to be included in the sensor information when the sensor information is acquired.
12. A remote control system comprising: Multiple mobile bodies; a sensor, which is disposed at a location different from the plurality of moving objects, and acquires sensor information indicating at least a portion of the plurality of moving objects; and The remote control device according to any one of claims 1 to 11.
13. A remote control method for remotely controlling the driving action of at least one of a plurality of moving objects, The remote control method comprises the following steps: an acquisition step of acquiring sensor information indicating at least a part of the plurality of moving objects, which is acquired by a sensor installed at a location different from the plurality of moving objects; A detection step of detecting the moving object included in the sensor information; a control value creating step of creating a control value that specifies the travel action of at least a part of the plurality of moving bodies using the sensor information; and a sending step of sending the control value to a control target moving body among the plurality of moving bodies that is a control target of the driving action, In the control value making process, in at least one of a first situation in which two or more of the multiple mobile bodies exist overlappingly in the sensor information and a second situation in which it is predicted before the sensor information is obtained that the two or more mobile bodies will exist overlappingly when the sensor information is obtained, a retreat control value is made as the control value for specifying the driving action of at least one of the two or more mobile bodies in order to form a positional relationship in which a detection target mobile body among the two or more mobile bodies that becomes a detection target based on the sensor information can be detected in the detection process without overlapping with other mobile bodies among the multiple mobile bodies.
