Vehicle positioning device and vehicle positioning method

Abstract

To accurately estimate the position of a vehicle.SOLUTION: A vehicle positioning device includes: an image acquisition unit that acquires an image of a vehicle from a camera positioned outside the vehicle that can travel by unmanned driving; and a position estimation unit that estimates the position of the vehicle using the image, the position estimation unit detecting skeletal information including the positions of a plurality of predetermined characteristic parts of the vehicle from the image, and estimating the position of the vehicle using the skeletal information.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present disclosure relates to a vehicle positioning device and a vehicle positioning method. [Background technology]

[0002] A technique is known in which a vehicle is imaged with a camera positioned outside the vehicle, a bounding box is generated surrounding the vehicle in the image acquired from the camera, and the vehicle's position is estimated using the bounding box (for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-175768 Summary of the Invention [Problem to be solved by the invention]

[0004] In a method of estimating the position of a vehicle using a bounding box, for example, if the vehicle is photographed from diagonally above, the difference between the bounding box and the vehicle's outline becomes large, making it impossible to accurately estimate the vehicle's position. [Means for solving the problem]

[0005] The present disclosure can be realized in the following forms.

[0006] (1) According to a first aspect of the present disclosure, there is provided a vehicle positioning device including: an image acquisition unit that acquires an image of a vehicle capable of traveling in an unmanned driving mode from a camera positioned outside the vehicle; and a position estimation unit that estimates a position of the vehicle using the image, the position estimation unit detecting, from the image, skeletal information including positions of a plurality of predetermined characteristic features of the vehicle, and estimating a position of the vehicle using the skeletal information. According to the vehicle positioning device of this embodiment, the position of the vehicle can be estimated accurately. (2) In the vehicle positioning device of the above aspect, the characteristic part may be a wheel of the vehicle. According to the vehicle positioning device of this embodiment, skeleton information can be easily detected from an image. (3) In the vehicle positioning device of the above aspect, the position estimation unit may determine an area including the vehicle from the image, and detect the skeleton information from the area. According to this type of vehicle positioning device, when an image contains multiple vehicles, skeleton information can be detected from each area containing the vehicles, thereby making it possible to accurately estimate the position of the vehicles. (4) The vehicle positioning device of the above aspect may further include a control unit that causes the vehicle to travel in an unmanned manner using the estimation result of the position estimation unit. According to this type of vehicle positioning device, the vehicle can be driven unmanned while its position is estimated. (5) According to a second aspect of the present disclosure, there is provided a vehicle positioning method, comprising: an image acquisition step of acquiring an image of a vehicle capable of traveling by unmanned driving from a camera positioned outside the vehicle; and a position estimation step of estimating a position of the vehicle using the image, the position estimation step detecting, from the image, skeleton information including positions of a plurality of predetermined characteristic parts of the vehicle, and estimating the position of the vehicle using the skeleton information. According to this embodiment of the vehicle positioning method, the vehicle position can be estimated accurately. The present disclosure may be realized in various forms other than the vehicle positioning device and vehicle positioning method, such as an unmanned driving system, a control device, a control method, a computer program, and a recording medium on which the computer program is recorded. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is an explanatory diagram showing the configuration of an unmanned driving system according to a first embodiment. [Figure 2]FIG. 2 is an explanatory diagram showing the state in which the vehicle of the first embodiment travels in an unmanned driving mode. [Figure 3] 3 is a flowchart showing a processing procedure for vehicle travel control in the first embodiment. [Figure 4] 3 is a flowchart showing a vehicle positioning method according to the first embodiment. [Figure 5] FIG. 3 is an explanatory diagram showing how the skeleton of a vehicle is estimated from an image according to the first embodiment. [Figure 6] FIG. 10 is an explanatory diagram showing the configuration of an unmanned driving system according to a second embodiment. [Figure 7] 10 is a flowchart showing a processing procedure for vehicle travel control according to a second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] A. First embodiment: 1 is an explanatory diagram showing the configuration of an unmanned driving system 10 equipped with a vehicle positioning device in a first embodiment. In this embodiment, the unmanned driving system 10 is used to drive the unmanned vehicles 100 in a factory that manufactures unmanned vehicles 100. Note that the unmanned driving system 10 may also be used to drive the unmanned vehicles 100 in places other than factories, such as commercial facilities, universities, and parks.

[0009] The vehicle 100 is, for example, a passenger car, a truck, a bus, a construction vehicle, etc. In this embodiment, the vehicle 100 is an electric vehicle (BEV: Battery Electric Vehicle). The vehicle 100 is not limited to an electric vehicle, and may be, for example, a gasoline-powered vehicle, a hybrid electric vehicle (HEV: Hybrid Electric Vehicle), a fuel cell electric vehicle (FCEV: Fuel Cell Electric Vehicle), etc.

[0010] In this disclosure, "unmanned driving" means driving without the driver's control. Driving control means at least one of driving, turning, and stopping of the vehicle 100. Unmanned driving is achieved by automatic or manual remote control using a device located outside the vehicle 100, or by autonomous control of the vehicle 100. A vehicle 100 that is driving in an unmanned manner may have a driver on board who does not control the driving. A driver who does not control the driving includes, for example, a person who simply sits in a seat of the vehicle 100, or a person who is riding in the vehicle 100 and performing work other than driving operations, such as assembly, inspection, or operating switches. Note that driving with a driver controlling the driving is sometimes called "manned driving."

[0011] In this disclosure, "remote control" includes "full remote control" in which all of the operations of vehicle 100 are completely determined from outside vehicle 100, and "partial remote control" in which some of the operations of vehicle 100 are determined from outside vehicle 100. Furthermore, "autonomous control" includes "full autonomous control" in which vehicle 100 autonomously controls its own operations without receiving any information from devices external to vehicle 100, and "partial autonomous control" in which vehicle 100 autonomously controls its own operations using information received from devices external to vehicle 100.

[0012] In this embodiment, the unmanned driving system 10 includes a vehicle 100, a server device 200, and at least one camera 300. In this embodiment, the server device 200 corresponds to the "vehicle positioning device" of the present disclosure.

[0013] In this embodiment, the vehicle 100 is configured to be able to travel under remote control. The vehicle 100 includes a vehicle control device 110 for controlling each part of the vehicle 100, an actuator group 120 that operates under the control of the vehicle control device 110, and a communication device 130 for communicating with a server device 200 via wireless communication. The actuator group 120 includes at least one actuator. In this embodiment, the actuator group 120 includes an actuator of a drive device that generates a propulsive force for the vehicle 100, an actuator of a steering device that changes the traveling direction of the vehicle 100, and an actuator of a braking device that generates a braking force for the vehicle 100. In this embodiment, the vehicle 100 is a four-wheeled vehicle and includes left and right front wheels 150F and left and right rear wheels 150R. In the following description, the left front wheel 150F may be referred to as the left front wheel 150FL, the right front wheel 150F may be referred to as the right front wheel 150FR, the left rear wheel 150R may be referred to as the left rear wheel 150RL, and the right rear wheel 150R may be referred to as the right rear wheel 150RR. When no particular distinction is made between these, they will simply be referred to as wheels 150.

[0014] The vehicle control device 110 is configured by a computer including a processor 111, a memory 112, an input / output interface 113, and an internal bus 114. The processor 111, the memory 112, and the input / output interface 113 are connected to each other via the internal bus 114 so as to be able to communicate bidirectionally. The input / output interface 113 is connected to an actuator group 120 and a communication device 130.

[0015] The processor 111 functions as a driving control unit 195 by executing a computer program PG1 stored in advance in the memory 112. The driving control unit 195 controls the actuator group 120. When a passenger is on board the vehicle 100, the driving control unit 195 controls the actuator group 120 in accordance with the operation of the passenger, thereby causing the vehicle 100 to drive. Regardless of whether a passenger is on board the vehicle 100 or not, the driving control unit 195 controls the actuator group 120 using a driving control signal received from the server device 200, thereby causing the vehicle 100 to drive. In this embodiment, the driving control signal includes the acceleration and steering angle of the vehicle 100 as parameters. Note that in other embodiments, the driving control signal may include the speed of the vehicle 100 as a parameter instead of or in addition to the acceleration of the vehicle 100.

[0016] The server device 200 is configured by a computer including a processor 201, a memory 202, an input / output interface 203, and an internal bus 204. The processor 201, the memory 202, and the input / output interface 203 are connected via the internal bus 204 to enable bidirectional communication. A communication device 205 is connected to the input / output interface 203 for communicating with the vehicle 100 via wireless communication. In this embodiment, the communication device 205 can also communicate with the camera 300 via wired communication or wireless communication.

[0017] The processor 201 executes a computer program PG2 pre-stored in the memory 202, thereby functioning as an image acquisition unit 211, a position estimation unit 212, and a remote control unit 215. The image acquisition unit 211 acquires an image of the vehicle 100 from the camera 300. The position estimation unit 212 estimates the position and orientation of the vehicle 100 using the image acquired by the image acquisition unit 211. The remote control unit 215 generates a driving control signal for remotely controlling the vehicle 100 to drive using the estimation result of the position estimation unit 212, and transmits the driving control signal to the vehicle 100 via the communication device 205. The memory 202 pre-stores a detection model MD and a reference route RR.

[0018] The camera 300 is located outside the vehicle 100. The position and orientation of the camera 300 are adjusted in advance. The camera 300 captures an image of the vehicle 100 and generates an image of the vehicle 100. The image may be a still image or a video image. The camera 300 is equipped with a communication device (not shown) and can communicate with the server device 200 via wired or wireless communication. The camera 300 transmits an image of the vehicle 100 to the server device 200 via the communication device.

[0019] FIG. 2 is an explanatory diagram showing how the vehicle 100 moves by remote control in a factory FC. The factory FC includes a first location PL1 and a second location PL2. The first location PL1 is, for example, a location where the vehicle 100 is assembled, and the second location PL2 is, for example, a location where the vehicle 100 is inspected. The vehicle 100 that passes the inspection at the second location PL2 is then shipped from the factory FC. In this embodiment, the vehicle 100 assembled at the first location PL1 is in a state where it can be driven by remote control. That is, the vehicle 100 assembled at the first location PL1 is equipped with at least a vehicle control device 110, an actuator group 120, a communication device 130, and wheels 150. The first location PL1 and the second location PL2 are connected by a track TR along which the vehicle 100 can travel. A plurality of cameras 300 are installed around the track TR to capture images of the vehicle 100 on the track TR. The server device 200 can remotely control the vehicle 100 to travel while estimating the position and orientation of the vehicle 100 on the track TR using images acquired from the camera 300. Therefore, the unmanned driving system 10 can remotely move the vehicle 100 from the first location PL1 to the second location PL2 without using a transport device such as a crane or conveyor.

[0020] 3 is a flowchart showing the steps of a driving control method for remotely controlling driving of the vehicle 100. Steps S11 to S14 are repeatedly executed at a predetermined cycle by the processor 201 of the server device 200. Steps S15 to S16 are repeatedly executed at a predetermined cycle by the processor 111 of the vehicle control device 110.

[0021] In step S11, the processor 201 of the server device 200 estimates the position and orientation of the vehicle 100 using the image output from the camera 300, and acquires vehicle position information from the estimation result. The vehicle position information is position information that serves as the basis for generating a driving control signal. In this embodiment, the vehicle position information includes the position and orientation of the vehicle 100 in a global coordinate system GC of the factory FC. The position of the vehicle 100 can be expressed, for example, by X, Y, and Z coordinates of the global coordinate system GC. The orientation of the vehicle 100 can be expressed, for example, by an angle relative to a reference axis of the global coordinate system GC. Details of the method for estimating the position and orientation of the vehicle 100 will be described later.

[0022] In step S12, the processor 201 of the server device 200 determines a target position to which the vehicle 100 should next head. In this embodiment, the target position is represented by X, Y, and Z coordinates in the global coordinate system GC. A reference route RR, which is a route to be traveled by the vehicle 100, is stored in advance in the memory 202 of the server device 200. The route is represented by nodes indicating the departure point, nodes indicating passing points, nodes indicating the destination, and links connecting the nodes. The processor 201 uses the vehicle position information and the reference route RR to determine a target position to which the vehicle 100 should next head. The processor 201 determines a target position on the reference route RR that is ahead of the current location of the vehicle 100.

[0023] In step S13, the processor 201 of the server device 200 generates a travel control signal for causing the vehicle 100 to travel toward the determined target position. The processor 201 calculates the travel speed of the vehicle 100 from the change in the position of the vehicle 100 and compares the calculated travel speed with the target speed. When the travel speed is lower than the target speed, the processor 201 determines an acceleration such that the vehicle 100 accelerates. When the travel speed is higher than the target speed, the processor 201 determines an acceleration such that the vehicle 100 decelerates. Furthermore, when the vehicle 100 is located on the reference route RR, the processor 201 determines a steering angle and acceleration such that the vehicle 100 does not deviate from the reference route RR. When the vehicle 100 is not located on the reference route RR, in other words, when the vehicle 100 has deviated from the reference route RR, the processor 201 determines a steering angle and acceleration such that the vehicle 100 returns to the reference route RR.

[0024] In step S14, the processor 201 of the server device 200 transmits the generated driving control signal to the vehicle 100. The processor 201 repeats, at a predetermined cycle, the acquisition of vehicle position information, the determination of a target position, the generation of a driving control signal, and the transmission of the driving control signal.

[0025] In step S15, the processor 111 of the vehicle control device 110 receives the driving control signal transmitted from the server device 200. In step S16, the processor 111 controls the actuator group 120 using the received driving control signal to cause the vehicle 100 to drive at the acceleration and steering angle indicated in the driving control signal. The processor 111 repeats receiving the driving control signal and controlling the actuator group 120 at a predetermined cycle.

[0026] Fig. 4 is a flowchart showing the steps of a vehicle positioning method for estimating the position and orientation of the vehicle 100. Fig. 5 is an explanatory diagram showing skeleton information of the vehicle 100 detected from the image IM. The vehicle positioning method shown in Fig. 4 is executed by the processor 201 of the server device 200 in step S11 of Fig. 3.

[0027] 4, in step S110, the image acquisition unit 211 acquires an image IM of the vehicle 100 from the camera 300. As shown in Fig. 5, the image IM may include, in addition to the vehicle 100, the road surface of the track TR and an object OB other than the vehicle 100. The object OB other than the vehicle 100 may be, for example, various pieces of equipment in the factory FC, workers at the factory FC, or vehicles other than the vehicle 100.

[0028] In step S120, the position estimation unit 212 detects skeleton information related to the skeleton SK of the vehicle 100 from the image IM. The skeleton SK of the vehicle 100 is composed of a plurality of predetermined feature portions FP of the vehicle 100 and line segments LS connecting the feature portions FP. In this embodiment, the skeleton information includes the coordinates of the plurality of feature portions FP in the camera coordinate system. In this embodiment, the skeleton SK is composed of four feature portions FP and four line segments LS. The four feature portions FP are the four wheels 150. More specifically, the four feature portions FP are the center points of the left front wheel 150FL, the right front wheel 150FR, the left rear wheel 150RL, and the right rear wheel 150RR. The four line segments LS are a line segment connecting the center point of left front wheel 150FL and the center point of right front wheel 150FR, a line segment connecting the center point of left front wheel 150FL and the center point of left rear wheel 150RL, a line segment connecting the center point of right front wheel 150FR and the center point of right rear wheel 150RR, and a line segment connecting the center point of left rear wheel 150RL and the center point of right rear wheel 150RR. In the following description, the center point of left front wheel 150FL may be referred to as the first characteristic portion FP1, the center point of right front wheel 150FR may be referred to as the second characteristic portion FP2, the center point of left rear wheel 150RL may be referred to as the third characteristic portion FP3, and the center point of right rear wheel 150RR may be referred to as the fourth characteristic portion FP4. When the four characteristic portions FP1 to FP4 are described without any particular distinction, they will simply be referred to as the characteristic portions FP. The line segment connecting the center point of the left front wheel 150FL and the center point of the right front wheel 150FR is referred to as the first line segment LS1, the line segment connecting the center point of the left front wheel 150FL and the center point of the left rear wheel 150RL is referred to as the second line segment LS2, the line segment connecting the center point of the right front wheel 150FR and the center point of the right rear wheel 150RR is referred to as the third line segment LS3, and the line segment connecting the center point of the left rear wheel 150RL and the center point of the right rear wheel 150RR is referred to as the fourth line segment LS4. When the four line segments LS1 to LS4 are not particularly distinguished, they are simply referred to as line segments LS. In this embodiment, the skeleton SK is configured in the shape of a rectangular frame. Since the relative positions of the wheels 150 are fixed, the relative positions of the characteristic portions FP are also fixed. The spacing between the characteristic portions FP and the length of each line segment LS are the same as the spacing between the center points of the wheels 150.The distance between the center points of the wheels 150 can be obtained, for example, by measuring the distance between the center points of the wheels 150 or by referring to the design data of the vehicle 100.

[0029] In this embodiment, the position estimation unit 212 first determines the region RG in the image IM that includes the vehicle 100. The position estimation unit 212 then detects skeletal information of the vehicle 100 from the region RG that includes the vehicle 100. Because the relative positional relationship between the feature portions FP is fixed, if the position estimation unit 212 can identify the positions of at least two feature portions FP from the region RG in the image IM, it can identify the positions of the remaining feature portions FP. For example, in FIG. 5, if the positions of the first feature portion FP1 and the third feature portion FP3 that appear in the image IM can be identified, it can identify the positions of the second feature portion FP2 and the fourth feature portion FP4 that are hidden in the image IM.

[0030] In this embodiment, the position estimation unit 212 uses the detection model DM to determine an area RG in the image IM that includes the vehicle 100, and detects skeletal information of the vehicle 100 from the area RG. The detection model DM is a trained model that, when an image IM acquired from the camera 300 is input, determines an area RG in the image IM that includes the vehicle 100, detects skeletal information of the vehicle 100 from the area RG, and outputs the skeletal information. The detection model DM can implement, for example, either semantic segmentation or instance segmentation. The detection model DM can be, for example, a convolutional neural network (hereinafter referred to as CNN) trained by supervised learning using multiple training datasets. The training dataset includes training images labeled with correct labels. The training images can be images of the vehicle 100 generated by the camera 300, or images of the vehicle 100 generated by 3D CAD software or 3D CG software. The correct labels include the coordinates of four feature portions FP1 to FP4 in the camera coordinate system. During CNN training, it is preferable to update the parameters of the CNN by backpropagation (error backpropagation method) so as to reduce the error between the output result of the detection model DM and the label.

[0031] In step S130, the position estimation unit 212 estimates the position and orientation of the vehicle 100 using the skeleton information. In this embodiment, the position estimation unit 212 estimates the position and orientation of the vehicle 100 using the coordinates of four feature portions FP1 to FP4 included in the skeleton information. The position estimation unit 212 outputs, as estimation results, the coordinates of the four feature portions FP1 to FP4 converted from coordinates in the camera coordinate system to coordinates in the global coordinate system GC. In this case, the four coordinates can represent the area in which the vehicle 100 exists. Note that the position estimation unit 212 may output, as estimation results, for example, the coordinates of the centers of the four feature portions FP1 to FP4 and a vector pointing forward of the vehicle 100. In this case, the position of the vehicle 100 can be simply represented by a single set of coordinates. Note that step S110 may be referred to as an image acquisition process, and steps S120 to S130 may be referred to as a position estimation process.

[0032] In the embodiment described above, the server device 200 estimates the position and orientation of the vehicle 100 using the skeletal information of the vehicle 100, thereby enabling accurate estimation of the position and orientation of the vehicle 100. Here, if the position and orientation of the vehicle 100 are estimated using the coordinates of the vertices of a bounding box surrounding the vehicle 100 in the image IM, for example, when the vehicle 100 is imaged by the camera 300 from diagonally above the vehicle 100, the difference between the bounding box and the outline of the vehicle 100 may become large, resulting in a problem of reduced accuracy of the estimation result. In contrast, in the embodiment, the skeletal information of the vehicle 100 includes the coordinates of the center points of the front, rear, left, and right wheels 150 of the vehicle 100 as the coordinates of the characteristic portions FP1 to FP4 of the vehicle 100. Therefore, the position and orientation of the vehicle 100 are estimated from the coordinates of the center points of the front, rear, left, and right wheels 150, making it less likely that the problem of a large difference from the outline of the vehicle 100 will occur. This makes it possible to prevent a decrease in the accuracy of the estimation result.

[0033] Furthermore, in this embodiment, the server device 200 detects an area RG containing the vehicle 100 from the image IM, and then detects skeletal information of the vehicle 100 from the area RG. A technique of detecting an area RG containing the vehicle 100 from the image IM and then detecting skeletal information of the vehicle 100 from the area RG is sometimes referred to as a top-down approach. In contrast, a technique of detecting skeletal information of the vehicle 100 from the image IM without detecting an area RG containing the vehicle 100 from the image IM is sometimes referred to as a bottom-up approach. The position estimation unit 212 may detect skeletal information of the vehicle 100 using a bottom-up approach. However, generally, when multiple vehicles 100 are included in the image IM, a top-down approach, in which an area RG containing each vehicle 100 is determined and then skeletal information is detected from each area RG, can detect skeletal information more accurately than a bottom-up approach. Therefore, it is preferable to use the top-down approach.

[0034] Furthermore, in this embodiment, the four wheels 150 of the vehicle 100 are used as the four feature portions FP of the vehicle 100, so the difference between the skeleton SK of the vehicle 100 and the contour of the vehicle 100 is relatively small. Therefore, the range in which the vehicle 100 exists can be grasped relatively accurately. Furthermore, in this embodiment, the wheels 150 that tend to appear in the image IM are used as the feature portions FP, so that skeleton information can be easily detected from the image IM. Furthermore, by using the four wheels 150 of the vehicle 100 as the four feature portions FP of the vehicle 100, it is possible to easily prepare a training dataset for training the detection model DM. Specifically, because the relative positional relationship between the four wheels 150 is fixed, the positions of the remaining feature portions FP hidden in the training image can be grasped by projecting a rectangle onto the training image so that at least two feature portions FP of the vehicle 100 appearing in the training image overlap with the vertices of the rectangle that reproduces the skeleton SK corresponding to the feature portions FP. Therefore, it is possible to easily generate correct labels to be assigned to the training images.

[0035] B. Second embodiment: 6 is an explanatory diagram showing the configuration of an unmanned driving system 10b in the second embodiment. The second embodiment differs from the first embodiment in that the unmanned driving system 10b does not include a server device 200, and the vehicle 100 is configured to be able to run by autonomous control of the vehicle 100 rather than by remote control. Unless otherwise specified, the other configurations are the same as those in the first embodiment. In this embodiment, the vehicle control device 110 corresponds to the "vehicle positioning device" of this disclosure.

[0036] In this embodiment, the communication device 130 mounted on the vehicle 100 communicates with the camera 300 via wireless communication. The processor 111 of the vehicle control device 110 executes a computer program PG1 pre-stored in the memory 112, thereby functioning as an image acquisition unit 191, a position estimation unit 192, and a driving control unit 195. The image acquisition unit 191 acquires an image IM from the camera 300. The position estimation unit 192 estimates the position and orientation of the host vehicle 100 using the image IM acquired by the image acquisition unit 191. The driving control unit 195 generates a driving control signal for driving the host vehicle 100 using the estimation result of the position estimation unit 192, and controls the actuator group 120 using the driving control signal. A detection model DM and a reference route RR are pre-stored in the memory 112 of the vehicle control device 110.

[0037] FIG. 7 is a flowchart showing a processing procedure for driving control of the vehicle 100 in the second embodiment. Steps S21 to S24 shown in FIG. 7 are executed by the processor 111 of the vehicle control device 110. In step S21, the vehicle control device 110 acquires vehicle position information using the image IM output from the camera 300. As in the first embodiment, the vehicle control device 110 acquires vehicle position information of the host vehicle 100 using a vehicle positioning method. In step S22, the vehicle control device 110 determines a target position to which the host vehicle 100 should next head. In step S23, the vehicle control device 110 generates a driving control signal for driving the host vehicle 100 toward the determined target position. In step S24, the vehicle control device 110 controls the actuator group 120 using the generated driving control signal, thereby causing the host vehicle 100 to drive in accordance with parameters represented in the driving control signal. The vehicle control device 110 repeats the acquisition of vehicle position information, determination of a target position, generation of a driving control signal, and control of the actuator group 120 at a predetermined cycle.

[0038] According to the unmanned driving system 10b of this embodiment described above, the vehicle 100 can be driven by autonomous control of the vehicle 100, without remote control of the vehicle 100 by the server device 200. Furthermore, according to the unmanned driving system 10b of this embodiment, the position and orientation of the vehicle 100 can be accurately estimated, similar to the first embodiment.

[0039] C. Other Embodiments: (C1) In the first and second embodiments, the position estimation units 212, 192 detect skeletal information of the vehicle 100 from the image IM and estimate the position and orientation of the vehicle 100. In contrast, the position estimation units 212, 192 may detect skeletal information of an object OB located around the road TR from the image IM and estimate the position and orientation of the object OB, rather than detecting skeletal information of the vehicle 100 from the image IM and estimating the position and orientation of the vehicle 100. For example, if the object OB in the image IM is surrounded by a rectangular bounding box and the position and orientation of the object OB are estimated from the coordinates of the vertices of the bounding box, the difference between the outline of the object OB and the bounding box will be large, and there is a possibility that the object OB will be mistakenly recognized as being positioned in a state where it has invaded the road TR, even though it is not actually positioned in a state where it has invaded the road TR. In particular, when the object OB has an appearance that combines a square and a cone, such as a traffic cone (registered trademark), the difference between the outline of the object OB and the bounding box is likely to be large, and the above-mentioned misrecognition is likely to occur. However, by detecting skeletal information of the object OB and estimating the position and orientation of the object OB, the possibility of the above-mentioned misrecognition occurring can be reduced.

[0040] (C2) In the first and second embodiments, the position estimation units 212, 192 detect skeletal information of the vehicle 100 from the image IM and estimate the position and orientation of the vehicle 100. Alternatively, the position estimation units 212, 192 may detect skeletal information of the vehicle 100 and skeletal information of an object OB located around the vehicle 100 from the image IM, estimate the position and orientation of the vehicle 100 from the skeletal information of the vehicle 100, and estimate the position and orientation of the object OB from the skeletal information of the object OB. The remote control unit 215 and the traveling control unit 195 may use the estimation results of the position estimation units 212, 192 to determine whether or not the vehicle 100 and the object OB will come into contact with each other, and if it is determined that the vehicle 100 and the object OB will come into contact with each other, control the vehicle 100 to avoid contact with the object OB. For example, if the vehicle 100 and the object OB in the image IM are each surrounded by a rectangular bounding box, and the position and orientation of the vehicle 100 are estimated from the coordinates of the vertices of the bounding box surrounding the vehicle 100, and the position and orientation of the object OB are estimated from the coordinates of the vertices of the bounding box surrounding the object OB, there is a possibility that an erroneous determination will be made that the vehicle 100 and the object OB are in contact, even though they are not actually in contact. However, by detecting skeletal information of the vehicle 100 and skeletal information of the objects OB located around the vehicle 100 from the image IM, estimating the position and orientation of the vehicle 100 from the skeletal information of the vehicle 100, and estimating the position and orientation of the object OB from the skeletal information of the object OB, the possibility of the above-mentioned erroneous determination occurring can be reduced.

[0041] (C3) In the first and second embodiments, the position estimation units 212, 192 detect skeleton information including the coordinates of four characteristic parts FP of the vehicle 100 from the image IM. However, the number of characteristic parts FP is not limited to four, and may be two to three, or five or more.

[0042] (C4) In the first and second embodiments, the front, rear, left and right wheels 150 of the vehicle 100 are used as the characteristic parts FP of the vehicle 100. However, the characteristic parts FP of the vehicle 100 do not have to be the front, rear, left and right wheels 150 of the vehicle 100. For example, the left and right headlamps and the left and right tail lamps of the vehicle 100 may be used as the characteristic parts FP of the vehicle 100. In this case as well, the position and orientation of the vehicle 100 can be accurately estimated.

[0043] (C5) In the first and second embodiments, the position estimation units 212, 192 estimate the position and orientation of the vehicle 100 using skeletal information of the vehicle 100. In contrast, the position estimation units 212, 192 may estimate the position of the vehicle 100 using skeletal information of the vehicle 100, but may not necessarily estimate the orientation of the vehicle 100. The position estimation units 212, 192 may obtain a movement vector of the vehicle 100 from position changes of predetermined feature points of the vehicle 100 using, for example, an optical flow method, without using the skeletal information of the vehicle 100, and estimate the orientation of the vehicle 100 based on the orientation of the movement vector.

[0044] (C6) In the first embodiment, the server device 200 executes the processes from acquiring the position information of the vehicle 100 to generating the driving control signal. However, at least a part of the processes from acquiring the position information of the vehicle 100 to generating the driving control signal may be executed by the vehicle 100. For example, the following forms (1) to (3) may be used.

[0045] (1) The server device 200 may acquire position information of the vehicle 100, determine a target position to which the vehicle 100 should next head, and generate a route from the current location of the vehicle 100 indicated in the acquired position information to the target position. The server device 200 may generate a route to a target position between the current location and the destination, or may generate a route to the destination. The server device 200 may transmit the generated route to the vehicle 100. The vehicle 100 may generate a driving control signal so that the vehicle 100 drives on the route received from the server device 200, and control the actuator group 120 using the generated driving control signal.

[0046] (2) The server device 200 may acquire location information of the vehicle 100 and transmit the acquired location information to the vehicle 100. The vehicle 100 may determine a target location to which the vehicle 100 should next travel, generate a route from the current location of the vehicle 100 indicated in the received location information to the target location, generate a travel control signal so that the vehicle 100 travels on the generated route, and control the actuator group 120 using the generated travel control signal. Note that in each of the above-described embodiments, the vehicle operation information may be a route from the current location of the vehicle 100 to the target location.

[0047] (3) In the above embodiments (1) and (2), the vehicle 100 may be equipped with an internal sensor, and detection results output from the internal sensor may be used for at least one of generating a route and generating a driving control signal. The internal sensor may include, for example, a camera, LiDAR, millimeter-wave radar, an ultrasonic sensor, a GPS sensor, an acceleration sensor, and a gyro sensor. For example, in the above embodiment (1), the server device 200 may acquire the detection results of the internal sensor and reflect the detection results of the internal sensor in the route when generating a route. In the above embodiment (1), the vehicle 100 may acquire the detection results of the internal sensor and reflect the detection results of the internal sensor in the driving control signal when generating a driving control signal. In the above embodiment (2), the vehicle 100 may acquire the detection results of the internal sensor and reflect the detection results of the internal sensor in the route when generating a route. In the above embodiment (2), the vehicle 100 may acquire the detection results of the internal sensor and reflect the detection results of the internal sensor in the route when generating a route.

[0048] (C7) In the second embodiment, the vehicle 100 may be equipped with an internal sensor, and the detection results output from the internal sensor may be used for at least one of generating a route and generating a driving control signal. For example, the vehicle 100 may acquire the detection results of the internal sensor and, when generating a route, reflect the detection results of the internal sensor in the route. The vehicle 100 may acquire the detection results of the internal sensor and, when generating a driving control signal, reflect the detection results of the internal sensor in the driving control signal.

[0049] (C8) In the second embodiment, the vehicle 100 acquires vehicle position information using the image IM acquired from the camera 300. In contrast, the vehicle 100 may be equipped with an internal sensor, and the vehicle 100 may acquire vehicle position information using the detection results of the internal sensor, determine a target position to which the vehicle 100 should next head, generate a route from the current location of the vehicle 100 represented in the acquired vehicle position information to the target position, generate a driving control signal for traveling along the generated route, and control the actuator group 120 using the generated driving control signal. In this case, the vehicle 100 can travel without using the image IM of the camera 300 or the detection results of an external sensor located outside the vehicle 100. Note that the vehicle 100 may acquire a target arrival time and traffic congestion information from outside the vehicle 100 and reflect the target arrival time and traffic congestion information in at least one of the route and the driving control signal.

[0050] (C9) In the first embodiment described above, the server device 200 automatically generates the driving control signal to be transmitted to the vehicle 100. However, the server device 200 may generate the driving control signal to be transmitted to the vehicle 100 in accordance with the operation of an external operator located outside the vehicle 100. For example, the external operator may operate a control device including a display that displays the image IM output from the camera 300, a steering wheel for remotely controlling the vehicle 100, an accelerator pedal, a brake pedal, and a communication device for communicating with the server device 200 via wired or wireless communication, and the server device 200 may generate the driving control signal in accordance with the operation applied to the control device.

[0051] (C10) In the first and second embodiments, the vehicle 100 may be configured to be able to travel by unmanned driving, and may be in the form of a platform having the configuration described below, for example. Specifically, the vehicle 100 may be equipped with at least a vehicle control device 110, an actuator group 120, and wheels 150 to perform the three functions of "running," "turning," and "stopping" by unmanned driving. When the vehicle 100 acquires information from the outside for unmanned driving, the vehicle 100 may further be equipped with a communication device 130. In other words, the vehicle 100 that can travel by unmanned driving may not be equipped with at least some of the interior parts such as a driver's seat and a dashboard, may not be equipped with at least some of the exterior parts such as a bumper and a fender, and may not be equipped with a body shell. In this case, the remaining parts such as the body shell may be attached to the vehicle 100 before the vehicle 100 is shipped from the factory FC, or the remaining parts such as the body shell may be attached to the vehicle 100 after the vehicle 100 is shipped from the factory FC without the remaining parts such as the body shell being attached to the vehicle 100. Each part may be attached from any direction, such as the upper side, lower side, front side, rear side, right side, or left side of the vehicle 100, and may be attached from the same direction or from different directions. Note that the position of the platform configuration may also be determined in the same way as for the vehicle 100 in the first embodiment.

[0052] (C11) The vehicle 100 may be manufactured by combining multiple modules. A module refers to a unit composed of multiple parts grouped according to the location or function of the vehicle 100. For example, the platform of the vehicle 100 may be manufactured by combining a front module that forms the front part of the platform, a central module that forms the center part of the platform, and a rear module that forms the rear part of the platform. The number of modules that form the platform is not limited to three, but may be two or less, or four or more. In addition to or instead of the parts that form the platform, parts that form parts of the vehicle 100 other than the platform may be modularized. The various modules may include any exterior parts such as a bumper or a grille, or any interior parts such as a seat or a console. In addition to the vehicle 100, any type of mobile object may be manufactured by combining multiple modules. Such a module may be manufactured, for example, by joining multiple parts by welding or fasteners, or by integrally molding at least some of the parts that form the module into a single part by casting. The molding method for integrally molding a single component, particularly a relatively large component, is also called gigacasting or megacasting. For example, the front module, center module, and rear module described above may be manufactured using gigacasting.

[0053] (C12) Transporting vehicle 100 by using the unmanned driving of vehicle 100 is also called "self-propelled transport." The configuration for realizing self-propelled transport is also called a "vehicle remote-controlled autonomous transport system." The production method for producing vehicle 100 by using self-propelled transport is also called "self-propelled production." In self-propelled production, for example, at a factory FC where vehicle 100 is manufactured, at least a portion of the transport of vehicle 100 is realized by self-propelled transport.

[0054] (C13) In the first and second embodiments, some or all of the functions and processes implemented by software may be implemented by hardware. Furthermore, some or all of the functions and processes implemented by hardware may be implemented by software. Hardware for implementing the various functions in each of the above embodiments may be implemented by various circuits, such as integrated circuits and discrete circuits.

[0055] The present disclosure is not limited to the above-described embodiments and can be realized in various configurations without departing from the spirit thereof. For example, the technical features in the embodiments corresponding to the technical features in each aspect described in the Summary of the Invention section can be appropriately replaced or combined to solve some or all of the above-described problems or achieve some or all of the above-described effects. Furthermore, if a technical feature is not described as essential in this specification, it can be appropriately deleted. [Explanation of symbols]

[0056] 10, 10b...unmanned driving system, 100...vehicle, 110...vehicle control device, 111...processor, 112...memory, 113...input / output interface, 114...internal bus, 120...actuator group, 130...communication device, 150...wheel, 191...image acquisition unit, 192...position estimation unit, 195...driving control unit, 200...server device, 201...processor, 202...memory, 203...input / output interface, 204...internal bus, 205...communication device, 211...image acquisition unit, 212...position estimation unit, 215...remote control unit, 300...camera, FC...factory, FP...feature unit, IM...image, LS...line segment, MD...detection model, OB...object, PG1, PG2...computer program, PL1...first location, PL2...second location, RG...area, RR...reference path, SK...skeleton, TR...road

Claims

1. A vehicle positioning device, an image acquisition unit that acquires an image of the vehicle from a camera located outside the unmanned vehicle; a position estimation unit that estimates a position of the vehicle using the image, the position estimation unit detecting skeleton information including positions of a plurality of predetermined characteristic parts of the vehicle from the image, and estimating the position of the vehicle using the skeleton information; A vehicle positioning device comprising:

2. 2. The vehicle positioning device according to claim 1, The vehicle positioning device, wherein the characteristic part is a wheel of the vehicle.

3. 2. The vehicle positioning device according to claim 1, The position estimation unit determines an area including the vehicle from the image, and detects the skeleton information from the area.

4. 2. The vehicle positioning device according to claim 1, The vehicle positioning device further includes a control unit that causes the vehicle to travel in an unmanned driving mode using the estimation result of the position estimation unit.

5. A vehicle positioning method, comprising: an image acquisition step of acquiring an image of the vehicle from a camera located outside the vehicle that is capable of traveling by unmanned driving; a position estimation step of estimating a position of the vehicle using the image, the position estimation step including detecting skeleton information including positions of a plurality of predetermined characteristic parts of the vehicle from the image, and estimating a position of the vehicle using the skeleton information; A vehicle positioning method comprising:

Images