Detection device, position calculation system, and detection method

Abstract

Provided is a technique capable of accurately detecting a vehicle included in a captured image. A detection device that detects a moving body included in a captured image includes: an image acquisition unit that acquires the captured image; a type acquisition unit that acquires type information indicating one type of vehicle body type of the moving body included in the captured image classified according to the appearance shape of the moving body; a model acquisition unit that acquires a first detection model selected on the basis of one vehicle body type determined by the type information acquired by the type acquisition unit from among a plurality of first detection models, which are machine learning models prepared for each vehicle body type; and a detection unit that detects a moving body included in the captured image by inputting the captured image into the first detection model acquired by the model acquisition unit and specifying a target region indicating the moving body in the captured image.

Description

Technical Field

[0001] The present disclosure relates to a detection device, a position calculation system and a detection method. Background Art

[0002] Conventionally, there is known a vehicle that travels without human control (Patent Document 1). Prior art literature Patent Literature

[0003] Patent Document 1: Japanese Patent Application No. 2017-538619 Summary of the invention Problems to be solved by the invention

[0004] Sometimes, in order to move a moving object such as a vehicle without human control, the moving object is photographed from the outside. However, the inventors of the present application have found that the detection accuracy of the moving object contained in the photographed image may be reduced because the appearance of the moving object varies depending on the type of vehicle body. Technical means to solve problems

[0005] The present disclosure can be implemented as the following aspects.

[0006] (1) According to a first aspect of the present disclosure, a detection device for detecting a moving object included in a captured image is provided. The detection device includes: an image acquisition unit that acquires the captured image; a type acquisition unit that acquires type information indicating a type of vehicle body of the moving object included in the captured image and classified according to the appearance shape of the moving object; a model acquisition unit that acquires a first detection model selected from a plurality of first detection models that are machine learning models prepared for each type of vehicle body according to the type of vehicle body determined by the type information acquired by the type acquisition unit; and a detection unit that detects the moving object included in the captured image by inputting the captured image to the first detection model acquired by the model acquisition unit to determine an object region representing the moving object in the captured image. According to this aspect, by acquiring the type information indicating a type of vehicle body of the moving object included in the captured image, it is possible to acquire a first detection model selected from a plurality of first detection models prepared for each type of vehicle body according to the type of vehicle body determined by the acquired type information. Furthermore, by inputting the captured image to the first detection model corresponding to the vehicle body type of the moving object included in the captured image, it is possible to detect the moving object included in the captured image with high accuracy. That is, by using a first detection model suitable for detecting a moving object classified as a vehicle body type, each area constituting a captured image can be accurately classified into an object area and an area outside the object area. Based on the above, it is possible to suppress the detection accuracy of a moving object included in a captured image from being reduced depending on the vehicle body type of the moving object. (2) In the above-mentioned method, the plurality of first detection models may be respectively learned in advance in such a manner that the object area is determined by inputting a plurality of training images, the plurality of training images including M (M is an integer greater than or equal to 2) first training images each including the mobile body classified as one of the vehicle body types, and N (N is an integer greater than or equal to 0 and less than M) second training images each including the mobile body classified as another vehicle body type different from the one vehicle body type. According to this method, by learning more first training images each including the mobile body classified as one of the vehicle body types than second training images each including the mobile body classified as another vehicle body type different from the one vehicle body type, a first detection model suitable for detecting a mobile body classified as one of the vehicle body types can be prepared for each vehicle body type. (3) According to a second aspect of the present disclosure, a detection device for detecting a moving object included in a captured image is provided. The detection device includes: an image acquisition unit that acquires the captured image; a type acquisition unit that acquires type information indicating a type of vehicle body of the moving object included in the captured image, which is classified according to the appearance shape of the moving object; and a detection unit that detects the moving object included in the captured image by inputting the captured image and the type of vehicle body determined by the type information acquired by the type acquisition unit into a second detection model as a machine learning model to determine an object region in the captured image indicating the moving object. According to this aspect, by inputting a captured image associated with a type of vehicle body into the second detection model, the moving object included in the captured image can be detected with good accuracy. That is, each region constituting the captured image can be classified with good accuracy into an object region and a region other than the object region. According to the above, it is possible to suppress the decrease in the detection accuracy of the moving object included in the captured image depending on the vehicle body type of the moving object. (4) In the above-mentioned method, the second detection model may be preliminarily learned in a manner to determine the object area by inputting a plurality of training images including the mobile body and a type correct answer label associated with each of the plurality of training images and indicating the body type of the mobile body included in the training image. According to this method, it is possible to prepare a second detection model that is learned by associating a plurality of training images including the mobile body with a type correct answer label indicating the body type of the mobile body included in each training image. (5) In the above method, for each area in the training image, an area correct answer label indicating whether it is the object area or an out-of-object area other than the moving object is associated. According to this method, each area constituting the captured image can be accurately classified into the object area and the out-of-object area. As a result, the detection accuracy of the moving object included in the captured image can be improved. (6) In the above-mentioned method, the vehicle body type may represent the category of the mobile object. According to this method, the mobile object included in the captured image can be detected using any one of the first detection model and the second detection model that have learned the difference in the appearance shape of the mobile object that differs according to the category of the mobile object. As a result, the detection accuracy of the mobile object included in the captured image can be improved. (7) In the above-mentioned aspect, the mobile body may move by receiving a moving condition that specifies a moving action of the mobile body from outside the mobile body, and the type acquisition unit may acquire the type information by determining the one vehicle body type of the mobile body included in the captured image using predetermined information, the predetermined information being associated with each of the capturing ranges of a plurality of capturing devices that acquire the captured image and indicating a predetermined time at which the mobile body classified as the one vehicle body type moves within the capturing range. According to this aspect, by determining the one vehicle body type of the mobile body included in the captured image using predetermined information, the type information about the mobile body included in the captured image can be acquired. (8) In the above-mentioned method, the type acquisition unit may acquire the type information by inputting the captured image to a determination model that is a machine learning model that has been learned in such a way that it outputs the type information when the captured image is input. According to this method, the type information about the moving object included in the captured image can be acquired by inputting the captured image to the determination model that has been learned in such a way that it outputs the type information about the moving object included in the captured image. (9) In the above method, it is also possible to further include: a rotation processing unit that rotates the captured image in such a manner that the moving direction of the moving object is oriented toward a predetermined direction. According to this method, the captured image can be rotated in such a manner that the direction of the vector representing the moving direction of the moving object is oriented toward a predetermined direction. In this way, the moving object included in the captured image can be detected in a state where the direction of the vector representing the moving direction of the moving object is unified. In this way, the detection accuracy of the moving object included in the captured image can be improved. (10) In the above aspect, the device may further include a distortion correction unit that corrects the distortion of the captured image. According to this aspect, the distortion of the captured image can be corrected. As a result, the detection accuracy of the moving object included in the captured image can be improved. (11) According to a third aspect of the present disclosure, there is provided a position calculation system for calculating the position of a moving object included in a captured image. The position calculation system comprises: a detection device described in the above-mentioned mode; and a position calculation device for calculating the position of the moving body, wherein the detection device further generates a first mask image in which the mask area is added to the object area by masking the area representing the moving body in the captured image, and the position calculation device comprises: a perspective transformation unit for generating a second mask image by performing perspective transformation on the first mask image; a coordinate point calculation unit for calculating an image coordinate point representing the position of the moving body in the image coordinate system by setting a designated vertex of a first circumscribed rectangle set in the mask area in the first mask image as a first coordinate point, setting a vertex in a second circumscribed rectangle set in the mask area in the second mask image that indicates the same position as the first coordinate point as a second coordinate point, and correcting the first coordinate point using the second coordinate point; and a position transformation unit for transforming the image coordinate point into a moving body coordinate point representing the position of the moving body in the global coordinate system using a distance from the capturing device to a reference point calculated based on the position of the capturing device in the global coordinate system and a distance from a predetermined positioning point of the moving body to the reference point. According to this method, a first mask image that masks the target area representing the moving object in each area constituting the captured image and a second mask image that performs perspective transformation on the first mask image can be generated. In this way, the image coordinate points can be calculated by extracting the first coordinate points from the first mask image and the second coordinate points from the second mask image. As a result, the image coordinate points can be calculated more accurately. Therefore, the calculation accuracy of the position of the moving object can be improved. (12) According to a fourth aspect of the present disclosure, a detection method for detecting a moving object included in a captured image is provided. The detection method comprises: an image acquisition step for acquiring the captured image; a type acquisition step for determining a body type of the moving object included in the captured image that is classified according to the appearance shape of the moving object; a model acquisition step for acquiring a first detection model selected from a plurality of first detection models that are machine learning models prepared for each body type according to the body type determined in the type acquisition step; and a detection step for inputting the captured image to the first detection model acquired in the model acquisition step to determine an object region representing the moving object in the captured image, thereby detecting the moving object included in the captured image. According to this aspect, by acquiring type information representing a body type of the moving object included in the captured image, it is possible to acquire a first detection model selected from a plurality of first detection models prepared for each body type according to a body type determined by the acquired type information. Furthermore, by inputting the captured image to the first detection model corresponding to the body type of the moving object included in the captured image, it is possible to detect the moving object included in the captured image with good accuracy. That is, by using a first detection model suitable for detecting a moving object classified as a vehicle body type, each area constituting a captured image can be accurately classified into an object area and an area outside the object area. Based on the above, it is possible to suppress the detection accuracy of a moving object included in a captured image from being reduced depending on the vehicle body type of the moving object. (13) According to a fifth aspect of the present disclosure, a detection method for detecting a moving object included in a captured image is provided. The detection method comprises: an image acquisition step for acquiring the captured image; a type acquisition step for acquiring type information indicating a type of vehicle body of the moving object included in the captured image, which is classified according to the appearance shape of the moving object; and a detection step for inputting the captured image and the type of vehicle body determined by the type information acquired in the type acquisition step into a second detection model as a machine learning model to determine an object region in the captured image indicating the moving object, thereby detecting the moving object included in the captured image. According to this aspect, by inputting the captured image associated with a type of vehicle body into the second detection model, the moving object included in the captured image can be detected with good accuracy. That is, each region constituting the captured image can be classified into an object region and a region other than the object region with good accuracy. According to the above, it is possible to suppress the decrease in the detection accuracy of the moving object included in the captured image depending on the vehicle body type of the moving object. The present disclosure can be implemented in various ways other than the above-mentioned detection device, position calculation system and detection method. For example, it can be implemented in the form of a manufacturing method of a mobile body, a detection device and a position calculation system, a position calculation method, a control method of a detection device and a position calculation system, a computer program for implementing the control method, a non-temporary recording medium recording the computer program, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 It is a diagram showing a schematic configuration of a position calculation system in the first embodiment. Figure 2 It is a diagram showing a schematic configuration of a detection device in the first embodiment. Figure 3 This is a diagram showing details of a CPU mounted on the detection device in the first embodiment. Figure 4 It is a diagram showing a schematic configuration of a position calculation device. Figure 5 It is a diagram showing a schematic configuration of a remote control device. Figure 6 This is a flowchart showing the detection method and position calculation method in the first embodiment. Figure 7 2 is a schematic diagram showing examples of various images. Figure 8 This is a diagram for explaining the details of the coordinate point calculation process. Fig. 9 This is a diagram for explaining a method of acquiring basic coordinate points. Fig.10 This is the first diagram for explaining the details of the position conversion process. Fig.11 This is a second diagram for explaining the details of the position conversion process. Fig. 12A This is a flowchart showing the processing procedure of the travel control of the vehicle in the first embodiment. Fig. 12B is a flowchart showing a method for controlling the behavior of a vehicle. Fig.13 It is a diagram showing a schematic configuration of a detection device in a second embodiment. Fig.14 It is a diagram showing the details of the CPU mounted on the detection device in the second embodiment. Fig.15 This is a flowchart showing the detection method and position calculation method in the second embodiment. Fig.16 It is a diagram showing a schematic configuration of a position calculation system in the third embodiment. Fig.17It is a diagram showing a schematic configuration of a vehicle control device in a third embodiment. Fig.18 This is a flowchart showing a processing procedure of driving control of a vehicle in the third embodiment. DETAILED DESCRIPTION

[0008] A. First Implementation Method: A-1. Composition of the position calculation system: Figure 1 : is a diagram showing a schematic structure of a position calculation system 1. The position calculation system 1 is a system that detects a vehicle 10 included in a captured image and calculates the position of the vehicle 10. In the present disclosure, a captured image is an original image as raw data obtained by a camera 9 as an external device installed in a place different from the vehicle 10, and at least one of a corrected image, a rotated image, and a processed image after processing the original image. The details of the corrected image, the rotated image, and the processed image will be described later. The position calculation system 1 includes one or more vehicles 10, a detection device 5 that detects the vehicle 10 included in the captured image, and a position calculation device 6 that calculates the position of the vehicle 10 using the detection result of the detection device 5.

[0009] The vehicle 10 is an example of a mobile body. In the present disclosure, a "mobile body" means an object that can move, such as a vehicle, an electric vertical take-off and landing aircraft (a so-called flying car). The vehicle can be a vehicle that runs on wheels or a vehicle that runs on tracks, such as a passenger car, a truck, a bus, a two-wheeled vehicle, a four-wheeled vehicle, a tank, an engineering vehicle, etc. Vehicles include electric vehicles (BEV: Battery Electric Vehicle), gasoline vehicles, hybrid vehicles, and fuel cell vehicles. In the case where the mobile body is other than a vehicle, the expressions "vehicle" and "car" in the present disclosure can be appropriately replaced with "mobile body", and the expression "running" can be appropriately replaced with "moving".

[0010] The vehicle 10 is configured to be able to travel by unmanned driving. "Unmanned driving" means driving that does not rely on the driving operation of the passengers. The driving operation means an operation related to at least one of "driving", "steering" and "stopping" of the vehicle 10. Unmanned driving is achieved by automatic or manual remote control using a device located outside the vehicle 10, or autonomous control of the vehicle 10. In the vehicle 10 that travels by unmanned driving, there may also be passengers who do not perform driving operations. Passengers who do not perform driving operations include, for example, people who only sit on the seats of the vehicle 10, and people who perform operations other than driving operations such as assembly, inspection, and operation of switch groups while riding in the vehicle 10. In addition, driving based on the driving operation of the passengers is sometimes referred to as "manned driving".

[0011] In this specification, “remote control” includes “full remote control” in which all actions of the vehicle 10 are determined completely from outside the vehicle 10, and “partial remote control” in which part of the actions of the vehicle 10 are determined from outside the vehicle 10. In addition, “autonomous control” includes “full autonomous control” in which the vehicle 10 autonomously controls its own actions without receiving any information from a device outside the vehicle 10, and “partial autonomous control” in which the vehicle 10 autonomously controls its own actions using information received from a device outside the vehicle 10.

[0012] The photographing device 9 obtains an original image by photographing a photographing range RG including the vehicle 10 as the object of position calculation from the outside of the vehicle 10. In the present embodiment, the photographing device 9 sends the obtained original image together with the camera identification information and the time of obtaining the original image to the detection device 5. The camera identification information is information such as the camera ID for identifying multiple photographing devices 9. The original image is a two-dimensional image formed by a set of pixels arranged on the XcYc plane of the camera coordinate system. The camera coordinate system is a coordinate system that takes the focus of the photographing device 9 as the origin and has coordinate axes represented by the Xc axis and the Yc axis orthogonal to the Xc axis. The original image contains at least two-dimensional data of the vehicle 10 as the object of position calculation. As the original image, a color image is preferred, but it can also be a black and white image. The photographing device 9 is, for example, a camera having a photographing element and an optical system such as a CCD image sensor or a CMOS image sensor.

[0013] In the present embodiment, the camera 9 obtains an original image of the driving road 2 and the vehicle 10 traveling on the driving road 2 from above. In order to photograph the entire driving road 2 by one or more camera 9, the installation position and the number of the camera 9 are determined in consideration of the shooting range RG (angle of view) of the camera 9, etc. Specifically, each camera 9 is installed in such a manner that the shooting range RG of the adjacent first camera 901, i.e., the first shooting range RG1, and the shooting range RG of the second camera 902, i.e., the second shooting range RG2, overlap. Furthermore, each camera 9 is installed at a position where a positioning point 10e pre-set for a specific part of the vehicle 10 traveling on the driving road 2 can be photographed. In the present embodiment, the positioning point 10e is the rear end (hereinafter, the left rear end) of the left side of the vehicle 10. In addition, the positioning point 10e may be other than the left rear end of the vehicle 10. In addition, the camera 9 is not limited to obtaining information from above the vehicle 10, but may also obtain information from the front, rear, side, etc. of the vehicle 10.

[0014] The vehicle 10 has a manned driving mode and a remote unmanned driving mode. In the manned driving mode, the vehicle 10 is driven by the manned driving described above. Specifically, in the manned driving mode, the driver of the vehicle 10 operates the input devices such as the steering wheel and accelerator provided in the vehicle 10, thereby generating the driving conditions of the vehicle 10. Thus, the vehicle 10 drives according to the generated driving conditions. The driving conditions are conditions that specify the driving action of the vehicle 10. The driving conditions include, for example, the driving route, position, driving speed, acceleration, and steering angle of the wheels of the vehicle 10. The remote unmanned driving mode has a remote manual driving mode and a remote automatic driving mode. In the remote manual driving mode, the operator operates the operator input device provided at a location different from the vehicle 10, thereby generating the driving conditions of the vehicle 10. Thus, the vehicle 10 receives the driving conditions generated by the operator input device and drives according to the received driving conditions. The operator mentioned here is equivalent to the external operator described later. In addition, in the remote automatic driving mode, the remote control device 7 provided at a location different from the vehicle 10 generates a control value that specifies the driving action of the vehicle 10 and sends it to the vehicle 10. Thus, the vehicle 10 receives the control value and performs automatic driving according to the received control value. Specifically, in the remote automatic driving mode in the present embodiment, the remote control device 7 generates a driving control signal described below and sends it to the vehicle 10. The driving control signal includes a control value for specifying a driving action and indicates a driving condition.

[0015] The vehicle 10, for example, performs a plurality of manufacturing processes and thus travels in a factory where the vehicle 10 is manufactured in a remote automatic driving mode. The factory is not limited to being in one building, one land, or one residence, but may exist across multiple buildings, multiple lands, multiple residences, etc. In this case, the vehicle 10 is not limited to being driven on private roads, but may also be driven on public roads. In addition, the vehicle 10 may also be driven outside the factory in a remote unmanned driving mode.

[0016] The vehicle 10 is, for example, an electric vehicle, a hybrid vehicle, a fuel cell vehicle, a gasoline vehicle, or a diesel vehicle. The vehicle 10 may be a household vehicle such as a passenger car, or a commercial vehicle such as a truck, a bus, or a construction vehicle. The vehicle 10 may be at least one of a finished product, a semi-finished product, or a work in progress.

[0017] The vehicle 10 includes a driving device 110 for accelerating the vehicle 10, a steering device 120 for changing the traveling direction of the vehicle 10, and a braking device 130 for decelerating the vehicle 10. The driving device 110, the steering device 120, and the braking device 130 each include various actuators for driving the vehicle 100. The vehicle 10 also includes a vehicle communication unit 140 for communicating with external devices using wireless communication, a vehicle control device 150, and a vehicle-mounted sensor group 160. The external device is other devices other than the own vehicle 10, such as the detection device 5, the position calculation device 6, the remote control device 7, and the camera 9. The vehicle communication unit 140 is, for example, a wireless communication device. The vehicle communication unit 140 communicates with external devices connected to the network Nt via, for example, an access point in a factory. The vehicle control device 150 includes a CPU, a storage unit, and an input / output interface. In the vehicle control device 150, the CPU, the storage unit, and the input / output interface are connected to each other, for example, via an internal bus and an interface circuit. The input / output interface communicates with internal devices such as the drive device 110 mounted on the own vehicle 10. The input / output interface is connected to the vehicle communication unit 140 so as to be communicable. The vehicle-mounted sensor group 160 includes, for example, a vehicle-mounted camera, a vehicle-mounted radar, and a vehicle-mounted laser radar as sensors for obtaining peripheral information indicating the state of the surrounding area of ​​the vehicle 10. The vehicle-mounted camera captures the state of the surrounding area of ​​the vehicle 10. The vehicle-mounted radar and the vehicle-mounted laser radar detect objects existing in the surrounding area of ​​the vehicle 10. In addition, the configuration of the vehicle 10 is not limited to the above.

[0018] Figure 2 1 is a diagram showing a schematic configuration of the detection device 5 in the first embodiment. The detection device 5 detects at least the outer shape (outline) of the vehicle 10 included in the captured image by determining a target area as an area representing the vehicle 10 among various areas in the captured image. The detection device 5 includes a communication unit 51, a storage unit 53, and a CPU 52. In the detection device 5, the communication unit 51, the storage unit 53, and the CPU 52 are connected to each other via, for example, an internal bus and an interface circuit.

[0019] The communication unit 51 of the detection device 5 connects other devices such as the vehicle control device 150, the position calculation device 6, the remote control device 7, and the imaging device 9 to the detection device 5 so as to be communicable. The communication unit 51 of the detection device 5 is, for example, a wireless communication device.

[0020] The storage unit 53 of the detection device 5 stores various information including various programs for controlling the operation of the detection device 5, the first detection model Md1 as a detection model, the determination model Md3, and the distortion correction parameter Pa1. The storage unit 53 of the detection device 5 includes, for example, a RAM, a ROM, and a hard disk drive (HDD).

[0021] The detection model is a machine learning model that has been learned and used to detect the vehicle 10 included in the captured image. In the present embodiment, the storage unit 53 of the detection device 5 stores a plurality of first detection models Md1 prepared for each vehicle body type as the detection model. In the present embodiment, the vehicle body type is a group of vehicle types when a plurality of categories of the vehicle 10 are classified in a collective manner according to the appearance shape of the vehicle 10. In this case, the vehicle 10 is classified into a vehicle body type according to, for example, the vehicle class (also referred to as "vehicle body type", "body size") determined by the full length, vehicle width, and vehicle height of the vehicle 10 and the shape of the vehicle 10. In the present embodiment, the vehicle body type is, for example, "SUV", "sedan", "van", "minivan", "single box vehicle", "compact vehicle", and "light vehicle". In addition, the types of vehicle body types are not limited to the above.

[0022] The first detection model Md1 is a machine learning model that is learned by inputting the first teacher data group. The first teacher data group is prepared for each vehicle body type. The first teacher data group has a plurality of training images including the vehicle 10, and a region correct answer label associated with each of the plurality of regions constituting each training image. The region correct answer label is a correct answer label indicating whether each region in the training image is an object region representing the vehicle 10 or an object non-region representing a region other than the vehicle 10. In the present embodiment, each region constituting the training image is a pixel constituting the training image. In addition, each region constituting the training image may also be a plurality of pixels constituting the training image. Each region constituting the training image is classified as one of the object region and the object non-region, for example, based on the calculation result when comparing the RGB values ​​of adjacent regions constituting the training image. Each region constituting the training image may also be classified as one of the object region and the object non-region according to the calculated probability by calculating the probability of being the object region. When the region correct answer label is attached to each region constituting the training image, information indicating the position of the region with the region correct answer label in the training image is associated. When each region constituting the training image is one pixel, the information indicating the position in the training image is, for example, pixel coordinates indicating the pixel position in the training image.

[0023] The first teacher data set includes M (M is an integer greater than or equal to 2) first training images and N (N is an integer greater than or equal to 0 and less than M) second training images as a plurality of training images. The first training image is an image including a vehicle 10 classified as one body type. The second training image is an image including a vehicle 10 classified as another body type different from the one body type. The number M of first training images included in the first teacher data set may be, for example, more than 1.5 times the number N of second training images, or more than 2 times. In addition, in the case where there are a plurality of other body types different from one body type, the number M of first training images included in the first teacher data set is greater than the total number of second training images, for example, including vehicles 10 classified as a plurality of other body types different from one body type. That is, the first detection model Md1 is a machine learning model that preferentially learns the feature values ​​of vehicles 10 classified as one body type by learning more about the first training images than the second training images. Figure 2 In the figure, as an example of the first detection model Md1, the following three first detection models Md1a to Md1c are illustrated. The first first detection model Md1a is a first detection model Md1 that has been learned more for vehicles 10 classified as "SUV" than for vehicles 10 classified as body types other than "SUV". The second first detection model Md1b is a first detection model Md1 that has been learned more for vehicles 10 classified as "sedan" than for vehicles 10 classified as body types other than "sedan". The third first detection model Md1c is a first detection model Md1 that has been learned more for vehicles 10 classified as "passenger and cargo vehicle" than for vehicles 10 classified as body types other than "passenger and cargo vehicle". In addition, the number and types of first detection models Md1 are not limited to the above.

[0024] In this embodiment, when a captured image is input, the first detection model Md1 specifies a target region among regions constituting the input captured image, and then masks the target region to generate a first mask image in which the mask region is added to the target region.

[0025] As the algorithm of the first detection model Md1, a deep neural network (hereinafter, DNN) having a structure of a convolutional neural network (hereinafter, CNN) for implementing semantic segmentation and instance segmentation is used. In addition, the structure of the first detection model Md1 is not limited to the above. The first detection model Md1 can also be a machine learning model that has been learned using an algorithm other than a neural network.

[0026] The determination model Md3 is a learned machine learning model used to determine a body type of the vehicle 10 included in the captured image and obtain type information representing the determined body type. Specifically, the determination model Md3 is a machine learning model that has been learned in a manner that outputs type information about the vehicle 10 included in the captured image when the captured image is input. The determination model Md3 learns feature quantities corresponding to the body type in order to determine a body type of the vehicle 10. Feature quantities corresponding to the body type are, for example, the shape and compartment of the vehicle 10. As an algorithm for determining the model Md3, for example, CNN is used. In addition, the structure of the determination model Md3 is not limited to the above. The determination model Md3 may also be a learned machine learning model that uses, for example, an algorithm other than a neural network.

[0027] The distortion correction parameter Pa1 is a parameter used when correcting the distortion of a captured image. The details of the distortion correction parameter Pa1 will be described later.

[0028] Figure 3 1 is a diagram showing details of the CPU 52 mounted on the detection device 5 in the first embodiment. The CPU 52 of the detection device 5 expands various programs stored in the storage unit 53 of the detection device 5 to function as an image acquisition unit 521, a distortion correction unit 522, a rotation processing unit 523, a cropping processing unit 524, and a type acquisition unit 525. Furthermore, the CPU 52 of the detection device 5 expands various programs stored in the storage unit 53 of the detection device 5 to function as a model acquisition unit 526, a detection unit 527, and a first transmission unit 528.

[0029] The image acquisition unit 521 acquires an original image from the imaging device 9 which is an external device.

[0030] The distortion correction unit 522 corrects the distortion of the captured image. In the present embodiment, the distortion correction unit 522 generates a corrected image by correcting the distortion of the original image.

[0031] The rotation processing unit 523 rotates the captured image so that the direction of a vector indicating the moving direction of the vehicle 10 (hereinafter referred to as a moving vector) faces a predetermined direction. In the present embodiment, the rotation processing unit 523 generates a rotated image by rotating the corrected image.

[0032] The cropping processing unit 524 deletes the area (hereinafter, unnecessary area) other than the area (hereinafter, necessary area) consisting of the vehicle 10 and the surrounding area of ​​the vehicle 10 from the captured image. Thus, the cropping processing unit 524 crops out the necessary area from the captured image. In the present embodiment, when the vehicle 10 moves a distance exceeding a predetermined threshold, the cropping processing unit 524 deletes the moved area corresponding to the distance moved by the vehicle 10 as an unnecessary area from the rotated image. Thus, the cropping processing unit 524 generates a processed image in which the unmoved area including the vehicle 10 as the necessary area is cropped out from the rotated image.

[0033] The type acquisition unit 525 acquires type information indicating a type of vehicle body type of the vehicle 10 included in the captured image. In the present embodiment, the type acquisition unit 525 acquires type information indicating a type of vehicle body type of the vehicle 10 included in the captured image by inputting the processed image to the determination model Md3. In addition, the method of acquiring type information is not limited to the above.

[0034] The model acquisition unit 526 acquires the first detection model Md1 selected according to the one vehicle body type specified by the type information obtained by the type acquisition unit 525. In the present embodiment, the model acquisition unit 526 selects and acquires the first detection model Md1 corresponding to the one vehicle body type specified by the type information obtained by the type acquisition unit 525 from the plurality of first detection models Md1 prepared for each vehicle body type and stored in the storage unit 53 of the detection device 5. In other words, the model acquisition unit 526 acquires the first detection model Md1 that learns more about the feature quantity of the vehicle 10 classified as the one vehicle body type specified by the type information obtained by the type acquisition unit 525 than about the feature quantity of the vehicle 10 classified as another vehicle body type different from the one vehicle body type. For example, when the vehicle body type of the vehicle 10 included in the captured image is specified as "SUV" by the type information obtained by the type acquisition unit 525, the model acquisition unit 526 acquires the first detection model Md1a that learns more about the vehicle 10 classified as "SUV" than about the vehicle 10 classified as another vehicle body type.

[0035] The detection unit 527 detects the vehicle 10 included in the captured image by inputting the captured image to the first detection model Md1 acquired by the model acquisition unit 526 and determining the target region among the regions constituting the captured image. In the present embodiment, the detection unit 527 inputs the processed image to the first detection model Md1 acquired by the model acquisition unit 526. Thus, by determining the target region among the regions constituting the processed image, the first mask image is generated from the processed image.

[0036] The first transmission unit 528 transmits various information to other devices other than the detection device 5. The first transmission unit 528 transmits, for example, the detection result of the detection unit 527 to the position calculation device 6. In this embodiment, the detection result of the detection unit 527 is the first mask image. In addition, at least a part of the function of the detection device 5 can also be implemented as a function of any one of the vehicle control device 150, the position calculation device 6, the remote control device 7, and the imaging device 9.

[0037] Figure 4 1 is a diagram showing a schematic configuration of the position calculation device 6. The position calculation device 6 calculates the position of the vehicle 10 included in the captured image using the detection result of the detection device 5. The position calculation device 6 uses the positioning point 10e of the vehicle 10 as the position of the vehicle 10. In the present embodiment, the position calculation device 6 calculates the position of the vehicle 10 included in the captured image using the first mask image. The position calculation device 6 includes a communication unit 61, a storage unit 63, and a CPU 62. In the position calculation device 6, the communication unit 61, the storage unit 63, and the CPU 62 are connected to each other via, for example, an internal bus and an interface circuit.

[0038] The communication unit 61 of the position calculation device 6 connects the vehicle control device 150, the detection device 5, the remote control device 7, and the imaging device 9 to the position calculation device 6 so as to be communicable. The communication unit 61 of the position calculation device 6 is, for example, a wireless communication device.

[0039] The storage unit 63 of the position calculation device 6 stores various information including various programs for controlling the operation of the position calculation device 6, perspective transformation parameters Pa2, and camera database Db. The storage unit 63 of the position calculation device 6 includes, for example, RAM, ROM, and a hard disk drive (HDD).

[0040] The perspective transformation parameter Pa2 is a parameter used when perspective transformation is performed on the first mask image. The perspective transformation parameter Pa2 will be described in detail later.

[0041] The camera database Db is a database that shows, for each imaging device 9, imaging parameters calculated based on the installation position of the imaging device 9 in the global coordinate system. The imaging parameter is a parameter related to the distance between the imaging device 9 and a predetermined reference point. In this embodiment, the imaging parameter is the height H (described later) of the imaging device 9 from the road surface 20. Fig.10 ), is the height H relative to the road surface 20 where the vehicle 10 is located. The camera database Db is, for example, a database that associates the camera identification information with the shooting parameters of the shooting device 9 determined by the camera identification information. In addition, the structure of the camera database Db is not limited to the above. The camera identification information can also be associated with the shooting range RG, the installation position and the installation angle of the shooting device 9.

[0042] The CPU 62 of the position calculation device 6 develops various programs stored in the storage unit 63 of the position calculation device 6 to function as a data acquisition unit 621 , a perspective conversion unit 622 , a coordinate point calculation unit 623 , a position conversion unit 624 , and a second transmission unit 625 .

[0043] The data acquisition unit 621 acquires various information. For example, the data acquisition unit 621 acquires the detection result of the detection device 5 from the detection device 5. In the present embodiment, the data acquisition unit 621 acquires the first mask image from the detection device 5. In addition, the data acquisition unit 621 refers to the camera database Db stored in the storage unit 63 of the position calculation device 6 to acquire the imaging parameters of the imaging device 9, which is the acquisition source of the original image to be analyzed.

[0044] The perspective transformation unit 622 generates a second mask image after perspective transformation of the first mask image. The coordinate point calculation unit 623 uses the second coordinate point to correct the first coordinate point and calculate the image coordinate point. The first coordinate point is a coordinate point in the image coordinate system of a specified vertex of a first circumscribed rectangle set in a mask area in the first mask image. The second coordinate point is a coordinate point in the image coordinate system of a vertex in a second circumscribed rectangle set in a mask area in the second mask image that represents the same position as the first coordinate point. The image coordinate point is a coordinate point that represents the position of the vehicle 10 in the image coordinate system. The position transformation unit 624 uses the shooting parameters obtained by the data acquisition unit 621 and the image coordinate point calculated by the coordinate point calculation unit 623 to transform the image coordinate point into a vehicle coordinate point. The vehicle coordinate point is a coordinate point that represents the position of the vehicle 10 in the global coordinate system.

[0045] The second transmission unit 625 transmits various information to other devices other than the position calculation device 6. For example, the second transmission unit 625 transmits the vehicle coordinate point as information indicating the position of the vehicle 10 to the remote control device 7. In addition, at least a part of the function of the position calculation device 6 may be implemented as a function of any one of the vehicle control device 150, the detection device 5, the remote control device 7, and the imaging device 9.

[0046] Figure 5: is a diagram showing a schematic structure of the remote control device 7. The remote control device 7 uses the position information of the vehicle 10 and the like to generate a control value that specifies the driving action of the vehicle 10 and transmits it to the vehicle 10. Thus, the remote control device 7 remotely controls the action of the vehicle 10. The remote control device 7 includes a communication unit 71, a storage unit 73, and a CPU 72. In the remote control device 7, the communication unit 71, the storage unit 73, and the CPU 72 are connected to each other, for example, via an internal bus and an interface circuit. The communication unit 71 of the remote control device 7 connects the vehicle control device 150, the detection device 5, the position calculation device 6, and the imaging device 9 to the remote control device 7 so that they can communicate. The communication unit 71 of the remote control device 7 is, for example, a wireless communication device. The storage unit 73 of the remote control device 7 stores various programs that control the action of the remote control device 7. The storage unit 53 of the remote control device 7 includes, for example, a RAM, a ROM, and a hard disk drive (HDD).

[0047] The CPU 72 of the remote control device 7 develops various programs stored in the storage unit 73 of the remote control device 7 to function as an information acquisition unit 721 , a control value generation unit 722 , and a third transmission unit 723 .

[0048] The information acquisition unit 721 acquires various information. The information acquisition unit 721 acquires, for example, information related to the driving conditions of the vehicle 10 (hereinafter referred to as driving information). The driving information includes, for example, the vehicle coordinate point sent from the position calculation device 6 as information indicating the position of the vehicle 10, the driving speed and actual steering angle of the vehicle 10 sent from the vehicle control device 150, and the driving route information pre-stored in the storage unit 73 of the remote control device 7. The driving route information is information indicating the target driving route of the vehicle 10 driven by the remote automatic driving mode. In addition, the types of information included in the driving information are not limited to the above.

[0049] The control value generation unit 722 generates a control value that specifies the driving action of the vehicle 10 using the driving information obtained by the information acquisition unit 721. Specifically, the control value generation unit 722 generates, for example, a reference control value and a correction control value. The reference control value is a control value for causing the vehicle 10 to travel along the target driving route. The correction control value is a control value for correcting the relative position of the vehicle 10 relative to the target driving route. The reference control value and the correction control value respectively include, for example, an acceleration control value that specifies the acceleration in the forward direction of the vehicle 10 and a steering angle control value that specifies the steering angle of the vehicle 10. In addition, the reference control value and the correction control value may also be control values ​​that include either a track control value or a destination control value instead of the acceleration control value and the steering angle control value. The track control value is a control value that specifies the driving trajectory of the vehicle 10 by arranging the target driving positions of the vehicle 10 at each predetermined time in chronological order. The destination control value is a control value that indicates the target arrival time of the vehicle 10 at the target arrival place.

[0050] The third transmission unit 723 transmits various information to other devices other than the remote control device 7. For example, the third transmission unit 723 transmits the control value generated by the control value generation unit 722 to the vehicle 10 as the control target. In addition, at least a part of the function of the remote control device 7 can also be implemented as a function of any one of the vehicle control device 150, the detection device 5, the position calculation device 6, and the imaging device 9.

[0051] A-2. Vehicle detection method and position calculation method: Figure 6 This is a flowchart showing a method for detecting and calculating the position of the vehicle 10 in the first embodiment. Figure 6 The method shown is repeatedly executed at predetermined time intervals after the vehicle 10 as the detection target starts traveling in the remote automatic driving mode, for example.

[0052] Figure 7 It shows execution Figure 6 Schematic diagram of examples of various images when the detection method and position calculation method are shown. Figure 7 In the Figure 6 The step numbers corresponding to the steps of FIG. 1 and FIG. 2 are given in the embodiment. In the present embodiment, a case where the vehicle 10 travels (moves) along the road surface 20 on which the grid lines 21 are drawn along the Xg axis parallel to the traveling direction of the vehicle 10 and the Yg axis orthogonal to the Xg axis is taken as an example for explanation. The Xg axis and the Yg axis are coordinate axes in the global coordinate system. In addition, in other embodiments, the grid lines 21 may be omitted.

[0053] like Figure 6 As shown in FIG. 1 , first, an image acquisition process is performed (step S1). Figure 7 As shown, the image acquisition step is a step of acquiring an original image Im1 including the vehicle 10 . In the present embodiment, in the image acquisition step, the image acquisition unit 521 acquires the original image Im1 from the imaging device 9 .

[0054] like Figure 6 As shown in FIG. 1 , after the image acquisition process, a distortion correction process (step S2) is performed. The distortion correction process is a process for correcting the distortion of the captured image. Figure 7 As shown, in the distortion correction process, the distortion correction unit 522 generates a corrected image Im2 by correcting the distortion of the original image Im1. Specifically, the distortion correction unit 522 corrects the distortion of the original image Im1 using, for example, the distortion correction parameter Pa1 pre-stored in the storage unit 53 of the detection device 5. The distortion correction parameter Pa1 is, for example, a parameter related to the position information of the grid line 21 obtained by calibration. In addition, the distortion correction method is not limited to the above. The distortion correction parameter Pa1 may also be any parameter other than the above.

[0055] like Figure 6 As shown in FIG. 1 , after the distortion correction step, a rotation processing step (step S3) is performed. The rotation processing step is a step of rotating the captured image in such a way that the direction of the movement vector of the vehicle 10 included in the captured image is oriented toward a predetermined direction. Figure 7 As shown, in the rotation processing step, the rotation processing unit 523 rotates the corrected image Im2 in such a manner that the direction of the movement vector of the vehicle 10 contained in the corrected image Im2 is oriented toward a predetermined direction. Thus, the rotation processing unit 523 generates a rotated image Im3. Specifically, the rotation processing unit 523 rotates the corrected image Im2 with the center of gravity of the vehicle 10 in the corrected image Im2 as the rotation center, for example, in such a manner that the direction of the movement vector of the vehicle 10 is oriented toward an upward direction on the screen of the display device displaying the corrected image Im2. The movement of the feature point (for example, the center of gravity) of the vehicle 10 can be represented as the direction of the movement vector using, for example, the optical flow method. The size of the movement vector of the vehicle 10 and the direction of the movement vector are estimated, for example, based on the change in the position between image frames of the feature point appropriately set on the corrected image Im2. In addition, the rotation processing method is not limited to the above.

[0056] like Figure 6 As shown, after the rotation process, the cropping process is performed (step S4). The cropping process is a process of cutting out a necessary area from the captured image. Figure 7As shown, in the cropping process, when the vehicle 10 moves a distance exceeding a predetermined threshold, the cropping process unit 524 deletes the moved area A2 corresponding to the distance moved by the vehicle 10 as an unnecessary area from the rotated image Im3. As a result, the cropping process unit 524 generates a processed image Im4 in which the non-moved area A1 of the vehicle 10 is cropped out as a necessary area from the rotated image Im3. At this time, the cropping process unit 524 estimates the moved area A2 by, for example, recognizing the distance moved by the vehicle 10 based on the estimated movement vector size of the vehicle 10. In addition, in the detection method of the vehicle 10, either the rotation process or the cropping process may be performed first. In addition, the cropping process method is not limited to the above.

[0057] like Figure 6 As shown, following the cropping process, the type acquisition process (step S5) is executed. The type acquisition process is a process for acquiring type information indicating a type of vehicle body type of the vehicle 10 included in the captured image. In the type acquisition process, the type acquisition unit 525 acquires type information indicating a type of vehicle body type of the vehicle 10 included in the captured image. In addition, the type acquisition process can be executed at any time after the image acquisition process is completed and before the detection process (step S7) is started.

[0058] Following the type acquisition process, the model acquisition process is executed (step S6). The model acquisition process is a process of acquiring a first detection model Md1 selected from a plurality of first detection models Md1 prepared for each vehicle body type according to a vehicle body type determined by the type information obtained in the type acquisition process. In the present embodiment, in the model acquisition process, the model acquisition unit 526 selects and acquires a first detection model Md1 corresponding to a vehicle body type determined by the type information obtained in the type acquisition process from a plurality of first detection models Md1 pre-stored in the storage unit 53 of the detection device 5. In addition, the model acquisition process can be executed at any time after the end of the type acquisition process and before the start of the detection process.

[0059] After the model acquisition process, the detection process is performed. Figure 7As shown, the detection process is a process of detecting the shape of the vehicle 10 contained in the captured image using a detection model such as the first detection model Md1. In the present embodiment, in the detection process, the detection unit 527 inputs the processed image Im4 to the first detection model Md1 selected in the model acquisition process. Thus, the detection unit 527 detects the shape of the vehicle 10 contained in the processed image Im4. Then, the detection unit 527 generates a first mask image Im5 in which the mask area Ms is added to the object area by masking the object area in the processed image Im4. The first sending unit 528 sends the first mask image Im5 to the position calculation device 6. In addition, the detection method of the vehicle 10 is not limited to the above.

[0060] like Figure 6 As shown, after the detection process, the perspective transformation process is performed (step S8). Figure 7 As shown in FIG. 1 , the perspective transformation process is a process of performing perspective transformation on the first mask image Im5. In the perspective transformation process, the data acquisition unit 621 acquires the first mask image Im5 from the detection device 5. Then, the perspective transformation unit 622 generates the second mask image Im6 by performing perspective transformation on the first mask image Im5. Specifically, the perspective transformation unit 622 uses, for example, the perspective transformation parameter Pa2 pre-stored in the storage unit 63 of the position calculation device 6 to perspectively transform the first mask image Im5 into a bird's-eye view image observed from a viewpoint above the vehicle 10 (for example, directly above the vehicle 10) that is approximately perpendicular to the road surface 20. The perspective transformation parameter Pa2 is, for example, a parameter related to the position information and internal parameters of the imaging device 9 obtained by calibration. Thus, the perspective transformation unit 622 generates the second mask image Im6 represented by the image coordinate system from the first mask image Im5 represented by the camera coordinate system. The image coordinate system is a coordinate system having a point on the image plane projected by perspective transformation as the origin and having coordinate axes represented by the Xi axis and the Yi axis orthogonal to the Xi axis. In addition, the perspective transformation method is not limited to the above, and the perspective transformation parameter Pa2 may be any parameter other than the above.

[0061] like Figure 6 As shown, following the perspective transformation step, a coordinate point calculation step (step S9) is executed. Figure 8 2 is a diagram for explaining the details of the coordinate point calculation process. The coordinate point calculation process is a process of calculating an image coordinate point P3 indicating the position of the vehicle 10 in the image coordinate system.

[0062] In the coordinate point calculation step, the coordinate point calculation unit 623 calculates the basic coordinate point P0 based on the first circumscribed rectangle R1 set in the mask region Ms in the first mask image Im5 which is an image before perspective transformation. Fig. 9: is a diagram for explaining the calculation method of the basic coordinate point P0. In order to calculate the basic coordinate point P0, the coordinate point calculation unit 623 sets a basic circumscribed rectangle R0 for the mask area Ms in the first mask image Im5. Next, the coordinate point calculation unit 623 rotates the first mask image Im5 by a required amount of rotation, using the center of gravity C of the mask area Ms as the rotation center, so that the direction of the movement vector V of the vehicle 10 corresponding to the mask area Ms in the first mask image Im5 is oriented in a predetermined direction. The predetermined direction is, for example, the upward direction on the screen of the display device displaying the first mask image Im5. Next, the coordinate point calculation unit 623 sets a first circumscribed rectangle R1 for the mask area Ms of the rotated first mask image Im5 so that the long side is parallel to the direction of the movement vector V. Next, the coordinate point calculation unit 623 rotates the first mask image Im5 to which the first circumscribed rectangle R1 is attached by the above-mentioned amount of rotation, using the center of gravity C of the mask area Ms as the rotation center. Thus, the coordinate point calculation unit 623 sets the coordinate point of one of the four vertices of the first circumscribed rectangle R1, which has the coordinates closest to the positioning point 10e of the vehicle 10, as the basic coordinate point P0. Figure 8 As shown, the coordinate point calculation unit 623 performs perspective transformation on the first mask image Im5 after reverse rotation, that is, the first mask image Im5 after calculating the basic coordinate point P0. Thus, the coordinate point calculation unit 623 sets the coordinate point corresponding to the basic coordinate point P0 in the first circumscribed rectangle R1 deformed by the perspective transformation as the first coordinate point P1.

[0063] Furthermore, the coordinate point calculation unit 623 sets a second circumscribed rectangle R2 for the mask area Ms in the second mask image Im6 after perspective transformation of the first mask image Im5. Then, the coordinate point calculation unit 623 sets the vertex of the second circumscribed rectangle R2 that indicates the same position as the first coordinate point P1 as the second coordinate point P2. That is, the first coordinate point P1 and the second coordinate point P2 are coordinate points indicating the same position, and therefore have a correlation relationship with each other.

[0064] Furthermore, the coordinate point calculation unit 623 performs correction to replace the coordinates (Xi1, Yi1) of the first coordinate point P1 with the coordinates (Xi2, Yi2) of the second coordinate point P2 based on the magnitude relationship between the coordinate values ​​of the first coordinate point P1 and the coordinate values ​​of the second coordinate point P2. When the coordinate value Xi1 in the Xi direction of the first coordinate point P1 is greater than the coordinate value Xi2 in the Xi direction of the second coordinate point P2 (Xi1>Xi2), the coordinate point calculation unit 623 replaces the coordinate value Xi1 in the Xi direction of the first coordinate point P1 with the coordinate value Xi2 in the Xi direction of the second coordinate point P2. When the coordinate value Yi1 in the Yi direction of the first coordinate point P1 is greater than the coordinate value Yi2 in the Yi direction of the second coordinate point P2 (Yi1>Yi2), the coordinate point calculation unit 623 replaces the coordinate value Yi1 in the Yi direction of the first coordinate point P1 with the coordinate value Yi2 in the Yi direction of the second coordinate point P2. Figure 8 In the example shown, the coordinate value Xi1 of the first coordinate point P1 in the Xi direction is greater than the coordinate value Xi2 of the second coordinate point P2 in the Xi direction. In addition, the coordinate value Yi1 of the first coordinate point P1 in the Yi direction is less than the coordinate value Yi2 of the second coordinate point P2 in the Yi direction. Therefore, the image coordinate point P3 has the coordinates (Xi2, Yi1). In this way, the coordinate point calculation unit 623 calculates the image coordinate point P3 representing the position of the vehicle 10 in the image coordinate system by correcting the first coordinate point P1 using the second coordinate point P2. In addition, the calculation method of the image coordinate point P3 is not limited to the above.

[0065] like Figure 6 As shown, following the coordinate point calculation process, the position transformation process (step S10) is executed. The position transformation process is a process of calculating a vehicle coordinate point representing the position of the vehicle 10 in the global coordinate system by transforming the image coordinate point P3 into a vehicle coordinate point. The vehicle coordinate point calculated in this way is equivalent to the "position" contained in the vehicle position information described later. In the position transformation process, the position transformation unit 624 transforms the image coordinate point P3 into a vehicle coordinate point using the shooting parameters and vehicle parameters acquired by the data acquisition unit 621. The vehicle parameter is a parameter related to the distance between the positioning point 10e of the vehicle 10 and the reference point. In the present embodiment, the vehicle parameter is the height h (described later) of the positioning point 10e of the vehicle 10 from the road surface 20. Fig.10 ). In addition, the vehicle parameter may be a variable corresponding to the body type of the vehicle 10 or a predetermined constant.

[0066] The position conversion unit 624 uses the relational expressions (1) to (3) described later, which include the vehicle coordinate point as a target variable and include the image coordinate point P3, the shooting parameter, and the vehicle parameter as explanatory variables, to convert the image coordinate point P3 into the vehicle coordinate point. In this case, the position conversion unit 624 substitutes the coordinate value of the image coordinate point P3 calculated by the coordinate point calculation unit 623 into the relational expressions represented by the expressions (1) to (3). Furthermore, the position conversion unit 624 substitutes the shooting parameter obtained by the data acquisition unit 621, that is, the shooting parameter value corresponding to the camera 9 that obtained the original image Im1, into the relational expressions represented by the expressions (1) to (3).

[0067] Fig.10 This is the first figure for explaining the details of the position conversion process. Fig.10 , the diagram shows a state where the vehicle 10 is viewed from the left side. Fig.11 FIG. 2 is a second diagram for explaining the details of the position conversion process. Fig.11 , the diagram shows a state where the vehicle 10 is viewed from the ceiling side. Fig.10 and Fig.11 The global coordinate system shown is a coordinate system having a fixed coordinate point Pf representing an arbitrary reference position of the road surface 20 as an origin and having coordinate axes represented by an Xg axis and a Yg axis orthogonal to the Xg axis. The photographing coordinate point Pc is a coordinate point representing the position of the photographing device 9 that obtained the original image Im1 used in calculating the image coordinate point P3 in the global coordinate system. The fixed coordinate point Pf and the photographing coordinate point Pc are stored in advance in the storage unit 63 of the position calculation device 6.

[0068] like Fig.10 As shown, the observation distance on the XgYg plane between the position of the camera 9 and the position of the vehicle 10 (image coordinate point P3) is set to Do. The observation error is set to ΔD. The height [m] of the camera 9 from the road surface 20 as a shooting parameter is set to H. The height [m] of the positioning point 10e of the vehicle 10 from the road surface 20 as a vehicle parameter is set to h. In this case, the observation error ΔD can be expressed by the following formula (1). ΔD=h / H×Do Formula (1) That is, the larger the observation distance Do is, the larger the observation error ΔD is.

[0069] Next, when the actual distance (hereinafter referred to as the first distance) between the position of the imaging device 9 and the position of the positioning point 10e of the vehicle 10 is defined as D, the first distance D can be expressed by the following equation (2). D=Do×(1-h / H) Formula (2) That is, the first distance D is determined by the observation distance Do, the height H of the imaging device 9 as an imaging parameter, and the height h of the positioning point 10 e of the vehicle 10 as a vehicle parameter.

[0070] And, if Fig.11 As shown, when the estimated distance between the reference position and the position of the vehicle 10 is set to Dp and the actual distance between the reference position and the vehicle 10 (hereinafter, the second distance) is set to Dt, the second distance Dt can be expressed by the following formula (3). Dt=Dp×(1-h / H) Formula (3)

[0071] Here, the estimated distance Dp can be calculated using the actual distance (hereinafter, the third distance Dc) obtained from the fixed coordinate point Pf and the photographed coordinate point Pc, the image coordinate point P3, and the fixed coordinate point Pf. Therefore, the position conversion unit 624 can calculate the vehicle coordinate point Pv using the second distance Dt and the fixed coordinate point Pf obtained by correcting the estimated distance Dp using the above formula (3). At this time, the calculated vehicle coordinate point Pv is a coordinate point indicating the position of the vehicle 10 in the global coordinate system, so it is equivalent to the position of the vehicle 10 in the actual space. The second sending unit 625 sends the vehicle coordinate point Pv to the remote control device 7.

[0072] A-3. Driving control of vehicles based on remote control: Fig. 12A 1 is a flowchart showing a processing procedure of the travel control of the vehicle 10 in the first embodiment. Fig. 12A The example in which the above-mentioned detection device 5, position calculation device 6, and remote control device 7 function together as a "server" for executing remote control of the vehicle 100 is shown. In this embodiment, step S11 is executed by the CPU 52 of the detection device 5 and the CPU 62 of the position calculation device 6. Steps S12 to S14 are executed by the CPU 72 of the remote control device 7.

[0073] In step S11, the server obtains vehicle position information using the detection result output from the external sensor, which is a sensor located outside the vehicle 10. The vehicle position information is position information that serves as a basis for generating a driving control signal. In the present embodiment, the vehicle position information includes the position and orientation of the vehicle 10 in the reference coordinate system of the factory. In the present embodiment, the reference coordinate system of the factory is a global coordinate system, and any position in the factory can be represented by the X, Y, and Z coordinates in the global coordinate system. In the present embodiment, the external sensor is a camera installed in the factory, and a captured image is output from the external sensor as a detection result. That is, in step S11, the server obtains vehicle position information using the captured image obtained from the camera, which is an external sensor.

[0074] Specifically, in step S11, the server detects the shape of the vehicle 10 from the captured image, calculates the coordinates of the positioning point of the vehicle 10 in the coordinate system of the captured image, that is, the local coordinate system, and transforms the calculated coordinates into coordinates in the global coordinate system, thereby obtaining the position of the vehicle 10. The shape of the vehicle 10 contained in the captured image can be detected by, for example, inputting the captured image into a detection model using artificial intelligence. In addition, the server can obtain the orientation of the vehicle 10 by estimating the direction of the moving vector of the vehicle 10 calculated from the position change of the feature point of the vehicle 10 between frames of the captured image, for example, using the optical flow method.

[0075] In addition, specifically, in step S11 of the present embodiment, the execution Figure 6 The detection method and position calculation method shown.

[0076] In step S12, the server determines the target position to which the vehicle 10 should go next. In the present embodiment, the target position is represented by the coordinates of X, Y, and Z in the global coordinate system. In the memory of the server, a reference path is pre-stored as the path that the vehicle 10 should travel. The path is represented by a node showing a departure point, a node showing a passing point, a node showing a destination, and a link connecting each node. The server uses the vehicle position information and the reference path to determine the target position to which the vehicle 10 should go next. The server determines the target position on the reference path that is further ahead than the current position of the vehicle 10.

[0077] In step S13, the server generates a driving control signal for driving the vehicle 10 toward the determined target position. In the present embodiment, the driving control signal includes the acceleration and steering angle of the vehicle 10 as parameters. The server calculates the driving speed of the vehicle 10 according to the change of the position of the vehicle 10, and compares the calculated driving speed with the target speed. In general, the server determines the acceleration in a manner to accelerate the vehicle 10 when the driving speed is lower than the target speed, and determines the acceleration in a manner to decelerate the vehicle 10 when the driving speed is higher than the target speed. In addition, when the vehicle 10 is on the reference path, the server determines the steering angle and acceleration in a manner to prevent the vehicle 10 from deviating from the reference path, and when the vehicle 10 is not on the reference path, in other words, when the vehicle 10 deviates from the reference path, the server determines the steering angle and acceleration in a manner to return the vehicle 10 to the reference path. In other embodiments, the driving control signal may also include the speed of the vehicle 10 as a parameter instead of or on the basis of the acceleration of the vehicle 10.

[0078] In step S14, the server transmits the generated travel control signal to the vehicle 10. The server repeatedly acquires vehicle position information, determines a target position, generates a travel control signal, transmits the travel control signal, and the like in a predetermined cycle.

[0079] In step S15, the vehicle 10 receives the driving control signal transmitted from the server. In step S16, the vehicle 10 controls the actuator of the vehicle 10 using the received driving control signal, so that the vehicle 10 travels at the acceleration and steering angle indicated by the driving control signal. The vehicle 10 repeatedly receives the driving control signal and controls the actuator of the vehicle 10 at a predetermined cycle. According to the position calculation system 1 in this embodiment, the vehicle 10 can be driven by remote control, and the vehicle 10 can be moved without using a transport device such as a crane or a conveyor.

[0080] A-4. Vehicle motion control method: Fig. 12B 1 is a flowchart showing a method of controlling the operation of the vehicle 10 traveling in the remote automatic driving mode. Fig. 12B The operation control method shown is repeatedly executed every time the vehicle coordinate point Pv is received from the position calculation device 6 as information indicating the position of the vehicle 10 , for example.

[0081] The information acquisition unit 721 of the remote control device 7 acquires the driving information including the vehicle coordinate point Pv (step S101). Then, the control value generation unit 722 generates the control value that specifies the driving action of the vehicle 10 using the driving information (step S102). Then, the third sending unit 723 sends the control value to the vehicle 10 (step S103). The vehicle control device 150 mounted on the vehicle 10 drives the drive device 110 and the like according to the received control value (step S104). In addition, steps S101 and S102 are equivalent to Fig. 12A The processing executed between steps S11 to S13 of the embodiment. Step S103 is equivalent to the processing executed in step S14. Step S104 is equivalent to the processing executed in steps S15 and S16.

[0082] According to the first embodiment, when the vehicle 10 is driven automatically by remote control, the position calculation system 1 can calculate the position of the vehicle 10 using the captured image including the vehicle 10. Specifically, the detection device 5 inputs the captured image to the detection model, which is a machine learning model that has been learned and can detect the vehicle 10 included in the captured image. Thus, the detection device 5 can detect the vehicle 10 included in the captured image. Then, the position calculation device 6 can calculate the vehicle coordinate point Pv indicating the position of the vehicle 10 included in the captured image using the first mask image Im5 as the detection result of the detection device 5.

[0083] Here, the appearance of the vehicle 10 varies according to the vehicle body type, so the feature quantity extracted when the detection model is learned varies according to the vehicle body type. Therefore, when a detection model is applied to vehicles 10 classified into any vehicle body type, in which only a plurality of training images respectively including vehicles 10 classified into different vehicle body types are learned one by one in the same number for each vehicle body type, the detection accuracy of vehicles 10 of some vehicle body types may be reduced. In the case where the detection accuracy of vehicles 10 of some vehicle body types is reduced, in order to improve the detection accuracy of vehicles 10 of the vehicle body types with reduced detection accuracy, it may be considered to add training images including vehicles 10 of the vehicle body types with reduced detection accuracy and re-learn the detection model. However, by learning more training images including vehicles 10 of some vehicle body types than training images including vehicles 10 of other vehicle body types, the feature quantity included in the additional training images may be learned more emphatically than other feature quantities. As a result, a new difference in detection accuracy may be generated between vehicle body types. In the case where the detection accuracy of the vehicle 10 included in the captured image is reduced, the calculation accuracy of the vehicle coordinate point Pv indicating the position of the vehicle 10 is reduced. When the calculation accuracy of the vehicle coordinate point Pv decreases, there is a possibility that a desired control value cannot be generated.

[0084] Therefore, in the first embodiment, the first detection model Md1 suitable for detecting the vehicle 10 classified into one body type is prepared for each body type by learning more first training images including the vehicle 10 classified into one body type than second training images including the vehicle 10 classified into other body types. In addition, according to the first embodiment, the detection device 5 obtains type information indicating one body type of the vehicle 10 included in the captured image, and can obtain the first detection model Md1 selected according to the one body type determined by the obtained type information from the plurality of first detection models Md1 prepared for each body type. Then, the detection device 5 inputs the captured image to the first detection model Md1 corresponding to the body type of the vehicle 10 included in the captured image. Thus, the detection device 5 can detect the vehicle 10 included in the captured image with good accuracy. That is, by obtaining the first detection model Md1 suitable for detecting the vehicle 10 classified into one body type, each region constituting the captured image can be classified into a target region and an out-of-target region with good accuracy. According to the above, it is possible to suppress the detection accuracy of the vehicle 10 included in the captured image from being reduced according to the body type of the vehicle 10. As a result, it is possible to suppress a decrease in the calculation accuracy of the vehicle coordinate point Pv. Thus, it is possible to reduce the difference between the position of the vehicle 10 calculated by the position calculation device 6 and the actual position of the vehicle 10. Therefore, a more appropriate control value can be generated.

[0085] In addition, according to the above-mentioned first embodiment, the first teacher data set includes M (M is an integer greater than 2) first training images and N (N is an integer greater than 0 and less than M) second training images. In this case, the first teacher data set may not include the second training images but only include the first training images. In this case, for example, the more the number M of first training images including vehicles 10 classified as one body type is, the more the detection accuracy of the first detection model Md1 can be improved. In addition, the first teacher data set may also include the first training images and the number N of second training images which is less than the first training images. In this case, by including the second training images in the first teacher data set, it is possible to improve the detection accuracy of the part of the vehicle 10 having a common appearance shape regardless of the body type, for example.

[0086] In addition, according to the first embodiment, when detecting the vehicle 10 included in the captured image, the distortion of the original image Im1 can be corrected to generate the corrected image Im2. This can further improve the detection accuracy of the vehicle 10 included in the captured image. As a result, the calculation accuracy of the position of the vehicle 10 can be improved.

[0087] Furthermore, according to the first embodiment, when detecting the vehicle 10 included in the captured image, the rotated image Im3 can be generated by rotating the corrected image Im2 so that the direction of the movement vector V of the vehicle 10 faces a predetermined direction. In this way, the vehicle 10 included in the captured image can be detected in a state where the direction of the movement vector V is unified. Thus, the detection accuracy of the vehicle 10 included in the captured image can be further improved.

[0088] Furthermore, according to the first embodiment, when detecting the vehicle 10 included in the captured image, by performing a cropping process to crop a necessary area including the vehicle 10 from the rotated image Im3, a processed image Im4 excluding an unnecessary area can be generated. In this way, the vehicle 10 can be detected while excluding elements other than the vehicle 10 to be detected from the captured image. As a result, the detection accuracy of the vehicle 10 included in the captured image can be further improved.

[0089] In addition, according to the first embodiment, when detecting the vehicle 10 included in the captured image, a processed image Im4 can be generated by performing a cropping process of cropping the unmoved area A1 including the vehicle 10 from the rotated image Im3. In this way, the area occupied by the vehicle 10 in the captured image can be increased compared to the case where the cropping process is not performed. As a result, it is possible to easily detect the vehicle 10 located farther from the camera 9. Therefore, the detection accuracy for the vehicle 10 located farther from the camera 9 can be improved.

[0090] In addition, according to the first embodiment, by inputting the processed image Im4 to the first detection model Md1 and masking the object area in each area constituting the processed image Im4, it is possible to generate a first mask image Im5 in which the mask area Ms is added to the object area. At this time, according to the first embodiment, as the algorithm of the first detection model Md1, a DNN having a CNN structure that realizes semantic segmentation and instance segmentation can be used. Thus, it is possible to suppress the reduction in the detection accuracy of the vehicle 10 due to the diversity of the non-object area in the captured image.

[0091] Furthermore, according to the first embodiment, the second mask image Im6 can be generated by perspective-transforming the first mask image Im5 . This allows the camera coordinate system to be transformed into the image coordinate system.

[0092] In addition, according to the first embodiment, by setting the first circumscribed rectangle R1 for the mask area Ms before the perspective transformation of the first mask image Im5, the basic coordinate point P0, which is the vertex of the first circumscribed rectangle R1 and has the coordinates closest to the positioning point 10e of the vehicle 10, can be calculated. And by performing perspective transformation on the first mask image Im5 after the basic coordinate point P0 is calculated, the first coordinate point P1, which is the coordinate point corresponding to the basic coordinate point P0, can be calculated. Furthermore, by setting the second circumscribed rectangle R2 for the mask area Ms of the second mask image Im6, the second coordinate point P2, which is the vertex of the second circumscribed rectangle R2 and has the coordinates closest to the positioning point 10e of the vehicle 10, can be calculated. And by correcting the first coordinate point P1 using the second coordinate point P2, the image coordinate point P3 can be calculated. In this way, by comparing and correcting the coordinate points before and after the perspective transformation, the image coordinate point P3 can be calculated more accurately. As a result, the calculation accuracy of the position of the vehicle 10 can be further improved.

[0093] In addition, according to the first embodiment, by inputting a captured image to the determination model Md3 which is a learned machine learning model, type information indicating a type of vehicle body type of the vehicle 10 included in the captured image can be obtained. In other words, using machine learning, type information about the vehicle 10 included in the captured image can be obtained.

[0094] In addition, according to the first embodiment, it is possible to obtain the imaging parameters related to the imaging device 9 that has obtained the original image Im1. Furthermore, by substituting the calculated image coordinate point P3 and the acquired imaging parameter values ​​into a relational expression including the vehicle coordinate point Pv as a target variable and including the image coordinate point P3, the imaging parameter, and the vehicle parameter as explanatory variables, it is possible to transform the image coordinate point P3 into the vehicle coordinate point Pv. Thus, it is possible to transform the image coordinate system into the global coordinate system, and calculate the position of the positioning point 10e of the vehicle 10 in the global coordinate system as the vehicle coordinate point Pv.

[0095] In addition, according to the first embodiment, the shooting parameter is the height H of the shooting device 9 from the road surface 20 calculated based on the position of the shooting device 9 in the global coordinate system. In addition, the vehicle parameter is the height h of the positioning point 10e of the vehicle 10 from the road surface 20. In this way, the observation error ΔD can be calculated based on the similarity relationship between the shooting parameter and the vehicle parameter. And, using the calculated observation error ΔD, the image coordinate point P3 can be transformed into the vehicle coordinate point Pv.

[0096] Furthermore, according to the first embodiment, the position of the vehicle 10 can be calculated without installing any sensor, marker, transceiver or other installation objects for calculating the position of the vehicle 10 on the vehicle 10. Furthermore, the position of the vehicle 10 can be calculated without installing the position calculation device 6 on the vehicle 10. Thus, the versatility of the position calculation system 1 can be improved.

[0097] B. Second Implementation Method: Fig.13 1 is a diagram showing a schematic configuration of a detection device 5a in a second embodiment. In this embodiment, a detection model used when detecting a vehicle 10 included in a captured image is different from that in the first embodiment. As a result, a part of the configuration of the detection device 5a and a part of the processing of the detection method of the vehicle 10 are different from those in the first embodiment. The other configurations are the same as those in the first embodiment. The same steps as those in the first embodiment and the same configurations as those in the first embodiment are marked with the same reference numerals and the description thereof is omitted.

[0098] In the present embodiment, the storage unit 53a of the detection device 5a stores a second detection model Md2 as a detection model. Like the first detection model Md1, the second detection model Md2 is a machine learning model that has been learned to detect the vehicle 10 included in the captured image. The second detection model Md2 is a machine learning model that has been learned by inputting the second teacher data set. The second teacher data set includes a plurality of training images each including a vehicle 10 classified into different vehicle body types, a region correct answer label associated with each of a plurality of regions constituting each training image, and a type correct answer label associated with each of a plurality of training images. As in the first embodiment, the region correct answer label is a correct answer label indicating whether each region in the training image is an object region representing the vehicle 10 or an object-free region representing other than the vehicle 10. The type correct answer label is a correct answer label indicating a vehicle body type of the vehicle 10 included in the training image. In the present embodiment, the second detection model Md2 performs the following processing when a vehicle body type determined by the type information obtained by the type acquisition unit 525 and the captured image are input. In this case, the second detection model Md2 determines the object area included in the input captured image. Then, the second detection model Md2 generates a first mask image Im5 with the mask area Ms added to the object area by masking the object area. As an algorithm of the second detection model Md2, for example, CNN is used. In addition, the structure of the second detection model Md2 is not limited to the above. The second detection model Md2 can also be a machine learning model that has been learned using an algorithm other than a neural network.

[0099] Fig.14 1 is a diagram showing details of a CPU 52a mounted on a detection device 5a in a second embodiment. The CPU 52a of the detection device 5a in this embodiment includes a model acquisition unit 526a instead of the model acquisition unit 526 in the first embodiment. The model acquisition unit 526a acquires a second detection model Md2. In addition, the CPU 52a of the detection device 5a includes a detection unit 527a instead of the detection unit 527 in the first embodiment. The detection unit 527a detects the vehicle 10 included in the captured image by inputting a captured image and a vehicle body type determined by type information obtained by the type acquisition unit 525 to the second detection model Md2, thereby determining an object area among the areas constituting the input captured image.

[0100] Fig.15 It is a flowchart showing a detection method and a position calculation method of the vehicle 10 in the second embodiment. Fig.15 The method shown is repeatedly executed at predetermined time intervals after the vehicle 10 as the detection object starts driving in the remote automatic driving mode, for example. Figure 6The method shown is similar, Fig.15 The method shown is for example in Fig. 12A The process is performed in step S11.

[0101] In the present embodiment, in the model acquisition process (step S6a), the model acquisition unit 526a acquires the second detection model Md2. Then, in the detection process (step S7a), the detection unit 527a inputs the processed image Im4 and a vehicle body type determined by the type information acquired by the type acquisition unit 525 to the second detection model Md2. Thus, the detection unit 527a detects the outer shape of the vehicle 10 included in the processed image Im4. Then, the detection unit 527a masks the object area in the processed image Im4, thereby generating a first mask image Im5 with the mask area Ms added to the object area.

[0102] According to the above-mentioned second embodiment, a second detection model Md2 is prepared as a detection model that is learned by associating a plurality of training images including a vehicle 10 with a type correct answer label indicating the body type of the vehicle 10 included in each training image. Thus, by inputting a captured image associated with the body type to the second detection model Md2, the vehicle 10 included in the input captured image can be detected with good accuracy. That is, each area constituting the captured image input to the second detection model Md2 can be accurately classified into an object area and an out-of-object area. Based on the above, it is possible to suppress the detection accuracy of the vehicle 10 included in the captured image from being reduced according to the body type of the vehicle 10. As a result, it is possible to suppress the reduction in the calculation accuracy of the vehicle coordinate point Pv. Thus, it is possible to reduce the difference between the position of the vehicle 10 calculated by the position calculation device 6 and the actual position of the vehicle 10. Therefore, a more appropriate control value can be generated.

[0103] C. Third Implementation Method: Fig.16 It is a diagram showing a schematic configuration of a position calculation system 1 according to the third embodiment. Fig.17 1 is a diagram showing a schematic structure of a vehicle control device 150v in the third embodiment. In this embodiment, unlike the first embodiment, the detection device 5 and the position calculation device 6 which are separate from the vehicle control device 150v are not provided, and the vehicle control device 150v functions as a detection device and a position calculation device. That is, the vehicle control device 150v in the third embodiment is equivalent to the "detection device" in the present disclosure. The device structure of the vehicle in this embodiment is the same as that in the first embodiment, so for convenience, the vehicle in this embodiment is denoted as vehicle 10.

[0104] like Fig.17As shown in FIG. 1 , the vehicle control device 150v includes a CPU 151, a storage unit 152, an input / output interface 153, and an internal bus 154. The CPU 151, the storage unit 152, and the input / output interface 153 are connected via the internal bus 154 so as to be able to communicate bidirectionally. Fig.16 The drive device 110, the steering device 120, the brake device 130, the vehicle communication unit 140 and the vehicle-mounted sensor group 160 shown in the figure. The storage unit 152 includes, for example, a RAM, a ROM and a HDD. The storage unit 152 stores various information including a program PG1, a first detection model Md1, a determination model Md3, a distortion correction parameter Pa1, a perspective transformation parameter Pa2 and a camera database Db. The CPU 151, by developing the program PG1, functions as an image acquisition unit 521, a distortion correction unit 522, a rotation processing unit 523, a cropping processing unit 524, a type acquisition unit 525, a model acquisition unit 526, a detection unit 527, a first transmission unit 528, a data acquisition unit 621, a perspective transformation unit 622, a coordinate point calculation unit 623, a position transformation unit 624 and a second transmission unit 625. In addition, the CPU 151, by developing the program PG1, realizes a function of driving the vehicle 100 by autonomous control.

[0105] Fig.18 1 is a flowchart showing the processing order of the driving control of the vehicle 10 in the third embodiment. In step S901, the vehicle 10 obtains the vehicle position information using the detection result output from the camera as an external sensor. In step S902, the vehicle 10 determines the target position to which the vehicle 10 should go next. In step S903, the vehicle 10 generates a driving control signal for driving the vehicle 10 toward the determined target position. In step S904, the vehicle 10 controls the actuator of the vehicle 10 using the generated driving control signal, so that the vehicle 10 drives according to the parameters represented by the driving control signal. The vehicle 10 repeatedly obtains the vehicle position information, determines the target position, generates the driving control signal, and controls the actuator in a prescribed cycle. According to the driving control in this embodiment, even if the vehicle 10 is not remotely controlled by the server, the vehicle 10 can be driven by the autonomous control of the vehicle 10.

[0106] According to the third embodiment, it is also possible to suppress the decrease in the detection accuracy of the vehicle 10 included in the captured image depending on the body type of the vehicle 10. As a result, the vehicle 10 can generate a more appropriate control value for driving the vehicle 10 by autonomous control. In addition, in step S901 of the present embodiment, the operation is performed in substantially the same manner as in the first embodiment. Figure 6 In other embodiments, in step S901, the following steps may be performed in substantially the same manner as in the second embodiment: Fig.15 In this case, the storage unit 152 may also store the second detection model Md2. In addition, the CPU 151 may also function as the model acquisition unit 526a and the detection unit 527a by expanding the program PG1. In addition, in other embodiments, for example, the vehicle 10 may function as either the detection device or the position calculation device, and the other may be provided separately from the vehicle 10.

[0107] D. Other implementation methods: D-1. Other implementation methods 1: One body type may also represent the category of a single vehicle 10. The category of the vehicle 10 is, for example, a type of vehicle 10 classified by a vehicle name given to vehicles 10 of the same or common appearance shape. In this case, the storage unit 53 of the detection device 5 stores at least one of a plurality of first detection models Md1 prepared for each category of the vehicle 10 and a second detection model Md2 learned by associating one category of the vehicle 10 with a training image. In this case, the first training image is an image containing a vehicle 10 classified into one category. The second training image is an image containing a vehicle 10 classified into another category different from the one category. The type acquisition unit 525 of the detection device 5, 5a acquires category information indicating the category of the vehicle 10 included in the captured image. The model acquisition unit 526 acquires a first detection model Md1 selected from the plurality of first detection models Md1 prepared for each category according to one category determined by the category information acquired by the type acquisition unit 525. With such a method, the difference in the appearance shape of the vehicle 10 that differs according to the category of the vehicle 10 can be captured more accurately. This makes it possible to improve the detection accuracy of the vehicle 10 included in the captured image.

[0108] D-2. Other implementation methods 2: The type acquisition unit 525 may also use predetermined information pre-stored in the storage unit 53 of the detection device 5 to acquire type information indicating a type of vehicle body type of the vehicle 10 included in the captured image. The predetermined information is information indicating a predetermined time when a vehicle 10 classified as a type of vehicle body travels in the capturing range RG, which is associated with each capturing range RG of the plurality of capturing devices 9 that captured the original image Im1. In this manner, the type information indicating a type of vehicle body type of the vehicle 10 included in the captured image can be acquired using the predetermined information. Thus, the type information about the vehicle 10 included in the captured image can be acquired without performing image analysis. Therefore, the processing load of the type acquisition unit 525 in the type acquisition process can be reduced.

[0109] D-3. Other implementation methods 3: The training images included in the first teacher data set and the second teacher data set may also be images that have not been processed by distortion correction, rotation, cropping, etc., like the original image Im1. In this way, image processing is not required when preparing the first teacher data set and the second teacher data set. This can reduce the processing load when learning the first detection model Md1 and the second detection model Md2.

[0110] D-4. Other implementation methods 4: The training image may also be any one of the corrected image Im2, the rotated image Im3, and the processed image Im4. In this way, as in the first embodiment and the second embodiment, when a captured image processed from the original image Im1 is input to any one of the first detection model Md1 and the second detection model Md2, the detection accuracy of the vehicle 10 can be improved.

[0111] D-5. Other implementation methods 5: exist Figure 6 and Fig.15 In the detection method of the vehicle 10 shown, the distortion correction step is not an essential step. For example, when the distortion correction step is not performed and the steps after the rotation processing step are performed, the rotation processing unit 523 rotates the original image Im1 instead of the corrected image Im2 in the rotation processing step. Even in this manner, the image coordinate point P3 can be calculated.

[0112] D-6. Other implementation methods 6: exist Figure 6 and Fig.15 In the detection method of the vehicle 10 shown, the rotation process is not an essential process. For example, when the distortion correction process and the rotation process are not performed, the cropping process unit 524 performs cropping on the original image Im1 instead of the rotated image Im3 in the cropping process. Even in this way, the image coordinate point P3 can be calculated.

[0113] D-7. Other implementation methods 7: exist Figure 6 and Fig.15 In the detection method of the vehicle 10 shown, the cropping process is not an essential process. For example, when the distortion correction process, the rotation process, and the cropping process are not performed, the detection units 527 and 527a generate the first mask image Im5 in which the mask area Ms is added to the original image Im1 instead of the processed image Im4 in the detection process. Even in this manner, the image coordinate point P3 can be calculated.

[0114] D-8. Other implementation methods 8: The captured image may also include a plurality of vehicles 10. In this case, the CPU 52, 52a of the detection device 5, 5a may include, for example, a deletion unit that deletes the mask area Ms of the vehicle 10 that is not the object of position calculation from the first mask image Im5. The deletion unit, for example, determines that the mask area Ms that exists outside the identification target area among the mask areas Ms generated in the detection process is the mask area Ms of the vehicle 10 that is not the object of position calculation, and deletes it from the first mask image Im5. The identification target area is, for example, a predetermined area in which the vehicle 10 moves in the first mask image Im5. The predetermined area in which the vehicle 10 moves is, for example, an area corresponding to the area of ​​the grid line 21. The identification target area is pre-stored in the storage unit 53, 53a of the detection device 5, 5a. In this way, when a plurality of vehicles 10 are reflected in the captured image, the influence of the vehicle 10 that is not the object of position calculation can be eliminated. As a result, the calculation accuracy of the position of the vehicle 10 can be improved.

[0115] D-9. Other implementations 9: The captured image may also include a plurality of vehicles 10. In this case, a DNN that performs instance segmentation may be used as an algorithm for the first detection model Md1 and the second detection model Md2. In this way, the plurality of vehicles 10 included in the captured image can be classified, and a first mask image Im5 masked for each vehicle 10 can be generated. Thus, when a plurality of vehicles 10 are included in the captured image, the vehicle 10 to be calculated as the position can be selected, and the position of the selected vehicle 10 can be calculated.

[0116] D-10. Other implementations 10: The position calculation device 6 can also calculate the position of the stationary vehicle 10. When calculating the position of the stationary vehicle 10, the position calculation device 6 calculates the position of the vehicle 10 using, for example, the initial vector direction of the vehicle 10 estimated from the captured image first acquired after the position calculation device 6 is activated, instead of the direction of the movement vector V of the moving vehicle 10, and calculates the position of the vehicle 10. With this method, even when the vehicle 10 is stationary, the position of the vehicle 10 can be calculated using the captured image.

[0117] D-11. Other implementations 11: At least one of the detection device 5, 5a, the position calculation device 6, and the remote control device 7 may be configured as an integrated unit. In addition, in other embodiments, the detection device 5, the position calculation device 6, and the remote control device 7 may be implemented by, for example, cloud computing composed of one or more computers. In such a manner, the configuration of the detection device 5, 5a, the position calculation device 6, and the remote control device 7 may be appropriately changed.

[0118] D-12. Other implementations 12: The image acquisition unit 521 of the detection device 5 or 5a may acquire the original image Im1 via another external device (e.g., the remote control device 7) other than the imaging device 9 instead of directly acquiring the original image Im1 from the imaging device 9. Even in this manner, the image acquisition unit 521 of the detection device 5 or 5a can acquire the original image Im1 acquired by the imaging device 9.

[0119] D-13. Other implementations 13: In the above-mentioned embodiments, the vehicle 10 only needs to have a structure that can be moved by unmanned driving, for example, it can also be a platform having the following structure. Specifically, in order to perform the three functions of "driving", "steering" and "stopping" by unmanned driving, the vehicle 10 at least has an actuator such as a drive device, a steering device and a brake device and a control device. The actuator is controlled by the control device that controls the driving of the vehicle 10. In the case where the vehicle 10 obtains information from the outside for unmanned driving, the vehicle 10 also has a communication device. That is, the vehicle 10 that can be moved by unmanned driving may not be equipped with at least a part of the interior parts such as the driver's seat and the instrument panel, may not be equipped with at least a part of the exterior parts such as the bumper and the fender, and may not be equipped with the body shell. In this case, the remaining parts such as the body shell can be assembled on the vehicle 10 during the period until the vehicle 10 is shipped from the factory, or the remaining parts such as the body shell can be assembled on the vehicle 10 after the vehicle 10 is shipped from the factory in a state where the remaining parts such as the body shell are not assembled on the vehicle 10. Each component can be assembled from any direction such as the upper side, lower side, front side, rear side, right side or left side of the vehicle 10, and can be assembled from the same direction or from different directions. In addition, the position of the stand can also be determined in the same way as the vehicle 10 in the first embodiment.

[0120] D-14. Other implementations 14: The detection device 5 or 5a may detect the vehicle 10 using the first detection model Md1 stored in and selected by another device other than the detection device 5 or 5a. Even in this manner, the first detection model Md1 corresponding to the body type of the vehicle 10 can be obtained.

[0121] D-15. Other implementations 15: In the first embodiment, the server performs the processing from the acquisition of the vehicle position information to the generation of the travel control signal. In contrast, at least a part of the processing from the acquisition of the vehicle position information to the generation of the travel control signal may be performed by the vehicle 10. For example, the following methods (1) to (3) may be used.

[0122] (1) The server may obtain vehicle position information, determine the target position to which the vehicle 10 should go next, and generate a path from the current position of the vehicle 10 indicated by the obtained vehicle position information to the target position. The server may generate a path to the target position between the current position and the destination, or may generate a path to the destination. The server may send the generated path to the vehicle 10. The vehicle 10 may generate a travel control signal so that the vehicle 10 travels on the path received from the server, and control the actuator of the vehicle 10 using the generated travel control signal.

[0123] (2) The server may obtain the vehicle position information and transmit the obtained vehicle position information to the vehicle 10. The vehicle 10 may determine the target position to which the vehicle 10 should go next, generate a path from the current position of the vehicle 10 indicated by the received vehicle position information to the target position, generate a travel control signal so that the vehicle 10 travels on the generated path, and control the actuator of the vehicle 10 using the generated travel control signal.

[0124] (3) In the above-mentioned methods (1) and (2), the vehicle 10 may be equipped with an internal sensor, and the detection result outputted from the internal sensor may be used in at least one of the generation of the route and the generation of the driving control signal. The internal sensor is a sensor mounted on the vehicle 10. Specifically, the internal sensor may include, for example, a camera, LiDAR, a millimeter wave radar, an ultrasonic sensor, a GPS sensor, an acceleration sensor, a gyroscope sensor, etc. For example, in the above-mentioned method (1), the server may obtain the detection result of the internal sensor, and reflect the detection result of the internal sensor in the route when generating the route. In the above-mentioned method (1), the vehicle 10 may obtain the detection result of the internal sensor, and reflect the detection result of the internal sensor in the driving control signal when generating the driving control signal. In the above-mentioned method (2), the vehicle 10 may obtain the detection result of the internal sensor, and reflect the detection result of the internal sensor in the route when generating the route. In the above-mentioned method (2), the vehicle 10 may obtain the detection result of the internal sensor, and reflect the detection result of the internal sensor in the route when generating the driving control signal.

[0125] D-16. Other implementations 16: In the above embodiment in which the vehicle 10 can travel by autonomous control, the vehicle 10 may be equipped with an internal sensor, and the detection result outputted from the internal sensor may be used in at least one of the generation of the route and the generation of the travel control signal. For example, the vehicle 10 may obtain the detection result of the internal sensor, and when generating the route, the detection result of the internal sensor may be reflected in the route. Alternatively, the vehicle 10 may obtain the detection result of the internal sensor, and when generating the travel control signal, the detection result of the internal sensor may be reflected in the travel control signal.

[0126] D-17. Other implementations 17: In the above-mentioned embodiment in which the vehicle 10 can travel by autonomous control, it is also possible that the vehicle 10 is equipped with an internal sensor, the vehicle 10 uses the detection results of the internal sensor to obtain vehicle position information, determines the target position to which the vehicle 10 should go next, generates a path from the current position of the vehicle 10 represented by the obtained vehicle position information to the target position, generates a travel control signal for traveling on the generated path, and controls the actuator of the vehicle 10 using the generated travel control signal. In this case, the vehicle 10 can travel without using the detection results of the external sensor. In addition, the vehicle 10 can also obtain the target arrival time and / or congestion information from the outside of the vehicle 10, so that the target arrival time and / or congestion information is reflected in at least one of the path and the travel control signal.

[0127] D-18. Other implementations 18: In the first embodiment described above, the server automatically generates the travel control signal to be sent to the vehicle 10. In contrast, the server may generate the travel control signal to be sent to the vehicle 10 in accordance with the operation of an external operator located outside the vehicle 10. For example, the external operator may operate an operating device including a display for displaying a captured image output from an external sensor, a steering wheel for remotely operating the vehicle 10, an accelerator pedal, a brake pedal, and a communication device for communicating with the server via wired communication or wireless communication, and the server may generate the travel control signal in accordance with the operation applied to the operating device.

[0128] D-19. Other implementations 19: In the above-mentioned embodiments, the vehicle 10 only needs to have a structure that can be moved by unmanned driving, for example, it can also be a platform having the following structure. Specifically, in order to perform the three functions of "driving", "steering" and "stopping" by unmanned driving, the vehicle 10 at least has a control device that controls the driving of the vehicle 10, and actuators such as a drive device, a steering device and a brake device. In the case where the vehicle 10 obtains information from the outside for unmanned driving, the vehicle 10 can also have a communication device. That is, the vehicle 10 that can be moved by unmanned driving may not be equipped with at least a part of the interior parts such as the driver's seat and the instrument panel, may not be equipped with at least a part of the exterior parts such as the bumper and the fender, and may not be equipped with a body shell. In this case, the remaining parts such as the body shell can be assembled on the vehicle 10 during the period until the vehicle 10 is shipped from the factory, or the remaining parts such as the body shell can be assembled on the vehicle 10 after the vehicle 10 is shipped from the factory in a state where the remaining parts such as the body shell are not assembled on the vehicle 10. Each component can be assembled from any direction such as the upper side, lower side, front side, rear side, right side or left side of the vehicle 10, and can be assembled from the same direction or from different directions. In addition, the position of the stand can also be determined in the same way as the vehicle 10 in the first embodiment.

[0129] D-20. Other implementations 20: The vehicle 10 can be manufactured by combining a plurality of modules. A module means a unit composed of one or more parts that are aggregated according to the configuration and function of the vehicle 10. For example, the rack of the vehicle 10 can be manufactured by combining a front module that constitutes the front portion of the rack, a central module that constitutes the central portion of the rack, and a rear module that constitutes the rear portion of the rack. In addition, the number of modules that constitute the rack is not limited to three, and may be two or less or four or more. In addition, in addition to the rack, a portion of the vehicle 10 that is different from the rack may be modularized, or a portion of the vehicle 10 that is different from the rack may be modularized instead of the rack. In addition, various modules may also include any exterior parts such as a bumper and a grille, and any interior parts such as a seat and a console. In addition, not limited to the vehicle 10, any type of mobile body may be manufactured by combining a plurality of modules. Such a module may be manufactured, for example, by joining a plurality of parts by welding or a fixing, or may be manufactured by integrally molding at least a portion of the module into a single part by casting. The molding method of integrally molding at least a portion of the module into a single part is also called Giga-casting or Mega-casting. By using Giga-casting, various parts of the moving body that were previously formed by joining multiple parts can be formed into a single part. For example, the front module, the center module, and the rear module mentioned above can also be manufactured using Giga-casting.

[0130] D-21. Other implementations 21: Transporting the vehicle 10 by driving the vehicle 10 without human driver is also referred to as "self-propelled transportation". In addition, the configuration for realizing self-propelled transportation is also referred to as "vehicle remote-controlled autonomous driving transportation system". In addition, the production method of producing the vehicle 10 by self-propelled transportation is also referred to as "self-propelled production". In self-propelled production, for example, in a factory that manufactures the vehicle 10, at least a part of the transportation of the vehicle 10 is realized by self-propelled transportation.

[0131] In the above-mentioned embodiments, part or all of the functions and processes implemented in software may also be implemented in hardware. In addition, part or all of the functions and processes implemented in hardware may also be implemented in software. As hardware for implementing the various functions in the above-mentioned embodiments, various circuits such as integrated circuits or discrete circuits may also be used.

[0132] The present disclosure is not limited to the above-mentioned embodiments, and can be implemented in various configurations within the scope of the main purpose. For example, the technical features of the embodiments corresponding to the technical features in the various modes recorded in the "Contents of the Invention" column can be appropriately replaced or combined in order to solve part or all of the above-mentioned problems, or to achieve part or all of the above-mentioned effects. In addition, as long as the technical feature is not described as a necessary part in this specification, it can be appropriately deleted. Description of Reference Numerals

[0133] 1…position calculation system, 2…driving road, 5, 5a…detection device, 6…position calculation device, 7…remote control device, 9…photographing device, 10…vehicle, 10e…positioning point, 20…road surface, 21…grid line, 51…communication unit of the detection device, 52, 52a…CPU of the detection device, 53, 53a…storage unit of the detection device, 61…communication unit of the position calculation device, 62…CPU of the position calculation device, 63…storage unit of the position calculation device, 71…communication unit of the remote control device, 72…CPU of the remote control device, 73…storage unit of the remote control device, 110…driving device, 120…operation Steering device, 130 ... braking device, 140 ... vehicle communication unit, 150, 150v ... vehicle control device, 151 ... CPU of vehicle detection device, 152 ... storage unit of vehicle control device, 153 ... input-output interface, 154 ... internal bus, 160 ... vehicle sensor group, 521 ... image acquisition unit, 522 ... distortion correction unit, 523 ... rotation processing unit, 524 ... cropping processing unit, 525 ... type acquisition unit, 526 ... model acquisition unit, 527, 527a ... detection unit, 528 ... first sending unit, 621 ... data acquisition unit, 622 ... perspective transformation unit, 623 ... coordinate point calculation unit, 624 ... position Transformation unit, 625…second transmission unit, 721…information acquisition unit, 722…control value generation unit, 723…third transmission unit, 901…first imaging device, 902…second imaging device, A1…unmoved area, A2…moved area, C…center of gravity, D…first distance, ΔD…observation error, Db…camera database, Dc…third distance, Do…observation distance, Dp…estimated distance, Dt…second distance, h…height of the vehicle's positioning point from the road surface, H…height of the imaging device from the road surface, Im1…original image, Im2…corrected image, Im3…rotated image, Im4…processed image, Im5…first mask image Image, Im6…second mask image, Md1, Md1a~Md1c…first detection model, Md2…second detection model, Md3…determination model, Ms…mask area, Nt…network, P0…basic coordinate point, P1…first coordinate point, P2…second coordinate point, P3…image coordinate point, Pa1…distortion correction parameter, Pa2…perspective transformation parameter, Pc…shooting coordinate point, Pf…fixed coordinate point, Pv…vehicle coordinate point, R0…basic circumscribed rectangle, R1…first circumscribed rectangle, R2…second circumscribed rectangle, RG…shooting range, RG1…first shooting range, RG2…second shooting range, V…moving vector.

Claims

1. A detection device for detecting a moving object contained in a captured image, comprising: An image acquisition unit, which acquires the captured image; a type acquisition unit that acquires type information indicating a body type of the mobile object included in the captured image, which is classified according to the appearance shape of the mobile object; a model acquisition unit that acquires the first detection model selected from a plurality of first detection models that are machine learning models prepared for each of the vehicle body types according to the one vehicle body type determined by the type information acquired by the type acquisition unit; as well as The detection unit detects the moving object included in the captured image by inputting the captured image to the first detection model acquired by the model acquisition unit to identify a target region representing the moving object in the captured image.

2. The detection device according to claim 1, in, The multiple first detection models are pre-learned in a manner of determining the object area by inputting multiple training images, respectively, and the multiple training images include M first training images each containing the mobile body classified as one of the vehicle body types, and N second training images each containing the mobile body classified as other vehicle body types different from the one vehicle body type, M is an integer greater than 2, and N is an integer greater than 0 and less than M.

3. A detection device for detecting a moving object contained in a captured image, comprising: An image acquisition unit, which acquires the captured image; a type acquisition unit that acquires type information indicating a vehicle body type of the mobile object included in the captured image, which is classified according to an external shape of the mobile object; and The detection unit determines the object area representing the moving object in the captured image by inputting the captured image and the one vehicle body type determined by the type information obtained by the type acquisition unit into a second detection model as a machine learning model, thereby detecting the moving object contained in the captured image.

4. The detection device according to claim 3, in, The second detection model is pre-learned in a manner that determines the object area by inputting multiple training images containing the moving body and a type correct label, wherein the type correct label is associated with each of the multiple training images and represents the body type of the moving body contained in the training image.

5. The detection device according to claim 2 or 4, in, For each region in the training image, a region correct answer label indicating whether it is the target region or a non-target region indicating other than the moving object is associated with the region.

6. The detection device according to claim 1 or 3, in, The vehicle body type indicates a category of the moving object.

7. The detection device according to claim 1 or 3, in, The moving body moves by receiving a moving condition that specifies a moving action of the moving body from outside the moving body, The type acquisition unit acquires the type information by determining the one vehicle body type of the mobile body contained in the captured image using predetermined information, wherein the predetermined information is associated with each shooting range of a plurality of shooting devices that acquire the captured image and indicates a predetermined time at which the mobile body classified as the one vehicle body type moves within the shooting range.

8. The detection device according to claim 1 or 3, in, The category acquisition unit acquires the category information by inputting the captured image to a determination model, wherein the determination model is a machine learning model that has been learned in such a manner as to output the category information when the captured image is input.

9. The detection device according to claim 1 or 3, further comprising: The rotation processing unit rotates the captured image so that the moving direction of the moving object faces a predetermined direction.

10. The detection device according to claim 1 or 3, further comprising: The distortion correction unit corrects the distortion of the captured image.

11. A position calculation system for calculating the position of a moving object contained in a captured image, comprising: The detection device according to claim 1 or 3; and a position calculating device for calculating the position of the moving body, The detection device further generates a first mask image in which the mask region is added to the target region by masking the target region, which is a region representing the moving object in the captured image. The position calculation device comprises: a perspective transformation unit, configured to generate a second mask image by performing perspective transformation on the first mask image; a coordinate point calculation unit that calculates an image coordinate point indicating the position of the moving object in an image coordinate system by setting a designated vertex of a first circumscribed rectangle set in the mask area in the first mask image as a first coordinate point, setting a vertex in a second circumscribed rectangle set in the mask area in the second mask image that indicates the same position as the first coordinate point as a second coordinate point, and correcting the first coordinate point using the second coordinate point; and The position transformation unit transforms the image coordinate point into a moving body coordinate point representing the position of the moving body in the global coordinate system, using the distance from the shooting device to a reference point calculated based on the position of the shooting device in the global coordinate system and the distance from a predetermined positioning point of the moving body to the reference point.

12. A detection method for detecting a moving object contained in a captured image, comprising: An image acquisition step of acquiring the captured image; a type acquisition step of acquiring type information indicating a body type of the mobile object included in the captured image, which is classified according to the appearance shape of the mobile object; a model acquisition step of acquiring the first detection model selected from a plurality of first detection models as machine learning models prepared for each of the vehicle body types according to the one vehicle body type determined by the type information acquired in the type acquisition step; and The detection step detects the moving object included in the captured image by inputting the captured image to the first detection model acquired in the model acquisition step to determine a target region representing the moving object in the captured image.

13. A detection method for detecting a moving object contained in a captured image, comprising: An image acquisition step of acquiring the captured image; a type acquisition step of acquiring type information indicating a vehicle body type of the mobile object included in the captured image, which is classified according to the appearance shape of the mobile object; and A detection process, in which the object area representing the moving object in the captured image is determined by inputting the captured image and the one vehicle body type determined by the type information obtained in the type acquisition process into a second detection model as a machine learning model, thereby detecting the moving object contained in the captured image.

Images