Detecting apparatus, position calculation system, and detecting method

Abstract

To provide a technique for accurately detecting a vehicle included in a captured image.SOLUTION: In a detecting apparatus configured to detect a vehicle in a captured image, the vehicle is configured to move in a factory in which manufacturing steps are performed to manufacture and ship vehicles. The vehicle is classified into multiple states in accordance with appearances of the vehicle, the appearances being different depending on the manufacturing steps. The detecting apparatus includes: an image acquisition unit configured to acquire a captured image; a state acquisition unit configured to acquire state information indicating one of the states of the vehicle included in the captured image; a model acquisition unit configured to acquire, from among first detection models that are machine learning models prepared for each of the states, a first detection model selected in accordance with the one state identified by the state information acquired by the state acquisition unit; and a detection unit configured to detect the vehicle included in the captured image by inputting the captured image to the first detection model acquired by the model acquisition unit and identifying a target region representing the vehicle in the captured image.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present disclosure relates to a detection device, a position calculation system, and a detection method. [Background technology]

[0002] 2. Description of the Related Art Conventionally, vehicles that run autonomously under remote control are known (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2017-538619 Summary of the Invention [Problem to be solved by the invention]

[0004] In order to make a vehicle run automatically by remote control, an image of the vehicle may be taken from outside. However, when the vehicle runs in a factory where the vehicle is manufactured, the appearance of the vehicle may differ depending on the manufacturing process. For example, the appearance of a platform-type vehicle in which various devices such as wheels and a vehicle control device are assembled to a chassis is significantly different from that of a vehicle in which a body such as a body shell is assembled to the chassis. The inventors of the present application have found that such a difference in the appearance of the vehicle depending on the manufacturing process may reduce the detection accuracy of the vehicle included in the captured image. [Means for solving the problem]

[0005] The present disclosure can be realized in the following forms.

[0006] (1) According to a first embodiment of the present disclosure, a detection device is provided. In the detection device for detecting a vehicle included in a captured image, the vehicle travels in a factory where a plurality of manufacturing processes are performed to manufacture and ship the vehicle, and the vehicle is classified into a plurality of states by having different appearances depending on the plurality of manufacturing processes. The detection device includes an image acquisition unit for acquiring the captured image, a state acquisition unit for acquiring state information indicating one of the states of the vehicle included in the captured image, a model acquisition unit for acquiring the first detection model selected from a plurality of first detection models that are machine learning models prepared for each of the states according to the one state identified by the state information acquired by the state acquisition unit, and a detection unit for inputting the captured image into the first detection model acquired by the model acquisition unit and identifying a target area indicating the vehicle in the captured image, thereby detecting the vehicle included in the captured image. According to this embodiment, the detection device can acquire state information indicating the state of the vehicle included in the captured image, and acquire the first detection model selected from a plurality of first detection models prepared for each state according to the one state identified by the acquired state information. Then, by inputting the captured image into a first detection model corresponding to the state of the vehicle included in the captured image, the vehicle included in the captured image can be detected with high accuracy. That is, by using a first detection model suitable for detecting a vehicle classified into one state, each area constituting the captured image can be classified with high accuracy into a target area and an area other than the target area. As a result, it is possible to prevent the detection accuracy of the vehicle included in the captured image from decreasing according to the difference in appearance in multiple manufacturing processes. (2) According to a second aspect of the present disclosure, there is provided a detection device. In a detection device that detects a vehicle included in an captured image, the vehicle runs in a factory where a plurality of manufacturing processes are carried out in order to manufacture and ship the vehicle, and the vehicle is classified into a plurality of states because the vehicle has different appearances depending on the plurality of manufacturing processes, and the plurality of states include a platform state in which the vehicle is in the form of a platform having at least wheels, a chassis, a drive device that accelerates the vehicle, a steering device that changes the direction of travel of the vehicle, a braking device that decelerates the vehicle, a vehicle control device that controls the operation of the vehicle, and a vehicle communication unit that communicates with other devices other than the host vehicle, and the detection device includes an image acquisition unit that acquires the captured image, a state acquisition unit that acquires state information indicating one of the states of the vehicle included in the captured image, a model acquisition unit that acquires the first detection model selected from a plurality of first detection models which are machine learning models prepared for each of the states in accordance with the one state identified by the state information acquired by the state acquisition unit, and a detection unit that detects the vehicle included in the captured image by inputting the captured image to the first detection model acquired by the model acquisition unit and identifying a target area indicating the vehicle in the captured image. According to this aspect, when the vehicle form is a platform form, the state of the vehicle can be classified into a platform state. Then, when detecting a vehicle classified into the platform state, the vehicle can be detected by using the first detection model suitable for detecting a vehicle classified into the platform state. Also, when detecting a vehicle classified into the platform state, the vehicle can be detected by inputting a captured image associated with the platform state into the second detection model. This can improve the detection accuracy of vehicles classified into the platform state. (3) According to a third embodiment of the present disclosure, a detection device is provided. In the detection device for detecting a vehicle included in a captured image, the vehicle travels in a factory where a plurality of manufacturing processes are performed to manufacture and ship the vehicle, and the vehicle is classified into a plurality of states by having different appearances depending on the plurality of manufacturing processes, and the detection device includes an image acquisition unit for acquiring the captured image, a state acquisition unit for acquiring state information indicating one of the states of the vehicle included in the captured image, and a detection unit for inputting the captured image and the one state identified by the state information acquired by the state acquisition unit into a second detection model that is a machine learning model, and identifying a target area indicating the vehicle in the captured image, thereby detecting the vehicle included in the captured image. According to this embodiment, the captured image associated with the state of the vehicle can be input into the second detection model, thereby accurately detecting the vehicle included in the captured image. In other words, each area constituting the captured image can be accurately classified into a target area and an area other than the target area. As a result, it is possible to suppress a decrease in detection accuracy of the vehicle included in the captured image depending on the difference in appearance in a plurality of manufacturing processes. (4) In the above embodiment, each of the plurality of first detection models may be trained in advance to identify the target region by inputting a plurality of training images including M (M is an integer of 2 or more) first training images each including the vehicle classified into the one state, and N (N is an integer of 0 or more and less than M) second training images each including the vehicle classified into another state different from the one state. According to this embodiment, by training more first training images each including vehicles classified into one state than second training images each including vehicles classified into another state different from the one state, a first detection model suitable for detecting vehicles classified into one state can be prepared for each state. (5) In the above embodiment, each of the plurality of first detection models may be trained in advance to identify the target region by inputting a plurality of training images including M (M is an integer of 2 or more) first training images each including the vehicle classified into the one state, and N (N is an integer of 0 or more and less than M) second training images each including the vehicle classified into another state different from the one state. According to this embodiment, by training more first training images each including vehicles classified into one state than second training images each including vehicles classified into another state different from the one state, a first detection model suitable for detecting vehicles classified into one state can be prepared for each state. (6) In the above embodiment, the second detection model may be trained in advance to identify the target region by inputting a plurality of training images including the vehicle and a state answer label associated with each of the plurality of training images, the state answer label indicating the state of the vehicle included in the training image. According to this embodiment, it is possible to prepare a second detection model trained in association with a plurality of training images including a vehicle and a state answer label indicating the state of the vehicle included in each training image. (7) In the above embodiment, a region answer label indicating whether each region in the training image is the target region or a non-target region indicating an area other than the vehicle may be associated with each region in the training image. According to this embodiment, each region constituting the captured image can be accurately classified into a target region and a non-target region. This can improve the detection accuracy of vehicles included in the captured image. (8) In the above embodiment, the status acquisition unit may acquire process information related to one of the manufacturing processes being performed on the vehicle, and acquire the status information by using a process database in which the status information is associated with each of the multiple manufacturing processes to identify the status information associated with the one of the manufacturing processes identified by the acquired process information. According to this embodiment, it is possible to acquire process information related to one of the manufacturing processes being performed on the vehicle. Then, it is possible to acquire status information indicating the status of the vehicle included in the captured image by identifying the status information associated with the one of the manufacturing processes identified by the acquired process information using the process database. (9) In the above aspect, the status acquisition unit may acquire the status information by inputting the captured image to a specific model that is a machine learning model trained to output the status information when the captured image is input. According to this aspect, status information indicating a status of a vehicle included in the captured image can be acquired by inputting the captured image to a specific model trained to output status information about a vehicle included in the captured image. (10) In the above embodiment, the status acquisition unit may acquire imaging information related to an imaging device, and acquire the status information by using a status database in which the imaging information and the status information are associated to identify the status information associated with one of the imaging devices identified by the acquired imaging information. According to this embodiment, it is possible to acquire imaging information related to an imaging device. Then, it is possible to acquire status information indicating the status of the vehicle included in the captured image by identifying the status information associated with one of the imaging devices identified by the acquired imaging information using a status database in which the imaging information and the status information are associated. (11) In the above embodiment, the manufacturing process may include a painting process for painting the vehicle, and the states may include (iii) an unpainted state indicating the state of the vehicle before painting in the painting process, and (iv) a painted state indicating the state of the vehicle after painting in the painting process. According to this embodiment, the state of the vehicle can be classified into an unpainted state and a painted state before and after painting in the painting process. This makes it possible to suppress a decrease in detection accuracy of the vehicle included in the captured image according to the difference in the appearance color of the vehicle that differs before and after the painting process is performed. (12) In the above aspect, the imaging device may further include a distortion correction unit that corrects distortion of the captured image. According to this aspect, it is possible to correct distortion of the captured image, thereby improving detection accuracy of a vehicle included in the captured image. (13) The above aspect may further include a rotation processing unit that rotates the captured image so that the moving direction of the vehicle faces a predetermined direction. According to this aspect, the captured image can be rotated so that the direction of a vector indicating the moving direction of the vehicle faces a predetermined direction. In this way, it is possible to detect a vehicle included in the captured image with the direction of the vector indicating the moving direction of the vehicle being unified. This can improve the detection accuracy of the vehicle included in the captured image. (14) According to a fourth aspect of the present disclosure, there is provided a position calculation system. The position calculation system for calculating a position of a vehicle included in a captured image includes the detection device according to the above aspects, and a position calculation device for calculating a position of the vehicle, the detection device further includes a position calculation device for generating a first mask image by masking a target area, which is an area indicating the vehicle, in the captured image and adding a mask area to the target area, and calculating a position of the vehicle using the first mask image, the position calculation device including a perspective transformation unit for generating a second mask image by performing perspective transformation on the first mask image, and a coordinate point calculation unit for calculating a position of the vehicle by calculating a position of the vehicle based on a specified circumscribing rectangle of a first circumscribing rectangle set in the mask area in the first mask image. The present invention includes a coordinate point calculation unit that calculates an image coordinate point indicating the position of the vehicle in an image coordinate system by setting a vertex of the second circumscribing rectangle set in the mask area in the second mask image as a first coordinate point, setting a vertex of the second circumscribing rectangle that indicates the same position as the first coordinate point as a second coordinate point, and correcting the first coordinate point using the second coordinate point, and a position conversion unit that converts the image coordinate point into a vehicle coordinate point indicating the position of the vehicle in the global coordinate system using a distance from a reference point of the imaging device calculated based on the position of the imaging device in a global coordinate system and a distance from the reference point of a predetermined positioning point of the vehicle. According to this embodiment, the position of the vehicle included in the captured image can be calculated. Also, a first mask image in which a target area indicating the vehicle is masked out of each area constituting the captured image, and a second mask image in which the first mask image is perspectively converted can be generated. In this way, the image coordinate point can be calculated by calculating the first coordinate point from the first mask image and the second coordinate point from the second mask image. This makes it possible to calculate the image coordinate point more accurately. Therefore, the calculation accuracy of the vehicle position can be improved. (15) According to a fifth aspect of the present disclosure, a detection method is provided. In the detection method for detecting a vehicle included in a captured image, the vehicle travels in a factory where a plurality of manufacturing processes are performed to manufacture and ship the vehicle, and the vehicle is classified into a plurality of states by having different appearances depending on the plurality of manufacturing processes, and the detection method includes an image acquisition step of acquiring the captured image, a state acquisition step of acquiring state information indicating one of the states of the vehicle included in the captured image, a model acquisition step of acquiring the first detection model selected from a plurality of first detection models that are machine learning models prepared for each of the states according to the one state identified by the state information acquired in the state acquisition step, and a detection step of inputting the captured image to the first detection model acquired in the model acquisition step and identifying a target area indicating the vehicle in the captured image, thereby detecting the vehicle included in the captured image. According to this aspect, it is possible to acquire state information indicating the state of the vehicle included in the captured image, and acquire the first detection model selected from a plurality of first detection models prepared for each state according to the one state identified by the acquired state information. Then, by inputting the captured image into a first detection model corresponding to the state of the vehicle included in the captured image, the vehicle included in the captured image can be detected with high accuracy. That is, by using a first detection model suitable for detecting a vehicle classified into one state, each area constituting the captured image can be classified with high accuracy into a target area and an area other than the target area. As a result, it is possible to prevent the detection accuracy of the vehicle included in the captured image from decreasing according to the difference in appearance in multiple manufacturing processes. (16) According to a sixth embodiment of the present disclosure, a detection method is provided. In the detection method for detecting a vehicle included in a captured image, the vehicle travels in a factory where a plurality of manufacturing processes are performed to manufacture and ship the vehicle, and the vehicle has different appearances depending on the plurality of manufacturing processes and is classified into a plurality of states, and the detection method includes an image acquisition step of acquiring the captured image, a state acquisition step of acquiring state information indicating one of the states of the vehicle included in the captured image, and a detection step of inputting the captured image and the one state identified by the state information acquired in the state acquisition step into a second detection model that is a machine learning model, and identifying a target area indicating the vehicle in the captured image, thereby detecting the vehicle included in the captured image. According to this embodiment, the captured image associated with the state of the vehicle is input into the second detection model, so that the vehicle included in the captured image can be detected with high accuracy. In other words, each area constituting the captured image can be classified with high accuracy into a target area and an area other than the target area. As a result, it is possible to suppress a decrease in detection accuracy of the vehicle included in the captured image depending on the difference in appearance in a plurality of manufacturing processes. The present disclosure can be realized in various forms other than the above-mentioned detection device, position calculation system, and detection method, for example, in the form of a manufacturing method of the detection device and the position calculation system, a position calculation method, a control method of the detection device and the position calculation system, a computer program for realizing the control method, a non-transitory recording medium on which the computer program is recorded, etc. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram showing the configuration of a position calculation system. [Diagram 2] 2A to 2C are diagrams showing a manufacturing process of the vehicle in the first embodiment. [Diagram 3] FIG. 2 is a diagram showing the configuration of a detection device according to the first embodiment. [Figure 4] FIG. 2 is a diagram showing details of a CPU mounted on the detection device in the first embodiment. [Diagram 5] FIG. 1 is a diagram showing the configuration of a position calculation device. [Figure 6] FIG. 2 is a diagram showing the configuration of a remote control device. [Figure 7] 4 is a flowchart showing a vehicle detection method and a position calculation method in the first embodiment. [Figure 8] Schematic diagrams showing examples of various images. [Figure 9] FIG. 11 is a diagram for explaining details of a coordinate point calculation step. [Figure 10] FIG. 4 is a diagram for explaining a method of calculating a base coordinate point. [Figure 11] FIG. 1 is a diagram for explaining details of a position conversion process. [Figure 12] FIG. 2 for explaining the details of the position conversion process. [Figure 13] 4 is a flowchart showing a vehicle operation control method. [Figure 14] 8A to 8C are diagrams showing a manufacturing process of a vehicle according to a second embodiment. [Figure 15] FIG. 11 is a diagram showing the configuration of a detection device according to a second embodiment. [Figure 16] FIG. 13 is a diagram showing the configuration of a detection device according to a third embodiment. [Figure 17] FIG. 11 is a diagram showing details of a CPU mounted on a detection device according to a third embodiment. [Figure 18] 13 is a flowchart showing a vehicle detection method and a position calculation method in a third embodiment. [Figure 19] FIG. 13 is a diagram showing the configuration of a detection device according to a fourth embodiment. [Figure 20] FIG. 13 is a diagram showing details of a CPU mounted on a detection device according to a fourth embodiment. [Figure 21] FIG. 13 is a diagram showing the configuration of a detection device according to a fifth embodiment. [Figure 22] FIG. 13 is a diagram showing details of a CPU mounted on a detection device according to a fifth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] A. First embodiment: A-1. Configuration of position calculation system: FIG. 1 is a diagram showing the configuration of a position calculation system 1. The position calculation system 1 is a system that detects a vehicle 10 included in a captured image and calculates the position of the vehicle 10. In the present disclosure, the captured image is at least one of an original image as raw data acquired by an imaging device 9 as an external device provided at a location different from the vehicle 10, a corrected image obtained by processing the original image, a rotated image, and a processed image. The corrected image, the rotated image, and the processed image will be described in detail later. The position calculation system 1 includes one or more vehicles 10, a detection device 5 that detects the vehicle 10 included in the captured image, and a position calculation device 6 that calculates the position of the vehicle 10 using the detection result by the detection device 5.

[0009] The imaging device 9 captures an image of the imaging range RG including the vehicle 10 whose position is to be calculated from outside the vehicle 10, thereby acquiring an original image. In this embodiment, the imaging device 9 transmits the acquired original image to the detection device 5 together with camera identification information and the acquisition time of the original image. The camera identification information is a unique ID (identifier) ​​that is assigned to each of the imaging devices 9 so as not to overlap with each other in order to identify a plurality of imaging devices 9. The original image is a two-dimensional image in which pixels arranged on the XcYc plane of the camera coordinate system are collected. The camera coordinate system is a coordinate system having the focal point of the imaging device 9 as the origin and coordinate axes indicated by the Xc axis and the Yc axis perpendicular to the Xc axis. The original image includes at least two-dimensional data of the vehicle 10 whose position is to be calculated. The original image is preferably a color image, but may be a gray image. The imaging device 9 is a camera having an imaging element such as a CCD image sensor and a CMOS image sensor, and an optical system.

[0010] In this embodiment, the imaging device 9 acquires an original image of the lane 2 and the vehicle 10 traveling on the lane 2, viewed from above. In order to capture the entire lane 2 by one or more imaging devices 9, the installation positions and the number of imaging devices 9 are determined in consideration of the imaging range RG (angle of view) of the imaging device 9. Specifically, each imaging device 9 is installed so that the first imaging range RG1, which is the imaging range RG of the adjacent first imaging device 901, and the second imaging range RG2, which is the imaging range RG of the second imaging device 902, overlap. Furthermore, each imaging device 9 is installed at a position where it can capture an image of a positioning point 10e that is set in advance for a specific part of the vehicle 10 traveling on the lane 2. In this embodiment, the positioning point 10e is the rear end of the left side of the vehicle 10 (hereinafter, the left rear end). Note that the positioning point 10e may be other than the left rear end of the vehicle 10. Further, the imaging device 9 is not limited to acquiring information from above the vehicle 10, but may acquire information from the front, rear, sides, etc. of the vehicle 10.

[0011] The vehicle 10 may be, for example, an electric vehicle, a hybrid vehicle, a fuel cell vehicle, a gasoline vehicle, or a diesel vehicle. The vehicle 10 may be a private vehicle such as a passenger car, or may be a commercial vehicle such as a truck, a bus, or a construction vehicle.

[0012] The vehicle 10 has a manned driving mode and a remote unmanned driving mode. In the manned driving mode, the driver in the vehicle 10 operates input devices such as a steering wheel and an accelerator provided on the vehicle 10 to generate driving conditions for the vehicle 10. As a result, the vehicle 10 runs according to the generated driving conditions. The driving conditions are conditions that define the driving operation of the vehicle 10. The driving conditions include, for example, the driving route, position, driving speed, acceleration, and steering angle of the wheels 170 of the vehicle 10. The remote unmanned driving mode has a remote manual driving mode and a remote automatic driving mode. In the remote manual driving mode, the operator operates an operator input device provided at a location different from the vehicle 10 to generate driving conditions for the vehicle 10. As a result, the vehicle 10 receives the driving conditions generated by the operator input device and runs according to the received driving conditions. In the remote automatic driving mode, a remote control device 7 such as a server provided at a location different from the vehicle 10 creates control values ​​that define the driving operation of the vehicle 10 and transmits them to the vehicle 10. As a result, the vehicle 10 receives the control value and performs automatic driving according to the received control value.

[0013] In this embodiment, the vehicle 10 travels in a factory in a remote automatic driving mode where a plurality of manufacturing processes are performed to manufacture and ship the vehicle 10. The factory is not limited to being located in one building, or at one site or address, but may be located across multiple buildings, sites, addresses, etc. The factory may also include a storage location such as a yard where the vehicle 10 manufactured by executing the plurality of manufacturing processes is stored as a finished product before being shipped. At this time, the vehicle 10 may travel on a public road, not limited to a private road.

[0014] Fig. 2 is a diagram showing the manufacturing process of the vehicle 10 in the first embodiment. Fig. 2 shows a representative portion of a plurality of manufacturing steps executed in the manufacturing process of the vehicle 10. In the example shown in Fig. 2, the vehicle 10 as a finished product is manufactured by assembling a painted body shell 180 to a vehicle 101 in the form of a platform (hereinafter, platform vehicle 101), and then assembling interior parts such as seats and exterior parts 190 such as door panels.

[0015] The platform manufacturing process is a manufacturing process for manufacturing the platform vehicle 101. The platform vehicle 101 is a vehicle 10 that can realize at least three functions, "run", "turn" and "stop", by remote control using a remote control device 7. Specifically, the platform vehicle 101 includes at least a driving device 110, a steering device 120, a braking device 130, a vehicle communication unit 140, a vehicle control device 150, a chassis 160, and wheels 170, for example. The driving device 110 accelerates the vehicle 10. The driving device 110 includes at least a driving force source and a power supply device that supplies power to the driving force source. When the vehicle 10 is an electric vehicle, the driving force source is a motor, and the power supply device is a battery such as a lithium-ion battery. The steering device 120 changes the traveling direction of the vehicle 10. The braking device 130 decelerates the vehicle 10. The vehicle communication unit 140 communicates with external devices using wireless communication or the like. The external devices are devices other than the vehicle 10, such as the detection device 5, the position calculation device 6, the remote control device 7, and the imaging device 9, and other vehicles 10. The vehicle communication unit 140 is, for example, a wireless communication device. The vehicle communication unit 140 communicates with external devices connected to the network Nt, for example, via an access point in a factory. The vehicle control device 150 includes a CPU, a storage unit, and an input / output interface. In the vehicle control device 150, the CPU, the storage unit, and the input / output interface are connected to each other, for example, via an internal bus or an interface circuit. The input / output interface communicates with internal devices such as the drive unit 110 mounted on the vehicle 10. The input / output interface is connected to the vehicle communication unit 140 so as to be able to communicate with each other. The chassis 160 is a chassis portion that supports the various devices 110 to 150 mounted on the vehicle 10, the body shell 180 that is assembled in a later manufacturing process, and various parts 190. The configuration of the vehicle 10, which is in the form of a platform, is not limited to the above.

[0016] The body shell assembly process is a manufacturing process in which the body shell 180 is assembled to the platform vehicle 101. The platform vehicle 101 and a vehicle 102 in which the body shell 180 is assembled to the platform vehicle 101 (hereinafter, referred to as the first assembled vehicle 102) have different external shapes, and therefore the external appearance of the vehicle 10 is different.

[0017] The part assembly process is a process of assembling an exterior part 190. The first assembly vehicle 102 and a vehicle 103 (hereinafter, second assembly vehicle 103) in which the exterior part 190 is assembled to the first assembly vehicle 102 have different exterior shapes, and therefore the appearance of the vehicle 10 is different. Note that at least a part of the interior parts may be assembled to the platform vehicle 101 before the body shell assembly process is performed, or may be assembled to the platform vehicle 101 either before or after the part assembly process is performed.

[0018] As described above, the vehicle 10 is classified into a plurality of states by having different appearances due to a plurality of manufacturing processes. At this time, the difference in appearance between the platform vehicle 101 and the first assembly vehicle 102 is smaller than, for example, the difference in appearance between the first assembly vehicle 102 and the second assembly vehicle 103. Therefore, in this embodiment, a state in which the vehicle 10 is in the form of a platform, such as the platform vehicle 101, is called a "platform state". A state in which at least the body shell 180 is assembled to the platform vehicle 101, such as the first assembly vehicle 102 and the second assembly vehicle 103, is called an "assembly state". That is, in this embodiment, the vehicle 10 is classified into a plurality of states according to the difference in the exterior shape of the vehicle 10. At this time, the state of the vehicle 10 in one manufacturing process may be classified as one state, or the states of the vehicle 10 in multiple manufacturing processes may be collectively classified as one state, depending on the magnitude of the difference in the exterior shape of the vehicle 10. It should be noted that the number and types of manufacturing processes in the manufacturing process of the vehicle 10, and the number and types of states of the vehicle 10 classified according to the multiple manufacturing processes are not limited to those described above.

[0019] In the present disclosure, the vehicle 10 is at least one of the vehicle 10 as a finished product and the vehicles 101-105 as semi-finished products and work-in-progress in each manufacturing process of manufacturing the finished product. In the following, when it is not necessary to distinguish between the vehicles 101-105 in a plurality of states, they will be simply referred to as "vehicle 10."

[0020] 3 is a diagram showing a configuration of the detection device 5 in the first embodiment. The detection device 5 detects at least the outer shape (contour) of the vehicle 10 included in the captured image by identifying a target area, which is an area indicating the vehicle 10, among the areas in the captured image. The detection device 5 includes a communication unit 51, a storage unit 53, and a CPU 52. In the detection device 5, the communication unit 51, the storage unit 53, and the CPU 52 are connected to each other via, for example, an internal bus and an interface circuit.

[0021] The communication unit 51 of the detection device 5 communicably connects the detection device 5 to other devices such as the vehicle control device 150, the position calculation device 6, the remote control device 7, and the imaging device 9. The communication unit 51 of the detection device 5 is, for example, a wireless communication device.

[0022] The storage unit 53 of the detection device 5 stores various information including various programs for controlling the operation of the detection device 5, a first detection model Md1 as a detection model, a process database Db1, and a distortion correction parameter Pa1. The storage unit 53 of the detection device 5 includes, for example, a RAM, a ROM, and a hard disk drive (HDD).

[0023] The detection model is a trained machine learning model used to detect the vehicle 10 included in the captured image. In this embodiment, the storage unit 53 of the detection device 5 stores, as the detection model, a plurality of first detection models Md1 prepared for each state.

[0024] The first detection model Md1 is a machine learning model trained by inputting a first teacher data set. The first teacher data set is prepared for each state. The first teacher data set has a plurality of training images including the vehicle 10, and a region correct answer label associated with each of a plurality of regions constituting each training image. The region correct answer label is a correct answer label indicating whether each region in the training image is a target region indicating the vehicle 10 or a non-target region indicating other than the vehicle 10. In this embodiment, each region constituting the training image is one pixel constituting the training image. Note that each region constituting the training image may be a plurality of pixels constituting the training image. Each region constituting the training image is classified into either a target region or a non-target region according to a calculation result when, for example, the RGB values ​​of adjacent regions constituting the training image are compared. Each region constituting the training image may be classified into either a target region or a non-target region according to the calculated probability by calculating the probability that it is a target region. When a region correct answer label is attached to each region constituting the training image, information indicating the position in the training image of the region to which the region correct answer label is attached is associated. When each region constituting the training image is one pixel, the information indicating the position in the training image is, for example, pixel coordinates indicating the pixel position in the training image.

[0025] The first teacher data set includes, as a plurality of training images, M (M is an integer of 2 or more) first training images and N (N is an integer of 0 or more and less than M) second training images. The first training images are images including a vehicle 10 classified into one state. The second training images are images including a vehicle 10 classified into another state different from the one state. The number M of first training images included in the first teacher data set may be, for example, 1.5 times or more the number N of second training images, or may be 2 times or more. In addition, when there are a plurality of other states different from the one state, the number of first training images included in the first teacher data set is, for example, greater than the total number of second training images including vehicles 10 classified into a plurality of other states different from the one state. In other words, the first detection model Md1 is a machine learning model that preferentially learns the feature amount of the vehicle 10 classified into one state by learning more first training images than second training images.

[0026] In this embodiment, the storage unit 53 of the detection device 5 stores two first detection models Md1a and Md1b suitable for detecting one state determined by the external shape of the vehicle 10. The first first detection model Md1a is a first detection model Md1 that has been trained on more vehicles 10 classified as "platform state" than vehicles 10 classified as states other than the "platform state" (for example, "assembled state"). The first first detection model Md1a is used when detecting a vehicle 10 classified as "platform state". The second first detection model Md1b is a first detection model Md1 that has been trained on more vehicles 10 classified as "assembled state" than vehicles 10 classified as states other than the "assembled state" (for example, "platform state"). The second first detection model Md1b is used when detecting a vehicle 10 classified as "assembled state". The number and type of the first detection models Md1 are not limited to those described above.

[0027] In this embodiment, when a captured image is input, the first detection model Md1 identifies a target area among the areas constituting the input captured image. Then, the first detection model Md1 masks the target area to generate a first mask image in which a mask area is added to the target area.

[0028] The algorithm of the first detection model Md1 is, for example, a deep neural network (hereinafter, DNN) having a structure of a convolutional neural network (hereinafter, CNN) that realizes semantic segmentation and instance segmentation. Note that the configuration of the first detection model Md1 is not limited to the above. The first detection model Md1 may be, for example, a trained machine learning model that uses an algorithm other than a neural network.

[0029] The process database Db1 is a database in which status information is associated with each of a plurality of manufacturing processes. The status information is information that indicates one status of the vehicle 10. In other words, the status information is information that identifies a plurality of statuses in the manufacturing process of the vehicle 10. Note that the configuration of the process database Db1 is not limited to the above.

[0030] The distortion correction parameter Pa1 is a parameter used when correcting the distortion of a captured image. The distortion correction parameter Pa1 will be described in detail later.

[0031] 4 is a diagram showing details of the CPU 52 mounted on the detection device 5 in the first embodiment. The CPU 52 of the detection device 5 deploys various programs stored in the storage unit 53 of the detection device 5, thereby functioning as an image acquisition unit 521, a distortion correction unit 522, a rotation processing unit 523, a crop processing unit 524, a state acquisition unit 525, and a model acquisition unit 526. Furthermore, the CPU 52 of the detection device 5 deploys various programs stored in the storage unit 53 of the detection device 5, thereby functioning as a detection unit 527 and a first transmission unit 528.

[0032] The image acquisition unit 521 acquires an original image from the imaging device 9 .

[0033] The distortion correction unit 522 corrects distortion in the captured image. In this embodiment, the distortion correction unit 522 generates a corrected image by correcting the distortion of the original image.

[0034] The rotation processing unit 523 rotates the captured image so that the direction of a vector indicating the moving direction of the vehicle 10 (hereinafter, the moving vector) faces a predetermined direction. In this embodiment, the rotation processing unit 523 generates a rotated image by rotating the corrected image.

[0035] The crop processing unit 524 deletes from the captured image, among the regions constituting the captured image, regions other than the region consisting of the vehicle 10 and the region surrounding the vehicle 10 (hereinafter, referred to as the necessary region) (hereinafter, referred to as the unnecessary region). In this way, the crop processing unit 524 cuts out the necessary region from the captured image. In this embodiment, when the vehicle 10 has moved a distance exceeding a predetermined threshold, the crop processing unit 524 deletes from the rotated image the moved region that corresponds to the distance moved by the vehicle 10 and is an unnecessary region. In this way, the crop processing unit 524 generates a processed image in which the unmoved region including the vehicle 10 and is cut out as a necessary region from the rotated image.

[0036] The state acquisition unit 525 acquires process information related to one manufacturing process being performed on the vehicle 10 included in the captured image, and state information indicating one state of the vehicle 10 included in the captured image. The process information is, for example, a unique ID (identifier) ​​assigned to each manufacturing process in order to identify multiple manufacturing processes included in the manufacturing process of the vehicle 10. The state acquisition unit 525, for example, uses information acquired by a sensor (hereinafter, an on-board sensor) mounted on the vehicle 10 to detect feature points that can identify the multiple manufacturing processes, thereby identifying one manufacturing process being performed on the vehicle 10 included in the captured image. The on-board sensor is, for example, one of an on-board camera, an on-board lidar, and an on-board radar. The on-board camera captures an image of the state of the surrounding area of ​​the vehicle 10. The on-board lidar and the on-board radar detect objects present in the surrounding area of ​​the vehicle 10. The state acquisition unit 525 then acquires state information about the vehicle 10 included in the captured image by identifying state information associated with one manufacturing process identified by the acquired process information using the process database Db1. The method of acquiring the process information and the state information is not limited to the above.

[0037] The model acquisition unit 526 acquires the first detection model Md1 selected according to one state specified by the state information acquired by the state acquisition unit 525. In this embodiment, the model acquisition unit 526 selects and acquires the first detection model Md1 according to one state specified by the state information acquired by the state acquisition unit 525 from among a plurality of first detection models Md1 prepared for each state stored in the storage unit 53 of the detection device 5. In other words, the model acquisition unit 526 selects the first detection model Md1 in which the feature amount of the vehicle 10 classified into one state specified by the state information acquired by the state acquisition unit 525 is learned more than the feature amount of the vehicle 10 classified into another state different from the one state. For example, when the state information acquired by the state acquisition unit 525 identifies the state of the vehicle 10 included in the captured image as the "platform state", the model acquisition unit 526 acquires the first detection model Md1a in which the vehicle 10 classified into the "platform state" is learned more than the vehicle 10 classified into another state.

[0038] The detection unit 527 detects the vehicle 10 included in the captured image by inputting the captured image to the first detection model Md1 acquired by the model acquisition unit 526 and identifying a target region among the regions constituting the captured image. In this embodiment, the detection unit 527 inputs the processed image to the first detection model Md1 selected by the model acquisition unit 526. As a result, a first mask image is generated from the processed image by identifying a target region among the regions constituting the processed image.

[0039] The first transmission unit 528 transmits various information to devices other than the detection device 5. The first transmission unit 528 transmits, for example, a detection result by the detection unit 527 to the position calculation device 6. In this embodiment, the detection result by the detection unit 527 is a first mask image. Note that at least a part of the functions of the detection device 5 may be realized as one function of any of the vehicle control device 150, the position calculation device 6, the remote control device 7, and the imaging device 9.

[0040] FIG. 5 is a diagram showing a configuration of the position calculation device 6. The position calculation device 6 calculates the position of the vehicle 10 included in the captured image using the detection result by the detection device 5. The position calculation device 6 adopts the positioning point 10e of the vehicle 10 as the position of the vehicle 10. In this embodiment, the position calculation device 6 calculates the position of the vehicle 10 included in the captured image using the first mask image. The position calculation device 6 includes a communication unit 61, a storage unit 63, and a CPU 62. In the position calculation device 6, the communication unit 61, the storage unit 63, and the CPU 62 are connected to each other via, for example, an internal bus and an interface circuit.

[0041] The communication unit 61 of the position calculation device 6 communicably connects the vehicle control device 150, the detection device 5, the remote control device 7, and the imaging device 9 to the position calculation device 6. The communication unit 61 of the position calculation device 6 is, for example, a wireless communication device.

[0042] The storage unit 63 of the position calculation device 6 stores various information including various programs for controlling the operation of the position calculation device 6, the perspective transformation parameters Pa2, and a camera database Db2. The storage unit 63 of the position calculation device 6 includes, for example, a RAM, a ROM, and a hard disk drive (HDD).

[0043] The perspective transformation parameter Pa2 is a parameter used when performing perspective transformation on the first mask image. The details of the perspective transformation parameter Pa2 will be described later.

[0044] The camera database Db2 is a database indicating, for each imaging device 9, imaging parameters calculated based on the installation position of the imaging device 9 in the global coordinate system. The imaging parameters are parameters related to the distance of the imaging device 9 from a predetermined reference point. In this embodiment, the imaging parameter is the height H of the imaging device 9 from the road surface 20 (FIG. 11 described later), which is the height H with respect to the road surface 20 on which the vehicle 10 is located. The camera database Db2 is, for example, a database that associates camera identification information with imaging parameters for the imaging device 9 identified by the camera identification information. Note that the configuration of the camera database Db2 is not limited to the above. The imaging range RG, installation position, installation angle, etc. of the imaging device 9 may be associated with the camera identification information.

[0045] The CPU 62 of the position calculation device 6 functions as a data acquisition unit 621, a perspective transformation unit 622, a coordinate point calculation unit 623, a position transformation unit 624, and a second transmission unit 625 by deploying various programs stored in the memory unit 63 of the position calculation device 6.

[0046] The data acquisition unit 621 acquires various information. In this embodiment, the data acquisition unit 621 acquires a first mask image from the detection device 5. Furthermore, the data acquisition unit 621 refers to the camera database Db2 stored in the storage unit 63 of the position calculation device 6 to acquire imaging parameters for the imaging device 9 that is the source of the original image to be analyzed.

[0047] The perspective transformation unit 622 generates a second mask image by performing perspective transformation on the first mask image. The coordinate point calculation unit 623 corrects the first coordinate point using the second coordinate point to calculate an image coordinate point. The first coordinate point is a coordinate point in the image coordinate system of a specified vertex of a first circumscribing rectangle set in the mask area in the first mask image. The second coordinate point is a coordinate point in the image coordinate system indicating the same position as the first coordinate point among the vertices of the second circumscribing rectangle set in the mask area in the second mask image. The image coordinate point is a coordinate point indicating the position of the vehicle 10 in the image coordinate system. The position transformation unit 624 transforms the image coordinate point into a vehicle coordinate point using the imaging parameters acquired by the data acquisition unit 621 and the image coordinate point calculated by the coordinate point calculation unit 623. The vehicle coordinate point is a coordinate point indicating the position of the vehicle 10 in the global coordinate system.

[0048] The second transmission unit 625 transmits various information to devices other than the position calculation device 6. For example, the second transmission unit 625 transmits a vehicle coordinate point to the remote control device 7 as information indicating the position of the vehicle 10. Note that at least a part of the function of the position calculation device 6 may be realized as one function of any of the vehicle control device 150, the detection device 5, the remote control device 7, and the imaging device 9.

[0049] FIG. 6 is a diagram showing the configuration of the remote control device 7. The remote control device 7 uses the position information of the vehicle 10 and the like to create a control value that specifies the traveling operation of the vehicle 10 and transmits it to the vehicle 10. In this way, the remote control device 7 remotely controls the operation of the vehicle 10. The remote control device 7 includes a communication unit 71, a storage unit 73, and a CPU 72. The communication unit 71, the storage unit 73, and the CPU 72 of the remote control device 7 are connected to each other via, for example, an internal bus and an interface circuit. The communication unit 71 of the remote control device 7 connects the vehicle control device 150, the detection device 5, the position calculation device 6, and the imaging device 9 to the remote control device 7 so that they can communicate with each other. The communication unit 71 of the remote control device 7 is, for example, a wireless communication device. The storage unit 73 of the remote control device 7 stores various programs that control the operation of the remote control device 7. The storage unit 73 of the remote control device 7 includes, for example, a RAM, a ROM, and a hard disk drive (HDD).

[0050] The CPU 72 of the remote control device 7 deploys various programs stored in the storage unit 73 of the remote control device 7, thereby functioning as an information acquisition unit 721, a control value creation unit 722, and a third transmission unit 723.

[0051] The information acquisition unit 721 acquires various information. For example, the information acquisition unit 721 acquires information related to the driving conditions of the vehicle 10 (hereinafter, driving information). The driving information includes, for example, vehicle coordinate points transmitted from the remote control device 7 as information indicating the position of the vehicle 10, the driving speed and actual steering angle of the vehicle 10 transmitted from the vehicle control device 150, and driving route information stored in advance in the storage unit 73 of the remote control device 7. The driving route information is information indicating a target driving route of the vehicle 10 traveling in the remote automatic driving mode. Note that the types of information included in the driving information are not limited to those described above.

[0052] The control value creation unit 722 creates a control value that specifies the driving operation of the vehicle 10 by using the driving information acquired by the information acquisition unit 721. Specifically, the control value creation unit 722 creates, for example, a reference control value and a correction control value. The reference control value is a control value for driving the vehicle 10 along a target driving route. The correction control value is a control value that corrects the relative position of the vehicle 10 with respect to the target driving route. The reference control value and the correction control value each include, for example, an acceleration control value that specifies the acceleration in the forward direction of the vehicle 10 and a steering angle control value that specifies the steering angle of the vehicle 10. Note that, instead of the acceleration control value and the steering angle control value, the reference control value and the correction control value each may be a control value that includes either a trajectory control value or a destination control value. The trajectory control value is a control value that specifies the driving trajectory of the vehicle 10 by arranging the target driving positions of the vehicle 10 for each predetermined time in a chronological order. The destination control value is a control value that indicates the target arrival time of the vehicle 10 to the target arrival location.

[0053] The third transmission unit 723 transmits various information to devices other than the remote control device 7. For example, the third transmission unit 723 transmits the control value created by the control value creation unit 722 to the vehicle 10 to be controlled. Note that at least a part of the functions of the remote control device 7 may be realized as one function of any of the vehicle control device 150, the detection device 5, the position calculation device 6, and the imaging device 9.

[0054] A-2. Vehicle detection and position calculation methods: Fig. 7 is a flowchart showing a method for detecting a vehicle 10 and a method for calculating a position of the vehicle 10 in the first embodiment. The method shown in Fig. 7 is repeatedly executed at predetermined time intervals after the vehicle 10 to be detected starts traveling in the remote autonomous driving mode, for example.

[0055] FIG. 8 is a schematic diagram showing examples of various images when the detection method and the position calculation method shown in FIG. 7 are executed. In FIG. 8, step numbers corresponding to the steps in FIG. 7 are assigned. In this embodiment, a case will be described in which the vehicle 10 runs (moves) along the traveling direction on a road surface 20 on which mesh-like lattice lines 21 are drawn along the Xg axis parallel to the traveling direction of the vehicle 10 and the Yg axis perpendicular to the Xg axis. The Xg axis and the Yg axis are each a coordinate axis in a global coordinate system. In other embodiments, the lattice lines 21 may be omitted.

[0056] As shown in Fig. 7, first, an image acquiring step (step S1) is executed. As shown in Fig. 8, the image acquiring step is a step of acquiring an original image Im1 including the vehicle 10. In this embodiment, in the image acquiring step, the image acquiring unit 521 acquires the original image Im1 from the imaging device 9.

[0057] As shown in FIG. 7, following the image acquisition step, a distortion correction step (step S2) is executed. The distortion correction step is a step of correcting distortion of a captured image. In this embodiment, as shown in FIG. 8, in the distortion correction step, the distortion correction unit 522 corrects the distortion of the original image Im1 to generate a corrected image Im2. Specifically, the distortion correction unit 522 corrects the distortion of the original image Im1, for example, by using a distortion correction parameter Pa1 stored in advance in the storage unit 53 of the detection device 5. The distortion correction parameter Pa1 is, for example, a parameter related to position information of the grid lines 21 obtained by calibration. Note that the distortion correction method is not limited to the above. The distortion correction parameter Pa1 may be any parameter other than those described above.

[0058] As shown in FIG. 7, following the distortion correction step, a rotation processing step (step S3) is executed. The rotation processing step is a step of rotating the captured image so that the direction of the movement vector of the vehicle 10 included in the captured image faces a predetermined direction. In this embodiment, as shown in FIG. 8, in the rotation processing step, the rotation processing unit 523 rotates the corrected image Im2 so that the direction of the movement vector for the vehicle 10 included in the corrected image Im2 faces a predetermined direction. In this way, the rotation processing unit 523 generates a rotated image Im3. Specifically, the rotation processing unit 523 rotates the corrected image Im2 with the center of gravity of the vehicle 10 in the corrected image Im2 as the center of rotation so that the direction of the movement vector of the vehicle 10 faces an upward direction on the screen of a display device that displays the corrected image Im2. The movement of a feature point (for example, the center of gravity) of the vehicle 10 can be expressed as the direction of the movement vector by, for example, an optical flow method. The amount and direction of the movement vector of the vehicle 10 are estimated based on, for example, a change in the position of a feature point between image frames that is appropriately set on the corrected image Im2. Note that the rotation processing method is not limited to the above.

[0059] As shown in FIG. 7, after the rotation process, a crop process (step S4) is executed. The crop process is a process of cutting out a necessary area from a captured image. In this embodiment, as shown in FIG. 8, in the crop process, when the vehicle 10 moves a distance exceeding a predetermined threshold, the crop processing unit 524 deletes a moved area A2 corresponding to the distance moved by the vehicle 10 from the rotated image Im3 as an unnecessary area. As a result, the crop processing unit 524 generates a processed image Im4 in which the unmoved area A1 including the vehicle 10 is cut out as a necessary area from the rotated image Im3. At this time, the crop processing unit 524 estimates the moved area A2 by recognizing the distance moved by the vehicle 10 from the estimated movement vector amount of the vehicle 10, for example. Note that, in the detection method for the vehicle 10, either the rotation process or the crop process may be executed first. Also, the crop process method is not limited to the above.

[0060] As shown in FIG. 7, after the cropping process step, a status acquisition step (step S5) is executed. The status acquisition step is a step of acquiring status information indicating one status of the vehicle 10 included in the captured image. In this embodiment, in the status acquisition step, the status acquisition unit 525 first acquires process information related to one manufacturing process being performed on the vehicle 10 included in the captured image. Then, the status acquisition unit 525 uses the process database Db1 to identify status information associated with the one manufacturing process identified by the acquired process information. In this way, the status acquisition unit 525 acquires status information about the vehicle 10 included in the captured image. Note that the status acquisition step may be executed at any point after the image acquisition step is completed and before the detection step (step S7) is started.

[0061] After the state acquisition step, a model acquisition step (step S6) is executed. The model acquisition step is a step of acquiring a first detection model Md1 selected from a plurality of first detection models Md1 prepared for each state according to a state specified by the state information acquired in the state acquisition step. In this embodiment, in the model acquisition step, the model acquisition unit 526 selects and acquires a first detection model Md1 corresponding to a state specified by the state information acquired in the state acquisition step from a plurality of first detection models Md1 pre-stored in the storage unit 53 of the detection device 5. Note that the model acquisition step may be executed at any time after the state acquisition step is completed and before the detection step is started.

[0062] After the model acquisition step, the detection step is executed. As shown in FIG. 8, the detection step is a step of detecting the outer shape of the vehicle 10 included in the captured image by using a detection model such as the first detection model Md1. In this embodiment, in the detection step, the detection unit 527 inputs the processed image Im4 to the first detection model Md1 selected in the model acquisition step. As a result, the detection unit 527 detects the outer shape of the vehicle 10 included in the processed image Im4. Then, the detection unit 527 generates a first mask image Im5 in which a mask area Ms is added to the target area by masking the target area in the processed image Im4. The first transmission unit 528 transmits the first mask image Im5 to the position calculation device 6. Note that the detection method of the vehicle 10 is not limited to the above.

[0063] As shown in FIG. 7, after the detection step, a perspective transformation step (step S8) is executed. The perspective transformation step is a step of performing perspective transformation on the first mask image Im5. In the perspective transformation step, the data acquisition unit 621 acquires the first mask image Im5 from the detection device 5. Then, the perspective transformation unit 622 performs perspective transformation on the first mask image Im5 to generate a second mask image Im6. Specifically, the perspective transformation unit 622 performs perspective transformation on the first mask image Im5 into a bird's-eye image viewed from a viewpoint above the vehicle 10 (for example, directly above the vehicle 10) that is substantially perpendicular to the road surface 20, using, for example, a perspective transformation parameter Pa2 stored in advance in the storage unit 63 of the position calculation device 6. The perspective transformation parameter Pa2 is, for example, a parameter related to position information and internal parameters of the imaging device 9 obtained by calibration. As a result, the perspective transformation unit 622 generates a second mask image Im6 expressed in the image coordinate system from the first mask image Im5 expressed in the camera coordinate system. The image coordinate system has a point on the image plane projected by the perspective transformation as its origin, and has coordinate axes indicated by the Xi axis and the Yi axis perpendicular to the Xi axis. Note that the perspective transformation method is not limited to the above. The perspective transformation parameter Pa2 may be any parameter other than those described above.

[0064] As shown in Fig. 7, after the perspective transformation step, a coordinate point calculation step (step S9) is executed. Fig. 9 is a diagram for explaining the details of the coordinate point calculation step. The coordinate point calculation step is a step for calculating an image coordinate point P3 indicating the position of the vehicle 10 in the image coordinate system.

[0065] In the coordinate point calculation step, the coordinate point calculation unit 623 calculates a base coordinate point P0 from a first circumscribing rectangle R1 set in a mask area Ms in a first mask image Im5, which is an image before perspective transformation. FIG. 10 is a diagram for explaining a method for calculating the base coordinate point P0. In order to calculate the base coordinate point P0, the coordinate point calculation unit 623 sets a base circumscribing rectangle R0 for the mask area Ms in the first mask image Im5. Next, the coordinate point calculation unit 623 rotates the first mask image Im5 by a required rotation amount with the center of gravity C of the mask area Ms as the rotation center so that the direction of the movement vector V of the vehicle 10 corresponding to the mask area Ms in the first mask image Im5 faces a predetermined direction. The predetermined direction is, for example, the upward direction on the screen of the display device that displays the first mask image Im5. Next, the coordinate point calculation unit 623 sets a first circumscribing rectangle R1 for the mask area Ms of the rotated first mask image Im5 so that the long side is parallel to the direction of the movement vector V. Next, the coordinate point calculation unit 623 rotates the first mask image Im5 to which the first circumscribing rectangle R1 has been added by the above-mentioned rotation amount with the center of gravity C of the mask area Ms as the rotation center inversely. As a result, the coordinate point calculation unit 623 sets the coordinate point of the vertex having the coordinates that are the closest to the positioning point 10e of the vehicle 10, which is one of the four vertices of the first circumscribing rectangle R1, as the base coordinate point P0. Then, as shown in FIG. 9, the coordinate point calculation unit 623 performs perspective transformation on the first mask image Im5 after the reverse rotation, that is, the first mask image Im5 after the calculation of the base coordinate point P0. As a result, the coordinate point calculation unit 623 determines the coordinate point corresponding to the base coordinate point P0 in the first circumscribing rectangle R1 transformed by the perspective transformation as the first coordinate point P1.

[0066] Furthermore, the coordinate point calculation unit 623 sets a second circumscribing rectangle R2 for the mask area Ms in the second mask image Im6 obtained by performing perspective transformation on the first mask image Im5. Then, the coordinate point calculation unit 623 sets the vertex indicating the same position as the first coordinate point P1 as the second coordinate point P2 among the vertices of the second circumscribing rectangle R2. That is, the first coordinate point P1 and the second coordinate point P2 are coordinate points indicating the same position, and therefore have a correlation with each other.

[0067] Furthermore, the coordinate point calculation unit 623 performs correction to replace the coordinates (Xi1, Yi1) of the first coordinate point P1 with the coordinates (Xi2, Yi2) of the second coordinate point P2 depending on the magnitude relationship between the coordinate values ​​of the first coordinate point P1 and the second coordinate point P2. If the coordinate value Xi1 in the Xi direction of the first coordinate point P1 is greater than the coordinate value Xi2 in the Xi direction of the second coordinate point P2 (Xi1>Xi2), the coordinate point calculation unit 623 replaces the coordinate value Xi1 in the Xi direction of the first coordinate point P1 with the coordinate value Xi2 in the Xi direction of the second coordinate point P2. When the coordinate value Yi1 of the first coordinate point P1 in the Yi direction is greater than the coordinate value Yi2 of the second coordinate point P2 in the Yi direction (Yi1>Yi2), the coordinate point calculation unit 623 replaces the coordinate value Yi1 of the first coordinate point P1 in the Yi direction with the coordinate value Yi2 of the second coordinate point P2 in the Yi direction. In the example shown in FIG. 9, the coordinate value Xi1 of the first coordinate point P1 in the Xi direction is greater than the coordinate value Xi2 of the second coordinate point P2 in the Xi direction. Also, the coordinate value Yi1 of the first coordinate point P1 in the Yi direction is smaller than the coordinate value Yi2 of the second coordinate point P2 in the Yi direction. Therefore, the image coordinate point P3 has coordinates (Xi2, Yi1). In this way, the coordinate point calculation unit 623 calculates the image coordinate point P3 indicating the position of the vehicle 10 in the image coordinate system by correcting the first coordinate point P1 using the second coordinate point P2. The method of calculating the image coordinate point P3 is not limited to the above.

[0068] As shown in FIG. 7, after the coordinate point calculation step, a position conversion step (step S10) is executed. The position conversion step is a step of converting the image coordinate point P3 into a vehicle coordinate point to calculate a vehicle coordinate point indicating the position of the vehicle 10 in the global coordinate system. In the position conversion step, the position conversion unit 624 converts the image coordinate point P3 into a vehicle coordinate point using the imaging parameters and vehicle parameters acquired by the data acquisition unit 621. The vehicle parameters are parameters related to the distance from a reference point to the positioning point 10e of the vehicle 10. In this embodiment, the vehicle parameters are the height h of the positioning point 10e of the vehicle 10 from the road surface 20 (FIG. 11 described later).

[0069] The position conversion unit 624 converts the image coordinate point P3 into a vehicle coordinate point using the relational expressions (1) to (3) described below, which use the vehicle coordinate point as a target variable and include the image coordinate point P3, the imaging parameters, and the vehicle parameters as explanatory variables. In this case, the position conversion unit 624 substitutes the coordinate value of the image coordinate point P3 calculated by the coordinate point calculation unit 623 into the relational expressions represented by the expressions (1) to (3). Furthermore, the position conversion unit 624 substitutes the imaging parameters acquired by the data acquisition unit 621, i.e., the values ​​of the imaging parameters corresponding to the imaging device 9 that acquired the original image Im1, into the relational expressions represented by the expressions (1) to (3).

[0070] FIG. 11 is a first diagram for explaining the details of the position conversion process. FIG. 11 illustrates a state in which the vehicle 10 is viewed from the left side. FIG. 12 is a second diagram for explaining the details of the position conversion process. FIG. 12 illustrates a state in which the vehicle 10 is viewed from the roof side. The global coordinate system shown in FIG. 11 and FIG. 12 is a coordinate system having a fixed coordinate point Pf indicating an arbitrary reference position of the road surface 20 as the origin and coordinate axes indicated by the Xg axis and the Yg axis perpendicular to the Xg axis. The imaging coordinate point Pc is the position of the imaging device 9 that acquired the original image Im1 used in calculating the image coordinate point P3, and is a coordinate point indicating the position of the imaging device 9 in the global coordinate system. The fixed coordinate point Pf and the imaging coordinate point Pc are stored in advance in the storage unit 63 of the position calculation device 6.

[0071] As shown in Fig. 11, the observation distance on the XgYg plane between the position of the imaging device 9 and the position of the vehicle 10 (image coordinate point P3) is set to Do. The observation error is set to ΔD. The height [m] of the imaging device 9 from the road surface 20 as an imaging parameter is set to H. The height [m] of the positioning point 10e of the vehicle 10 from the road surface 20 as a vehicle parameter is set to h. In this case, the observation error ΔD can be expressed by the following equation (1). ΔD=h / H×Do Equation (1) That is, the larger the observation distance Do, the larger the observation error ΔD.

[0072] Next, if the actual distance between the position of the imaging device 9 and the position of the positioning point 10e of the vehicle 10 (hereinafter, referred to as the first distance) is D, the first distance D can be expressed by the following formula (2). D=Do×(1-h / H) Equation (2) That is, the first distance D is determined by the observation distance Do, the height H of the imaging device 9 as an imaging parameter, and the height h of the positioning point 10e of the vehicle 10 as a vehicle parameter.

[0073] Then, as shown in FIG. 12, if the estimated distance between the reference position and the position of vehicle 10 is Dp, and the actual distance between the reference position and vehicle 10 (hereinafter, the second distance) is Dt, the second distance Dt can be expressed by the following equation (3). Dt = Dp × (1-h / H) Equation (3)

[0074] Here, the estimated distance Dp can be calculated using an actual distance (hereinafter, the third distance Dc) calculated from the fixed coordinate point Pf and the imaged coordinate point Pc, the image coordinate point P3, and the fixed coordinate point Pf. Therefore, the position conversion unit 624 can calculate the vehicle coordinate point Pv using the second distance Dt calculated by correcting the estimated distance Dp using the above formula (3) and the fixed coordinate point Pf. At this time, the calculated vehicle coordinate point Pv is a coordinate point indicating the position of the vehicle 10 in the global coordinate system, and therefore corresponds to the position of the vehicle 10 in the real space. The second transmission unit 625 transmits the vehicle coordinate point Pv to the remote control device 7.

[0075] A-3. Vehicle operation control method: Fig. 13 is a flowchart showing a method for controlling the operation of vehicle 10 traveling in remote automatic driving mode. The method for controlling the operation shown in Fig. 13 is repeatedly executed, for example, every time a vehicle coordinate point Pv is received from position calculation device 6 as information indicating the position of vehicle 10.

[0076] The information acquisition unit 721 of the remote control device 7 acquires driving information including the vehicle coordinate point Pv (step S101). Then, the control value creation unit 722 uses the driving information to create a control value that specifies the driving operation of the vehicle 10 (step S102). Then, the third transmission unit 723 transmits the control value to the vehicle 10 (step S103). The vehicle control device 150 mounted on the vehicle 10 drives the drive unit 110 and the like according to the received control value (step S104).

[0077] According to the first embodiment, when the vehicle 10 is made to travel automatically by remote control, the position calculation system 1 can calculate the position of the vehicle 10 by using a captured image including the vehicle 10. Specifically, the detection device 5 inputs the captured image to a detection model that is a machine learning model capable of detecting the vehicle 10 included in the captured image. This enables the detection device 5 to detect the vehicle 10 included in the captured image. Then, the position calculation device 6 can calculate a vehicle coordinate point Pv indicating the position of the vehicle 10 included in the captured image by using a first mask image Im5 that is a detection result by the detection device 5.

[0078] Here, since the vehicle 10 has different appearances due to multiple manufacturing processes, it can be classified into multiple states according to the difference in appearance. As a result, the feature amount extracted when training the detection model differs depending on the appearance state of the vehicle 10. Therefore, when a detection model in which only multiple training images including the vehicles 10 classified into different states are trained in the same number for each state is applied to the vehicles 10 classified into any state, the detection accuracy of the vehicle 10 in some states may decrease. When the detection accuracy of the vehicle 10 in some states decreases, it is possible to add training images including the vehicle 10 in the state in which the detection accuracy has decreased and re-train the detection model. However, by training more training images including the vehicle 10 in some states than training images including the vehicle 10 in other states, there is a possibility that the feature amount included in the added training image is learned more intensively than other feature amounts. As a result, a new difference in detection accuracy may occur between multiple states. When the detection accuracy of the vehicle 10 included in the captured image decreases, the calculation accuracy of the vehicle coordinate point Pv indicating the position of the vehicle 10 decreases. When the calculation accuracy of the vehicle coordinate point Pv decreases, there is a risk that the desired control value cannot be created.

[0079] Therefore, in the first embodiment, the first training image including the vehicle 10 classified into one state is trained more than the second training image including the vehicle 10 classified into another state, so that the first detection model Md1 suitable for detecting the vehicle 10 classified into one state is prepared for each state. According to the first embodiment, the detection device 5 acquires state information indicating the state of the vehicle 10 included in the captured image, and can select the first detection model Md1 corresponding to the one state specified by the acquired state information from among the multiple first detection models Md1 prepared for each state. In other words, when calculating the position of the vehicle 10 traveling in a factory, the first detection model Md1 corresponding to the state of the vehicle 10 classified into multiple states by having different appearances due to multiple manufacturing processes can be selected. Then, the detection device 5 inputs the captured image to the first detection model Md1 corresponding to the state of the vehicle 10 included in the captured image. This allows the detection device 5 to accurately detect the vehicle 10 included in the captured image. That is, by selecting the first detection model Md1 suitable for detecting the vehicle 10 classified into one state, each area constituting the captured image can be accurately classified into a target area and a non-target area. As a result, it is possible to prevent the detection accuracy of the vehicle 10 included in the captured image from decreasing according to the difference in appearance in a plurality of manufacturing processes. As a result, it is possible to prevent the calculation accuracy of the vehicle coordinate point Pv from decreasing. This makes it possible to reduce the difference between the position of the vehicle 10 calculated by the position calculation device 6 and the actual position of the vehicle 10. Therefore, it is possible to create a more appropriate control value.

[0080] According to the first embodiment, the first teacher data set includes M (M is an integer equal to or greater than 2) first training images and N (N is an integer equal to or greater than 0 and less than M) second training images. In this case, the first teacher data set may include only the first training images without including the second training images. In this case, for example, the detection accuracy of the first detection model Md1 can be improved by increasing the number M of first training images including the vehicle 10 classified into one state. The first teacher data set may also include the first training images and N number of second training images smaller than the first training images. In this case, by including the second training images in the first teacher data set, for example, the detection accuracy of a part of the vehicle 10 having a common appearance regardless of the one state can be improved.

[0081] Moreover, according to the first embodiment, when the form of the vehicle 10 is a platform form, the state of the vehicle 10 can be classified into a platform state. Then, when detecting a vehicle 10 classified into the platform state, the vehicle 10 can be detected by selecting the first detection model Md1 suitable for detecting the vehicle 10 classified into the platform state and inputting a captured image. Also, when detecting a vehicle 10 classified into the platform state, the vehicle 10 can be detected by inputting a captured image associated with the platform state into the second detection model Md2. This can improve the detection accuracy of the vehicle 10 classified into the platform state.

[0082] Furthermore, according to the first embodiment, when the platform vehicle 101 in the form of a platform is included in the captured image, the external shape of the platform vehicle 101 can be detected. This makes it possible to calculate the position of the vehicle 10 using the captured image not only for the vehicle 10 as a completed vehicle, but also for the vehicle 10 as a semi-finished product or work in progress.

[0083] According to the first embodiment, the platform vehicle 101 has a significantly different appearance from the first assembly vehicle 102 and the second assembly vehicle 103. This allows the state of the vehicle 10 to be classified into a platform state and an assembly state before and after the assembly of the vehicle body such as the body shell 180. This makes it possible to suppress a decrease in the detection accuracy of the vehicle 10 included in the captured image according to the difference in the external shape of the vehicle 10 that differs before and after the assembly of the vehicle body.

[0084] Furthermore, according to the first embodiment, when detecting the vehicle 10 included in the captured image, the distortion of the original image Im1 can be corrected to generate a corrected image Im2. In this way, the detection accuracy of the vehicle 10 included in the captured image can be improved. This can further improve the calculation accuracy of the position of the vehicle 10.

[0085] Furthermore, according to the first embodiment, when detecting a vehicle 10 included in a captured image, a rotated image Im3 can be generated by rotating the corrected image Im2 so that the direction of the movement vector V of the vehicle 10 faces a predetermined direction. In this way, the vehicle 10 included in the captured image can be detected with the direction of the movement vector V being unified. This can further improve the detection accuracy of the vehicle 10 included in the captured image.

[0086] Furthermore, according to the first embodiment, when detecting a vehicle 10 included in a captured image, a crop process is performed to cut out a necessary area including the vehicle 10 from the rotated image Im3, thereby generating a processed image Im4 from which unnecessary areas have been removed. In this way, the area of ​​the vehicle 10 in the captured image can be made larger than when the crop process is not performed. This makes it easier to detect a vehicle 10 that is farther away from the imaging device 9. Therefore, the detection accuracy for a vehicle 10 that is farther away from the imaging device 9 can be improved.

[0087] According to the first embodiment, the processed image Im4 is input to the first detection model Md1, and the target area among the areas constituting the processed image Im4 is masked. This makes it possible to generate a first mask image Im5 in which a mask area Ms is added to the target area. At this time, according to the first embodiment, a DNN having a CNN structure that realizes semantic segmentation and instance segmentation can be used as the algorithm of the first detection model Md1. This makes it possible to suppress a decrease in the detection accuracy of the vehicle 10 due to the diversity of non-target areas in the captured image.

[0088] Furthermore, according to the first embodiment, the second mask image Im6 can be generated by performing perspective transformation on the first mask image Im5, thereby making it possible to transform the camera coordinate system into the image coordinate system.

[0089] According to the first embodiment, the first circumscribing rectangle R1 is set for the mask area Ms before the perspective transformation of the first mask image Im5, so that the base coordinate point P0, which is the vertex of the first circumscribing rectangle R1 having the coordinates closest to the positioning point 10e of the vehicle 10, can be calculated. Then, the first coordinate point P1, which is the coordinate point corresponding to the base coordinate point P0, can be calculated by performing perspective transformation on the first mask image Im5 after the calculation of the base coordinate point P0. Furthermore, the second circumscribing rectangle R2 is set for the mask area Ms of the second mask image Im6, so that the second coordinate point P2, which is the vertex of the second circumscribing rectangle R2 having the coordinates closest to the positioning point 10e of the vehicle 10, can be calculated. Then, the first coordinate point P1 is corrected using the second coordinate point P2, so that the image coordinate point P3 can be calculated. In this way, the image coordinate point P3 can be calculated more accurately by comparing and correcting the coordinate points before and after the perspective transformation. This can further improve the calculation accuracy of the position of the vehicle 10.

[0090] Furthermore, according to the first embodiment, it is possible to acquire process information related to one manufacturing process being executed on the vehicle 10. Then, by using the process database Db1 to identify status information associated with one manufacturing process identified by the acquired process information and acquiring status information about the vehicle 10, it is possible to identify the status of the vehicle 10 included in the captured image.

[0091] Furthermore, according to the first embodiment, it is possible to acquire imaging parameters related to the imaging device 9 that acquired the original image Im1. Then, by substituting the calculated image coordinate point P3 and the acquired imaging parameter values ​​into a relational expression that uses the vehicle coordinate point Pv as a response variable and includes the image coordinate point P3, the imaging parameters, and the vehicle parameters as explanatory variables, it is possible to convert the image coordinate point P3 into the vehicle coordinate point Pv. This allows the image coordinate system to be converted into a global coordinate system, and the position of the positioning point 10e of the vehicle 10 in the global coordinate system to be calculated as the vehicle coordinate point Pv.

[0092] Moreover, according to the first embodiment, the imaging parameter is the height H of the imaging device 9 from the road surface 20, which is calculated based on the position of the imaging device 9 in the global coordinate system. Moreover, the vehicle parameter is the height h of the positioning point 10e of the vehicle 10 from the road surface 20. In this way, the observation error ΔD can be calculated based on the similarity relationship between the imaging parameter and the vehicle parameter. Then, the image coordinate point P3 can be converted into the vehicle coordinate point Pv using the calculated observation error ΔD.

[0093] Moreover, according to the first embodiment, the position of the vehicle 10 can be calculated without installing any installation object such as a marker, a transceiver, etc., on the vehicle 10 for use in calculating the position of the vehicle 10. Also, the position of the vehicle 10 can be calculated without mounting the position calculation device 6 on the vehicle 10. This can enhance the versatility of the position calculation system 1.

[0094] B. Second embodiment: FIG. 14 is a diagram showing the manufacturing process of the vehicle 10 in the second embodiment. FIG. 14 shows a representative portion of a plurality of manufacturing steps executed in the manufacturing process of the vehicle 10. FIG. 15 is a diagram showing the configuration of the detection device 5a in the second embodiment. In this embodiment, a portion of the manufacturing process of the vehicle 10 is different from that in the first embodiment. As a result, a portion of the configuration of the detection device 5a is different from that in the first embodiment. The other configurations are the same as those in the first embodiment. The same reference numerals are used for the same configurations as those in the first embodiment, and description thereof will be omitted.

[0095] As shown in FIG. 14, in this embodiment, the vehicle 10 as a finished product is manufactured by assembling a body shell 180 to a platform vehicle 101, assembling interior parts and exterior parts 190, and painting the vehicle body (body portion) consisting of the body shell 180, the exterior parts 190, etc. That is, in this embodiment, the platform manufacturing process, the body shell assembling process, the parts assembling process, and the painting process are performed in this order, thereby manufacturing the vehicle 10 as a finished product. The painting process is a process of painting the vehicle 10. In the painting process, the vehicle 104 before being painted (hereinafter, unpainted vehicle 104) and the vehicle 105 after being painted in the painting process (hereinafter, painted vehicle 105) have different exterior colors, and therefore the appearance of the vehicle 10 is different. Therefore, in this embodiment, the storage unit 53a of the detection device 5a stores three first detection models Md1c to Md1e as detection models suitable for detecting the vehicle 10 classified into one state determined by the exterior shape and exterior color of the vehicle 10. In the following, the state of the vehicle 10 before it is painted in the painting process, such as the unpainted vehicle 104, is referred to as the "unpainted state." The state of the vehicle 10 after it has been painted in the painting process, such as the painted vehicle 105, is referred to as the "painted state."

[0096] The first first detection model Md1c is a first detection model Md1 that has been trained on more vehicles 10 classified as "platform state" than on vehicles 10 classified as states other than the "platform state". In the first first detection model Md1c, the first training image is an image including a vehicle 10 classified as "platform state". The second training image is, for example, an image including a vehicle 10 classified as "assembled state" and "unpainted state". The first first detection model Md1a is used when detecting a vehicle 10 classified as "platform state".

[0097] The second first detection model Md1d is a first detection model Md1 that has been trained on more vehicles 10 classified as "assembled" and "unpainted" than on vehicles 10 classified as states other than "assembled" and "unpainted". In the second first detection model Md1d, the first training image is an image including a vehicle 10 classified as "assembled" and "unpainted". The second training image is, for example, an image including a vehicle 10 classified as "assembled" and "painted". The second first detection model Md1d is used when detecting a vehicle 10 in an "assembled" and "unpainted" state.

[0098] The third first detection model Md1e is a first detection model Md1 that has been trained on more vehicles 10 classified as "assembled" and "painted" than on vehicles 10 classified as states other than "assembled" and "painted". In the third first detection model Md1e, the first training image is an image including a vehicle 10 classified as "assembled" and "painted". The second training image is, for example, an image including a vehicle 10 classified as "assembled" and "unpainted". The third first detection model Md1e is used when detecting a vehicle 10 classified as "assembled" and "painted".

[0099] According to the second embodiment, the appearance of the unpainted vehicle 104 and the painted vehicle 105 is significantly different. This allows the state of the vehicle 10 to be classified into an unpainted state and a painted state before and after painting in the painting process. This makes it possible to suppress a decrease in the detection accuracy of the vehicle 10 included in the captured image according to the difference in the appearance color of the vehicle 10 that differs before and after the painting process is performed.

[0100] C. Third embodiment: FIG. 16 is a diagram showing the configuration of a detection device 5b in the third embodiment. In this embodiment, the detection model used when detecting a vehicle 10 included in a captured image is different from that in the first embodiment. As a result, part of the configuration of the detection device 5b and part of the processing of the method for detecting the vehicle 10 are different from those in the first embodiment. The other configurations are the same as those in the first embodiment. The same reference numerals are used for the steps that are the same as those in the first embodiment, and the description thereof will be omitted.

[0101] In this embodiment, the storage unit 53b of the detection device 5b stores one second detection model Md2 as a detection model. The second detection model Md2 is a trained machine learning model used to detect the vehicle 10 included in the captured image, similar to the first detection model Md1. The second detection model Md2 is a machine learning model trained by inputting a second teacher data set. The second teacher data set has a plurality of training images each including the vehicle 10 classified into different states, a region correct answer label associated with each of a plurality of regions constituting each training image, and a state correct answer label associated with each of the plurality of training images. As in the first embodiment, the region correct answer label is a correct answer label indicating whether each region in the training image is a target region indicating the vehicle 10 or a non-target region indicating other than the vehicle 10. The state correct answer label is a correct answer label indicating one state of the vehicle 10 included in the training image. In this embodiment, the second detection model Md2 executes the following process when a captured image is input together with one state specified by the state information acquired by the state acquisition unit 525. In this case, the second detection model Md2 identifies a target area included in the input captured image. Then, the second detection model Md2 generates a first mask image Im5 in which a mask area Ms is added to the target area by masking the target area. For example, CNN is used as an algorithm of the second detection model Md2. Note that the configuration of the second detection model Md2 is not limited to the above. For example, the second detection model Md2 may be a trained machine learning model that uses an algorithm other than a neural network.

[0102] 17 is a diagram showing details of the CPU 52b mounted on the detection device 5b in the third embodiment. The CPU 52b of the detection device 5b in this embodiment includes a model acquisition unit 526b instead of the model acquisition unit 526 in the first embodiment. The model acquisition unit 526b acquires a second detection model Md2. The CPU 52b of the detection device 5b includes a detection unit 527b instead of the detection unit 527 in the first embodiment. The detection unit 527b inputs a captured image and one state identified by the state information acquired by the state acquisition unit 525 to the second detection model Md2, and detects the vehicle 10 included in the captured image by identifying a target region among the regions constituting the input captured image.

[0103] Fig. 18 is a flowchart showing a method for detecting a vehicle 10 and a method for calculating a position of the vehicle 10 in the third embodiment. The method shown in Fig. 18 is repeatedly executed at predetermined time intervals after the vehicle 10 to be detected starts traveling in the remote autonomous driving mode, for example.

[0104] In this embodiment, in a model acquisition step (step S6b), the model acquisition unit 526b acquires a second detection model Md2. Then, in a detection step (step S7b), the detection unit 527b inputs the processed image Im4 and one state identified by the state information acquired by the state acquisition unit 525 to the second detection model Md2. As a result, the detection unit 527b detects the outline of the vehicle 10 included in the processed image Im4. Then, the detection unit 527b masks the target area in the processed image Im4 to generate a first mask image Im5 in which a mask area Ms is added to the target area.

[0105] According to the third embodiment, the second detection model Md2 is prepared as a detection model in which a plurality of training images including the vehicle 10 and state answer labels indicating the state of the vehicle 10 included in each training image are associated and learned. As a result, by inputting a captured image associated with the state of the vehicle 10 to the second detection model Md2, the vehicle 10 included in the input captured image can be accurately detected. That is, each area constituting the captured image input to the second detection model Md2 can be accurately classified into a target area and a non-target area. As a result, it is possible to suppress a decrease in the detection accuracy of the vehicle 10 included in the captured image according to the difference in appearance in a plurality of manufacturing processes. As a result, it is possible to suppress a decrease in the calculation accuracy of the vehicle coordinate point Pv. As a result, it is possible to reduce the difference between the position of the vehicle 10 calculated by the position calculation device 6 and the actual position of the vehicle 10. Therefore, it is possible to create a more appropriate control value.

[0106] D. Fourth embodiment: FIG. 19 is a diagram showing the configuration of a detection device 5c in the fourth embodiment. In this embodiment, the method of acquiring state information is different from that of the first embodiment. As a result, part of the configuration of the detection device 5c is different from that of the first embodiment. The other configurations are the same as those of the first embodiment. The same reference numerals are used for the same configurations as those of the first embodiment, and the description thereof will be omitted.

[0107] In this embodiment, the storage unit 53c of the detection device 5c stores the specific model Md3 instead of the process database Db1. The specific model Md3 is a trained machine learning model used to identify one state of the vehicle 10 included in the captured image and acquire state information indicating the identified one state. Specifically, the specific model Md3 is a trained machine learning model trained to output state information about the vehicle 10 included in the captured image when the captured image is input. The specific model Md3 learns a feature amount according to the state of the vehicle 10 in order to identify one state of the vehicle 10. For example, CNN is used as the algorithm of the specific model Md3. Note that the configuration of the specific model Md3 is not limited to the above. The specific model Md3 may be a trained machine learning model using an algorithm other than a neural network, for example.

[0108] 20 is a diagram illustrating details of a CPU 52c mounted on a detection device 5c in the fourth embodiment. In this embodiment, a state acquisition unit 525c inputs a captured image to a specific model Md3, and thereby acquires state information about the vehicle 10 included in the captured image.

[0109] According to the fourth embodiment, the captured image is input to the specific model Md3 that has been trained to output status information about the vehicle 10 included in the captured image when the captured image is input, so that the status information about the vehicle 10 included in the captured image can be acquired. In other words, the status information of the vehicle 10 can be acquired by using machine learning.

[0110] E. Fifth embodiment: FIG. 21 is a diagram showing the configuration of a detection device 5d in the fifth embodiment. In this embodiment, the method of acquiring state information is different from that of the first embodiment. As a result, part of the configuration of the detection device 5d is different from that of the first embodiment. The other configurations are the same as those of the first embodiment. Descriptions of the same configurations as those of the first embodiment will be omitted.

[0111] In this embodiment, the storage unit 53d of the detection device 5d stores a state database Db3 instead of the process database Db1. The state database Db3 is a database that associates imaging information and state information related to the imaging device 9. The imaging information is information for identifying the imaging device 9 that acquired the original image Im1. The imaging information is, for example, camera identification information. In this case, the state database Db3 is a database that associates, for example, the camera identification information, the imaging range RG of the imaging device 9 identified by the camera identification information, and state information indicating the state of the vehicle 10 according to the manufacturing process performed in the imaging range RG. Note that the configurations of the state database Db3 and the imaging information are not limited to those described above. The imaging information may include, for example, the imaging range RG instead of the camera identification information.

[0112] 22 is a diagram showing details of a CPU 52d mounted on a detection device 5d in the fifth embodiment. In this embodiment, the status acquisition unit 525d acquires camera identification information indicating the imaging device 9 that acquired the original image Im1 as imaging information related to the imaging device 9 that acquired the original image Im1. Then, the status acquisition unit 525d acquires status information about the vehicle 10 included in the captured image by identifying status information associated with one imaging device 9 identified by the acquired imaging information using the status database Db3.

[0113] According to the fifth embodiment, it is possible to acquire imaging information related to the imaging device 9. Then, by using the state database Db3 in which imaging information and state information are associated with each other, state information associated with one imaging device 9 identified by the acquired imaging information is identified, whereby state information indicating the state of the vehicle 10 included in the captured image can be acquired. This makes it possible to acquire state information about the vehicle 10 included in the captured image without performing image analysis, thereby reducing the processing load of the state acquisition unit 525d in the state acquisition step.

[0114] F. Other Embodiments: F-1. Alternative embodiment 1: The vehicle 10 as a finished product may be manufactured by assembling a plurality of painted units. In this case, the vehicle 10 as a finished product may be manufactured by assembling, for example, a front unit, a rear unit, a left unit, a right unit, a ceiling unit, and a bottom unit. The front unit is a semi-finished product obtained by integrally molding a group of parts that form the front side of the vehicle 10. The front unit includes, for example, a front lamp, a front bumper, a front grille, and a bonnet. The rear unit is a semi-finished product obtained by integrally molding a group of parts that form the rear side of the vehicle 10. The rear unit includes, for example, a rear lamp, a rear bumper, and a trunk. The left unit is a semi-finished product obtained by integrally molding a group of parts that form the left side of the vehicle 10. The right unit is a semi-finished product obtained by integrally molding a group of parts that form the right side of the vehicle 10. The left unit and the right unit each include, for example, a side sill and a pillar. The ceiling unit is a semi-finished product obtained by integrally molding a group of parts that form the upper side of the vehicle 10. The ceiling unit includes, for example, a roof. The bottom unit is a semi-finished product in which a group of parts that form the bottom side of the vehicle 10 are integrally molded. The bottom unit includes, for example, a chassis 160 and a seat. Even in this form, the state of the vehicle 10 can be classified into a plurality of states according to the appearance of the vehicle 10. This makes it possible to suppress a decrease in the detection accuracy of the vehicle 10 according to the appearance of the vehicle 10 by inputting a captured image to the first detection model Md1 suitable for one state determined by the appearance of the vehicle 10, or inputting a captured image associated with the state of the vehicle 10 to the second detection model Md2.

[0115] F-2. Alternative embodiment 2: In the captured image, the smaller the contrast between the vehicle 10 and the road 2, the smaller the color difference between the target area and the non-target area, so there is a risk that the detection accuracy of the vehicle 10 will decrease depending on the exterior color of the vehicle 10 and the color of the road 2. Therefore, a plurality of first detection models Md1 used when detecting the vehicle 10 classified as a painted state may be prepared according to the exterior color of the vehicle 10. For example, when detecting the vehicle 10 classified as a painted state, the following two first detection models Md1 may be prepared. In this case, a first detection model Md1 for detecting the vehicle 10 painted with a light-colored paint having a brightness equal to or greater than a predetermined threshold value, and a first detection model Md1 for detecting the vehicle 10 painted with a dark-colored paint having a brightness less than the threshold value may be prepared. In this form, the state of the vehicle 10 can be classified into a plurality of states depending on the color of the paint applied in the painting process. This makes it possible to suppress a decrease in the detection accuracy of the vehicle 10 in the captured image depending on the color of the paint applied in the painting process, i.e., the difference in the exterior color of the vehicle 10.

[0116] F-3. Alternative embodiment 3: The training images included in the first teacher data set and the second teacher data set may be images that have not been subjected to processing such as distortion correction, rotation, and cropping, like the original image Im1. In this form, no image processing is required when preparing the first teacher data set and the second teacher data set. This can reduce the processing load when learning the first detection model Md1 and the second detection model Md2.

[0117] F-4. Alternative embodiment 4: The training images included in the first teacher data set and the second teacher data set may be any one of the corrected image Im2, the rotated image Im3, and the processed image Im4. In this embodiment, the detection accuracy of the vehicle 10 can be improved when the captured image obtained by processing the original image Im1 is input to either the first detection model Md1 or the second detection model Md2 as in the first and third embodiments.

[0118] F-5. Alternative embodiment 5: In the detection method of the vehicle 10 shown in Fig. 7 and Fig. 18, the distortion correction step is not an essential step. For example, if the steps after the rotation processing step are executed without executing the distortion correction step, the rotation processing unit 523 rotates the original image Im1 instead of the corrected image Im2 in the rotation processing step. Even in this form, the image coordinate point P3 can be calculated.

[0119] F-6. Other embodiment 6: In the detection method of the vehicle 10 shown in Fig. 7 and Fig. 18, the rotation process is not an essential step. For example, if the distortion correction step and the rotation process step are not performed, the crop processing unit 524 performs crop processing on the original image Im1 instead of the rotated image Im3 in the crop processing step. Even in this form, the image coordinate point P3 can be calculated.

[0120] F-7. Other embodiment 7: In the detection method of the vehicle 10 shown in Fig. 7 and Fig. 18, the cropping process is not an essential process. For example, if the distortion correction process, the rotation process, and the cropping process are not performed, the detection units 527 and 527b generate a first mask image Im5 by adding a mask area Ms to the original image Im1 in place of the processed image Im4 in the detection process. Even in this form, the image coordinate point P3 can be calculated.

[0121] F-8. Other embodiment 8: The captured image may include a plurality of vehicles 10. In this case, the CPU 52, 52b to 52d of the detection device 5, 5a to 5d may include, for example, a deletion unit that deletes a mask region Ms of the vehicle 10 that is not a target for position calculation from the first mask image Im5. For example, the deletion unit determines that a mask region Ms that exists outside the recognition target region among the mask regions Ms generated in the detection process is a mask region of the vehicle 10 that is not a target for position calculation, and deletes it from the first mask image Im5. The recognition target region is, for example, a predetermined region in the first mask image Im5 in which the vehicle 10 moves. The predetermined region in which the vehicle 10 moves is, for example, a region corresponding to the region of the grid lines 21. The recognition target region is stored in advance in the storage unit 53, 53a to 53d of the detection device 5, 5a to 5d. In this embodiment, when a plurality of vehicles 10 are included in the captured image, it is possible to eliminate the influence of the vehicle 10 that is not a target for position calculation. This improves the calculation accuracy of the position of the vehicle 10.

[0122] F-9. Other embodiment 9: The captured image may include a plurality of vehicles 10. In this case, a DNN that performs instance segmentation may be used as the algorithm of the first detection model Md1 and the second detection model Md2. In this form, the plurality of vehicles 10 included in the captured image can be classified, and a first mask image Im5 in which each vehicle 10 is masked can be generated. As a result, when the captured image includes a plurality of vehicles 10, the vehicle 10 to be the target of position calculation can be selected, and the position of the selected vehicle 10 can be calculated.

[0123] F-10. Other embodiment 10: The position calculation system 1 may calculate the position of a stationary vehicle 10. When calculating the position of a stationary vehicle 10, the position calculation system 1 calculates the position of the vehicle 10 by using, for example, an initial vector direction of the vehicle 10 estimated from an original image Im1 acquired first after the start of the position calculation system 1, instead of the direction of the movement vector V of the vehicle 10 while traveling. In this manner, even when the vehicle 10 is stopped, the position of the vehicle 10 can be calculated by using the captured image.

[0124] F-11. Other embodiment 11: At least one of the detection devices 5, 5a to 5d, the position calculation device 6, and the remote control device 7 may be configured as an integrated device. Furthermore, each of the detection devices 5, 5a to 5d, the position calculation device 6, and the remote control device 7 may be realized by, for example, cloud computing configured by one or more computers. In such a form, the configurations of the detection devices 5, 5a to 5d, the position calculation device 6, and the remote control device 7 can be appropriately changed.

[0125] F-12. Other embodiment 12: The image acquiring unit 521 of the detection devices 5, 5a to 5d may acquire the original image Im1 via an external device (e.g., the remote control device 7) other than the imaging device 9, without acquiring the original image Im1 directly from the imaging device 9. Even in this embodiment, the image acquiring unit 521 of the detection devices 5, 5a to 5d can acquire the original image Im1 acquired by the imaging device 9.

[0126] F-13. Other embodiment 13: The detection devices 5, 5a to 5d may detect the vehicle 10 using the first detection model Md1 that is stored in a device other than the detection devices 5, 5a to 5d and selected by the device other than the detection devices 5, 5a to 5d. Even in this form, the first detection model Md1 according to the state of the vehicle 10 can be obtained.

[0127] The present disclosure is not limited to the above-mentioned embodiment, and can be realized in various configurations without departing from the spirit of the present disclosure. For example, the technical features of the embodiments corresponding to the technical features in each aspect described in the Summary of the Invention can be appropriately replaced or combined to solve some or all of the above-mentioned problems or to achieve some or all of the above-mentioned effects. Furthermore, if a technical feature is not described as essential in this specification, it can be appropriately deleted. [Explanation of symbols]

[0128] 1...position calculation system, 2...roadway, 5,5a to 5d...detection device, 6...position calculation device, 7...remote control device, 9...imaging device, 10...vehicle, 10e...positioning point, 20...road surface, 21...grid line, 51...communication unit of detection device, 52,52b to 52d...CPU of detection device, 53,53a to 53d...storage unit of detection device, 61...communication unit of position calculation device, 62...CPU of position calculation device, 63...storage unit of position calculation device, 71...communication unit of remote control device, 72...CPU of remote control device, 73...storage unit of remote control device, 101...platform vehicle, 102...first assembly vehicle, 103... Second assembly vehicle, 104... unpainted vehicle, 105... painted vehicle, 110... drive unit, 120... steering unit, 130... braking unit, 140... vehicle communication unit, 150... vehicle control device, 160... chassis, 170... wheels, 180... body shell, 190... exterior parts, 521... image acquisition unit, 522... distortion correction unit, 523... rotation processing unit, 524... crop processing unit, 525, 525c, 525d... status acquisition unit, 526, 526b... model acquisition unit, 527, 527b... detection unit, 528... first transmission unit, 621... data acquisition unit, 622... perspective transformation unit, 623... coordinate point calculation unit, 624... position position conversion unit, 625...second transmission unit, 721...information acquisition unit, 722...control value creation unit, 723...third transmission unit, 901...first imaging device, 902...second imaging device, A1...unmoved area, A2...moved area, C...center of gravity, ΔD...observation error, D...first distance, Db1...process database, Db2...camera database, Db3...state database, Dc...third distance, Do...observation distance, Dp...estimated distance, Dt...second distance, h...height of vehicle positioning point from road surface, H...height of imaging device from road surface, Im1...original image, Im2...corrected image, Im3...rotated image, Im4...processed image, I m5...first mask image, Im6...second mask image, Md1, Md1a to Md1e...first detection model, Md2...second detection model, Md3...specific model, Ms...mask area, Nt...network, P0...base coordinate point, P1...first coordinate point, P2...second coordinate point, P3...image coordinate point, Pa1...distortion correction parameter, Pa2...perspective transformation parameter, Pc...imaging coordinate point, Pf...fixed coordinate point, Pv...vehicle coordinate point, R0...base circumscribing rectangle, R1...first circumscribing rectangle, R2...second circumscribing rectangle, RG...imaging range, RG1...first imaging range, RG2...second imaging range, V...movement vector

Claims

1. A detection device for detecting a vehicle included in a captured image, The vehicle travels through a factory where a plurality of manufacturing processes are performed to manufacture and ship the vehicle; The vehicle is classified into a plurality of states by having different appearances according to the plurality of manufacturing processes; The detection device includes: an image acquisition unit for acquiring the captured image; a state acquisition unit that acquires state information indicating one of the states of the vehicle included in the captured image; a model acquisition unit that acquires the first detection model selected from a plurality of first detection models, which are machine learning models prepared for each of the one states, in accordance with the one state identified by the state information acquired by the state acquisition unit; a detection unit that detects the vehicle included in the captured image by inputting the captured image to the first detection model acquired by the model acquisition unit and identifying a target area indicating the vehicle in the captured image.

2. A detection device for detecting a vehicle included in a captured image, The vehicle travels through a factory where a plurality of manufacturing processes are performed to manufacture and ship the vehicle; The vehicle is classified into a plurality of states by having different appearances according to the plurality of manufacturing processes; the plurality of states includes a platform state in which the vehicle is in the form of a platform including at least wheels, a chassis, a drive unit for accelerating the vehicle, a steering unit for changing the traveling direction of the vehicle, a braking unit for decelerating the vehicle, a vehicle control device for controlling the operation of the vehicle, and a vehicle communication unit for communicating with devices other than the vehicle itself; The detection device includes: an image acquisition unit for acquiring the captured image; a state acquisition unit that acquires state information indicating one of the states of the vehicle included in the captured image; a model acquisition unit that acquires the first detection model selected from a plurality of first detection models, which are machine learning models prepared for each of the one states, in accordance with the one state identified by the state information acquired by the state acquisition unit; a detection unit that detects the vehicle included in the captured image by inputting the captured image to the first detection model acquired by the model acquisition unit and identifying a target area indicating the vehicle in the captured image.

3. A detection device for detecting a vehicle included in a captured image, The vehicle travels through a factory where a plurality of manufacturing processes are performed to manufacture and ship the vehicle; The vehicle is classified into a plurality of states by having different appearances according to the plurality of manufacturing processes; The detection device includes: an image acquisition unit for acquiring the captured image; a state acquisition unit that acquires state information indicating one of the states of the vehicle included in the captured image; a detection unit that detects the vehicle included in the captured image by inputting the captured image and the one state identified by the state information acquired by the state acquisition unit into a second detection model, which is a machine learning model, and identifying a target area in the captured image that indicates the vehicle.

4. 2. The detection device according to claim 1, A detection device, wherein each of the multiple first detection models has been pre-trained to identify the target area by inputting multiple training images including M (M is an integer equal to or greater than 2) first training images each including the vehicle classified into the one state, and N (N is an integer equal to or greater than 0 and less than M) second training images each including the vehicle classified into another state different from the one state.

5. 3. The detection device according to claim 2, A detection device, wherein each of the multiple first detection models has been pre-trained to identify the target area by inputting multiple training images including M (M is an integer equal to or greater than 2) first training images each including the vehicle classified into the one state, and N (N is an integer equal to or greater than 0 and less than M) second training images each including the vehicle classified into another state different from the one state.

6. 4. The detection device according to claim 3, The second detection model is pre-trained to identify the target region by inputting a plurality of training images including the vehicle and state answer labels associated with each of the plurality of training images, the state answer labels indicating the state of the vehicle included in the training images.

7. A detection device according to any one of claims 4 to 6, A detection device in which each region in the training image is associated with a region answer label indicating whether the region is the target region or a non-target region indicating a region other than the vehicle.

8. A detection device according to any one of claims 1 to 3, The status acquisition unit acquires process information related to one of the manufacturing processes being performed on the vehicle, and acquires the status information by using a process database in which the status information is associated with each of the multiple manufacturing processes, to identify the status information associated with the one manufacturing process identified by the acquired process information.

9. A detection device according to any one of claims 1 to 3, The state acquisition unit acquires the state information by inputting the captured image to a specific model, which is a machine learning model that has been trained to output the state information when the captured image is input.

10. A detection device according to any one of claims 1 to 3, The status acquisition unit acquires imaging information related to an imaging device, and acquires the status information by identifying the status information associated with one of the imaging devices identified by the acquired imaging information using a status database in which the imaging information and the status information are associated.

11. A detection device according to any one of claims 1 to 3, the plurality of manufacturing processes includes a painting process of painting the vehicle, The detection device, wherein the multiple states include (iii) an unpainted state indicating the state of the vehicle before it is painted in the painting process, and (iv) a painted state indicating the state of the vehicle after it is painted in the painting process.

12. The detection device according to any one of claims 1 to 3, further comprising: The detection device further comprises a distortion correction unit that corrects distortion of the captured image.

13. The detection device according to any one of claims 1 to 3, further comprising: a rotation processing unit that rotates the captured image so that the moving direction of the vehicle faces a predetermined direction.

14. A position calculation system for calculating a position of a vehicle included in a captured image, A detection device according to any one of claims 1 to 3, a position calculation device for calculating a position of the vehicle, The detection device further generates a first mask image by masking a target area, which is an area indicating the vehicle, in the captured image and adding a mask area to the target area, and a position calculation device that calculates a position of the vehicle using the first mask image, The position calculation device is a perspective transformation unit that performs perspective transformation on the first mask image to generate a second mask image; a coordinate point calculation unit that calculates an image coordinate point that indicates a position of the vehicle in an image coordinate system by setting a specified vertex of a first circumscribing rectangle set in the mask area in the first mask image as a first coordinate point, setting a vertex of a second circumscribing rectangle set in the mask area in the second mask image that indicates the same position as the first coordinate point as a second coordinate point, and correcting the first coordinate point using the second coordinate point; a position conversion unit that converts the image coordinate points into vehicle coordinate points that indicate the position of the vehicle in the global coordinate system using a distance from a reference point of the imaging device calculated based on the position of the imaging device in a global coordinate system and a distance from the reference point of a predetermined positioning point of the vehicle.

15. A method for detecting a vehicle included in a captured image, comprising: The vehicle travels through a factory where a plurality of manufacturing processes are performed to manufacture and ship the vehicle; The vehicle is classified into a plurality of states by having different appearances according to the plurality of manufacturing processes; The detection method includes: an image acquisition step of acquiring the captured image; a state acquiring step of acquiring state information indicating one of the states of the vehicle included in the captured image; a model acquisition step of acquiring the first detection model selected from a plurality of first detection models, which are machine learning models prepared for each of the states, in accordance with the state identified by the state information acquired in the state acquisition step; a detection process of detecting the vehicle included in the captured image by inputting the captured image to the first detection model acquired in the model acquisition process and identifying a target area indicating the vehicle in the captured image.

16. A method for detecting a vehicle included in a captured image, comprising: The vehicle travels through a factory where a plurality of manufacturing processes are performed to manufacture and ship the vehicle; The vehicle is classified into a plurality of states by having different appearances according to the plurality of manufacturing processes; The detection method includes: an image acquisition step of acquiring the captured image; a state acquiring step of acquiring state information indicating one of the states of the vehicle included in the captured image; a detection process for detecting the vehicle included in the captured image by inputting the captured image and the one state identified by the state information acquired in the state acquisition process into a second detection model, which is a machine learning model, and identifying a target area in the captured image that indicates the vehicle.

Images