Detection device, position calculation system, and detection method

Abstract

To provide a technique for accurately detecting a vehicle included in a captured image.SOLUTION: In a detecting apparatus configured to detect a vehicle in a captured image, the vehicle is configured to move in a factory in which manufacturing steps are performed to manufacture and ship vehicles. The vehicle is classified into multiple states in accordance with appearances of the vehicle, the appearances being different depending on the manufacturing steps. The detecting apparatus includes: an image acquisition unit configured to acquire a captured image; a state acquisition unit configured to acquire state information indicating one of the states of the vehicle included in the captured image; a model acquisition unit configured to acquire, from among first detection models that are machine learning models prepared for each of the states, a first detection model selected in accordance with the one state identified by the state information acquired by the state acquisition unit; and a detection unit configured to detect the vehicle included in the captured image by inputting the captured image to the first detection model acquired by the model acquisition unit and identifying a target region representing the vehicle in the captured image.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present disclosure relates to a detection device, a position calculation system, and a detection method. [Background technology]

[0002] BACKGROUND ART Conventionally, a vehicle that runs automatically under remote control has been known (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2017-538619 Summary of the Invention [Problem to be solved by the invention]

[0004] In order to remotely control a vehicle for autonomous driving, the vehicle may be externally captured. However, when the vehicle is traveling within a factory where the vehicle is manufactured, the vehicle's appearance may vary depending on the manufacturing process. For example, the appearance of a platform vehicle in which various devices such as wheels and a vehicle control device are mounted on a chassis is significantly different from that of a vehicle in which a body such as a body shell is mounted on the chassis. The inventors of the present application have discovered that such differences in the vehicle's appearance depending on the manufacturing process may reduce the accuracy of detecting the vehicle in the captured image. [Means for solving the problem]

[0005] The present disclosure can be realized in the following forms.

[0006] (1) According to a first aspect of the present disclosure, there is provided a detection device for detecting a vehicle included in a captured image. The vehicle travels through a factory where multiple manufacturing processes are performed to manufacture and ship the vehicle, and the vehicle is classified into multiple states due to different appearances resulting from the multiple manufacturing processes. The detection device includes: an image acquisition unit that acquires the captured image; a state acquisition unit that acquires state information indicating one of the states of the vehicle included in the captured image; a model acquisition unit that acquires, from multiple first detection models that are machine learning models prepared for each of the states, a first detection model selected according to the one state identified by the state information acquired by the state acquisition unit; and a detection unit that inputs the captured image into the first detection model acquired by the model acquisition unit and identifies a target area in the captured image that indicates the vehicle, thereby detecting the vehicle included in the captured image. According to this aspect, the detection device can acquire state information indicating the state of the vehicle included in the captured image and acquire, from multiple first detection models prepared for each state, a first detection model selected according to the one state identified by the acquired state information. Then, by inputting the captured image into a first detection model corresponding to the state of the vehicle included in the captured image, the vehicle included in the captured image can be detected with high accuracy. In other words, by using a first detection model suitable for detecting vehicles classified into one state, each area constituting the captured image can be classified with high accuracy into a target area and an area other than the target area. As a result, it is possible to prevent the detection accuracy of the vehicle included in the captured image from decreasing due to differences in appearance between multiple manufacturing processes. (2) According to a second aspect of the present disclosure, there is provided a detection device. a vehicle control device that controls the operation of the vehicle, and a vehicle communication unit that communicates with devices other than the vehicle itself; and a detection unit that detects a vehicle included in a captured image. The vehicle travels within a factory where multiple manufacturing processes are carried out to manufacture and ship the vehicle, and the vehicle is classified into multiple states because it has different appearances depending on the multiple manufacturing processes. The multiple states include a platform state in which the vehicle is in the form of a platform that includes at least wheels, a chassis, a drive unit that accelerates the vehicle, a steering unit that changes the direction of travel of the vehicle, a braking unit that decelerates the vehicle, a vehicle control device that controls the operation of the vehicle, and a vehicle communication unit that communicates with devices other than the vehicle itself. The detection device includes: an image acquisition unit that acquires the captured image; a state acquisition unit that acquires state information indicating one of the states of the vehicle included in the captured image; a model acquisition unit that acquires the first detection model selected from a plurality of first detection models that are machine learning models prepared for each of the states, in accordance with the one state identified by the state information acquired by the state acquisition unit; and a detection unit that detects the vehicle included in the captured image by inputting the captured image to the first detection model acquired by the model acquisition unit and identifying a target area in the captured image that indicates the vehicle. According to this aspect, when the vehicle is in the platform form, the state of the vehicle can be classified as a platform state. When detecting a vehicle classified as a platform state, the vehicle can be detected by using a first detection model suitable for detecting vehicles classified as a platform state. When detecting a vehicle classified as a platform state, the vehicle can be detected by inputting a captured image associated with the platform state into a second detection model. This improves the detection accuracy of vehicles classified as a platform state. (3) According to a third aspect of the present disclosure, a detection device is provided. The detection device detects a vehicle included in a captured image. The vehicle travels through a factory where multiple manufacturing processes are performed to manufacture and ship the vehicle. The vehicle has different appearances depending on the multiple manufacturing processes and is classified into multiple states. The detection device includes: an image acquisition unit that acquires the captured image; a state acquisition unit that acquires state information indicating one of the states of the vehicle included in the captured image; and a detection unit that inputs the captured image and the one state identified by the state information acquired by the state acquisition unit into a second detection model, which is a machine learning model, and identifies a target area indicating the vehicle in the captured image, thereby detecting the vehicle included in the captured image. According to this aspect, by inputting the captured image associated with the vehicle state into the second detection model, the vehicle included in the captured image can be accurately detected. In other words, each area constituting the captured image can be accurately classified into a target area and an area other than the target area. As a result, it is possible to prevent a decrease in detection accuracy of a vehicle included in a captured image due to differences in appearance between multiple manufacturing processes. (4) In the above aspect, each of the plurality of first detection models may be trained in advance to identify the target region by inputting a plurality of training images including M (M is an integer of 2 or more) first training images each including the vehicle classified into the one state, and N (N is an integer of 0 or more and less than M) second training images each including the vehicle classified into another state different from the one state. According to this aspect, by training more first training images each including the vehicle classified into one state than second training images each including the vehicle classified into another state different from the one state, it is possible to prepare a first detection model suitable for detecting a vehicle classified into one state for each state. (5) In the above aspect, each of the plurality of first detection models may be trained in advance to identify the target region by inputting a plurality of training images including M (M is an integer of 2 or more) first training images each including the vehicle classified into the one state, and N (N is an integer of 0 or more and less than M) second training images each including the vehicle classified into another state different from the one state. According to this aspect, by training more first training images each including vehicles classified into one state than second training images each including vehicles classified into another state different from the one state, it is possible to prepare a first detection model suitable for detecting vehicles classified into one state for each state. (6) In the above aspect, the second detection model may be trained in advance to identify the target region by inputting a plurality of training images including the vehicle and state answer labels associated with each of the plurality of training images, the state answer labels indicating the state of the vehicle included in the training images. According to this aspect, it is possible to prepare a second detection model trained by associating a plurality of training images including a vehicle with state answer labels indicating the state of the vehicle included in each training image. (7) In the above embodiment, a region correct answer label indicating whether each region in the training image is the target region or a non-target region indicating an area other than the vehicle may be associated with each region in the training image. According to this embodiment, each region constituting the captured image can be accurately classified into a target region and a non-target region. This can improve the accuracy of detecting vehicles included in the captured image. (8) In the above aspect, the status acquisition unit may acquire process information related to one of the manufacturing processes being performed on the vehicle, and acquire the status information by using a process database in which the status information is associated with each of the multiple manufacturing processes to identify the status information associated with the one manufacturing process identified by the acquired process information. According to this aspect, the process information related to the one manufacturing process being performed on the vehicle can be acquired. Then, by using the process database to identify the status information associated with the one manufacturing process identified by the acquired process information, status information indicating the status of the vehicle included in the captured image can be acquired. (9) In the above aspect, the status acquisition unit may acquire the status information by inputting the captured image to a specific model that is a machine learning model trained to output the status information when the captured image is input. According to this aspect, status information indicating the status of the vehicle included in the captured image can be acquired by inputting the captured image to the specific model trained to output status information about the vehicle included in the captured image. (10) In the above aspect, the status acquisition unit may acquire imaging information about an imaging device, and use a status database in which the imaging information and the status information are associated to identify the status information associated with one of the imaging devices identified by the acquired imaging information, thereby acquiring the status information. According to this aspect, imaging information about imaging devices can be acquired. Then, by using the status database in which imaging information and status information are associated to identify the status information associated with one of the imaging devices identified by the acquired imaging information, status information indicating the status of the vehicle included in the captured image can be acquired. (11) In the above aspect, the plurality of manufacturing processes may include a painting process for painting the vehicle, and the plurality of states may include (iii) an unpainted state indicating the state of the vehicle before painting in the painting process, and (iv) a painted state indicating the state of the vehicle after painting in the painting process. According to this aspect, the state of the vehicle can be classified into an unpainted state and a painted state before and after painting in the painting process. This makes it possible to prevent a decrease in the detection accuracy of the vehicle included in the captured image depending on the difference in the exterior color of the vehicle before and after performing the painting process. (12) In the above aspect, the image capturing device may further include a distortion correction unit that corrects distortion of the captured image. According to this aspect, distortion of the captured image can be corrected, thereby improving the detection accuracy of vehicles included in the captured image. (13) The above aspect may further include a rotation processing unit that rotates the captured image so that the moving direction of the vehicle faces a predetermined direction. According to this aspect, the captured image can be rotated so that the direction of a vector indicating the moving direction of the vehicle faces a predetermined direction. In this way, it is possible to detect a vehicle included in the captured image with the direction of the vector indicating the moving direction of the vehicle aligned. This improves the detection accuracy of a vehicle included in the captured image. (14) According to a fourth aspect of the present disclosure, there is provided a position calculation system. The position calculation system for calculating a position of a vehicle included in a captured image includes the detection device according to the above aspects and a position calculation device for calculating the position of the vehicle, wherein the detection device further includes a position calculation device for generating a first mask image by masking a target area, which is an area indicating the vehicle, in the captured image and adding a mask area to the target area, and for calculating the position of the vehicle using the first mask image, wherein the position calculation device includes a perspective transformation unit for generating a second mask image by performing perspective transformation on the first mask image, and a coordinate point calculation unit for calculating a coordinate point corresponding to a specified area of ​​a first circumscribing rectangle set in the mask area in the first mask image. The present invention further includes a coordinate point calculation unit that calculates an image coordinate point indicating the position of the vehicle in an image coordinate system by setting a vertex of a second circumscribing rectangle set in the mask area in the second mask image as a first coordinate point, setting a vertex of a second circumscribing rectangle set in the mask area in the second mask image that indicates the same position as the first coordinate point as a second coordinate point, and correcting the first coordinate point using the second coordinate point; and a position conversion unit that converts the image coordinate point into a vehicle coordinate point indicating the position of the vehicle in the global coordinate system using a distance from a reference point of the image capture device calculated based on the position of the image capture device in a global coordinate system and a distance from the reference point of a predetermined positioning point of the vehicle. According to this aspect, the position of the vehicle included in the captured image can be calculated. Furthermore, a first mask image in which a target area indicating the vehicle is masked out of each area constituting the captured image, and a second mask image in which the first mask image is subjected to perspective transformation can be generated. In this way, the image coordinate point can be calculated by calculating the first coordinate point from the first mask image and the second coordinate point from the second mask image. This allows for more accurate calculation of the image coordinate point. As a result, the accuracy of calculating the vehicle position can be improved. (15) According to a fifth aspect of the present disclosure, there is provided a detection method for detecting a vehicle included in a captured image. The vehicle travels in a factory where multiple manufacturing processes are performed to manufacture and ship the vehicle, and the vehicle is classified into multiple states due to different appearances resulting from the multiple manufacturing processes. The detection method includes an image acquisition step for acquiring the captured image, a state acquisition step for acquiring state information indicating one of the states of the vehicle included in the captured image, a model acquisition step for acquiring a first detection model selected from multiple first detection models, which are machine learning models prepared for each of the states, according to the one state identified by the state information acquired in the state acquisition step, and a detection step for inputting the captured image into the first detection model acquired in the model acquisition step and identifying a target area in the captured image that indicates the vehicle, thereby detecting the vehicle included in the captured image. According to this aspect, the state information indicating the state of the vehicle included in the captured image can be acquired, and the first detection model selected from multiple first detection models prepared for each state according to the one state identified by the acquired state information can be acquired. Then, by inputting the captured image into a first detection model corresponding to the state of the vehicle included in the captured image, the vehicle included in the captured image can be detected with high accuracy. In other words, by using a first detection model suitable for detecting vehicles classified into one state, each area constituting the captured image can be classified with high accuracy into a target area and an area other than the target area. As a result, it is possible to prevent the detection accuracy of the vehicle included in the captured image from decreasing due to differences in appearance between multiple manufacturing processes. (16) A sixth aspect of the present disclosure provides a detection method. The detection method detects a vehicle included in a captured image. The vehicle travels through a factory where multiple manufacturing processes are performed to manufacture and ship the vehicle. The vehicle has different appearances depending on the multiple manufacturing processes and is classified into multiple states. The detection method includes: an image acquisition step of acquiring the captured image; a state acquisition step of acquiring state information indicating one of the states of the vehicle included in the captured image; and a detection step of inputting the captured image and the one state identified by the state information acquired in the state acquisition step into a second detection model, which is a machine learning model, and identifying a target area indicating the vehicle in the captured image, thereby detecting the vehicle included in the captured image. According to this aspect, by inputting the captured image associated with the vehicle state into the second detection model, the vehicle included in the captured image can be accurately detected. In other words, each area constituting the captured image can be accurately classified into a target area and an area other than the target area. As a result, it is possible to prevent a decrease in detection accuracy of the vehicle included in the captured image due to differences in appearance between multiple manufacturing processes. The present disclosure can be realized in various forms other than the above-described detection device, position calculation system, and detection method, for example, a method for manufacturing a detection device and a position calculation system, a position calculation method, a control method for a detection device and a position calculation system, a computer program for implementing the control method, a non-transitory recording medium on which the computer program is recorded, etc. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram showing the configuration of a position calculation system. [Figure 2] 2A to 2C are diagrams showing a manufacturing process of a vehicle according to the first embodiment. [Figure 3] FIG. 1 is a diagram showing the configuration of a detection device according to a first embodiment. [Figure 4] FIG. 2 is a diagram showing details of a CPU mounted on the detection device according to the first embodiment. [Figure 5] FIG. 1 is a diagram showing the configuration of a position calculation device. [Figure 6] FIG. 2 is a diagram showing the configuration of a remote control device. [Figure 7] 4 is a flowchart showing a vehicle detection method and a position calculation method in the first embodiment. [Figure 8] Schematic diagrams showing examples of various images. [Figure 9] FIG. 10 is a diagram for explaining details of a coordinate point calculation step. [Figure 10] FIG. 10 is a diagram for explaining a method of calculating a base coordinate point. [Figure 11] FIG. 1 is a diagram for explaining the details of a position conversion process. [Figure 12] FIG. 2 is a diagram for explaining the details of the position conversion process. [Figure 13] 4 is a flowchart showing a vehicle operation control method. [Figure 14] 8A to 8C are diagrams showing a manufacturing process of a vehicle according to a second embodiment. [Figure 15] FIG. 10 is a diagram showing the configuration of a detection device according to a second embodiment. [Figure 16] FIG. 10 is a diagram showing the configuration of a detection device according to a third embodiment. [Figure 17] FIG. 10 is a diagram showing details of a CPU mounted on a detection device according to a third embodiment. [Figure 18] 10 is a flowchart showing a vehicle detection method and a position calculation method according to a third embodiment. [Figure 19] FIG. 10 is a diagram showing the configuration of a detection device according to a fourth embodiment. [Figure 20] FIG. 10 is a diagram showing details of a CPU mounted on a detection device according to a fourth embodiment. [Figure 21] FIG. 10 is a diagram showing the configuration of a detection device according to a fifth embodiment. [Figure 22] FIG. 13 is a diagram showing details of a CPU mounted on a detection device according to a fifth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] A. First embodiment: A-1. Position calculation system configuration: FIG. 1 is a diagram showing the configuration of a position calculation system 1. The position calculation system 1 is a system that detects a vehicle 10 included in a captured image and calculates the position of the vehicle 10. In the present disclosure, the captured image is at least one of an original image as raw data acquired by an imaging device 9, which is an external device installed in a location different from the vehicle 10, and a corrected image, a rotated image, or a processed image obtained by processing the original image. Details of the corrected image, the rotated image, and the processed image will be described later. The position calculation system 1 includes one or more vehicles 10, a detection device 5 that detects the vehicle 10 included in the captured image, and a position calculation device 6 that calculates the position of the vehicle 10 using the detection result by the detection device 5.

[0009] The imaging device 9 captures an image of the imaging range RG, including the vehicle 10 whose position is to be calculated, from outside the vehicle 10, thereby acquiring an original image. In this embodiment, the imaging device 9 transmits the acquired original image to the detection device 5 along with camera identification information and the acquisition time of the original image. The camera identification information is a unique ID (identifier) ​​assigned to each imaging device 9 so that the images are unique among the imaging devices 9. The original image is a two-dimensional image composed of pixels arranged on the XcYc plane of a camera coordinate system. The camera coordinate system has its origin at the focal point of the imaging device 9 and coordinate axes represented by the Xc axis and the Yc axis perpendicular to the Xc axis. The original image includes at least two-dimensional data of the vehicle 10 whose position is to be calculated. The original image is preferably a color image, but may also be a grayscale image. The imaging device 9 is a camera having an imaging element, such as a CCD image sensor or a CMOS image sensor, and an optical system.

[0010] In this embodiment, the imaging device 9 acquires an original image of the track 2 and the vehicle 10 traveling on the track 2, viewed from above. In order to capture an image of the entire track 2 using one or more imaging devices 9, the installation positions and number of imaging devices 9 are determined taking into consideration the imaging range RG (angle of view) of the imaging device 9, etc. Specifically, each imaging device 9 is installed so that a first imaging range RG1, which is the imaging range RG of the adjacent first imaging device 901, and a second imaging range RG2, which is the imaging range RG of the adjacent second imaging device 902, overlap. Furthermore, each imaging device 9 is installed in a position where it can capture an image of a predetermined positioning point 10e for a specific portion of the vehicle 10 traveling on the track 2. In this embodiment, the positioning point 10e is the rear end of the left side of the vehicle 10 (hereinafter referred to as the left rear end). Note that the positioning point 10e may be at a point other than the left rear end of the vehicle 10. Furthermore, the imaging device 9 is not limited to acquiring information from above the vehicle 10, but may also acquire information from the front, rear, sides, etc. of the vehicle 10.

[0011] The vehicle 10 may be, for example, an electric vehicle, a hybrid vehicle, a fuel cell vehicle, a gasoline vehicle, or a diesel vehicle. The vehicle 10 may be a private vehicle such as a passenger car, or may be a commercial vehicle such as a truck, a bus, or a construction vehicle.

[0012] The vehicle 10 has a manned driving mode and a remote unmanned driving mode. In the manned driving mode, the driver in the vehicle 10 operates input devices such as a steering wheel and an accelerator provided on the vehicle 10 to generate driving conditions for the vehicle 10. As a result, the vehicle 10 drives in accordance with the generated driving conditions. The driving conditions are conditions that define the driving behavior of the vehicle 10. The driving conditions include, for example, the driving route, position, driving speed, acceleration, and steering angle of the wheels 170 of the vehicle 10. The remote unmanned driving mode has a remote manual driving mode and a remote automatic driving mode. In the remote manual driving mode, the operator operates an operator input device provided in a location different from the vehicle 10 to generate driving conditions for the vehicle 10. As a result, the vehicle 10 receives the driving conditions generated by the operator input device and drives in accordance with the received driving conditions. In the remote automatic driving mode, a remote control device 7 such as a server provided in a location different from the vehicle 10 generates control values ​​that define the driving behavior of the vehicle 10 and transmits them to the vehicle 10. As a result, the vehicle 10 receives the control values ​​and performs automatic driving in accordance with the received control values.

[0013] In this embodiment, the vehicle 10 travels in a remote automated driving mode within a factory where multiple manufacturing processes are performed to manufacture and ship the vehicle 10. The factory is not limited to being located within a single building, or at a single site or address, but may be located across multiple buildings, sites, addresses, etc. The factory may also include a storage area such as a yard where the vehicle 10, as a finished product manufactured by performing multiple manufacturing processes, is stored before being shipped. In this case, the vehicle 10 may travel on public roads, not just private roads.

[0014] Fig. 2 is a diagram showing the manufacturing process of the vehicle 10 in the first embodiment. Fig. 2 shows a representative portion of a plurality of manufacturing steps executed in the manufacturing process of the vehicle 10. In the example shown in Fig. 2, the finished vehicle 10 is manufactured by assembling a painted body shell 180 to a vehicle 101 in the form of a platform (hereinafter referred to as platform vehicle 101), and then assembling interior parts such as seats and exterior parts 190 such as door panels.

[0015] The platform manufacturing process is a manufacturing process for manufacturing a platform vehicle 101. The platform vehicle 101 is a vehicle 10 that can perform at least three functions, "running," "turning," and "stopping," by remote control using a remote control device 7. Specifically, the platform vehicle 101 includes at least a drive unit 110, a steering unit 120, a braking unit 130, a vehicle communication unit 140, a vehicle control device 150, a chassis 160, and wheels 170. The drive unit 110 accelerates the vehicle 10. The drive unit 110 includes at least a drive power source and a power supply unit that supplies power to the drive power source. If the vehicle 10 is an electric vehicle, the drive power source is a motor, and the power supply unit is a battery such as a lithium-ion battery. The steering unit 120 changes the traveling direction of the vehicle 10. The braking unit 130 decelerates the vehicle 10. The vehicle communication unit 140 communicates with external devices using wireless communication or the like. The external devices include other devices other than the host vehicle 10, such as the detection device 5, position calculation device 6, remote control device 7, and imaging device 9, as well as other vehicles 10. The vehicle communication unit 140 is, for example, a wireless communication device. The vehicle communication unit 140 communicates with external devices connected to the network Nt, for example, via an access point in a factory. The vehicle control device 150 includes a CPU, a storage unit, and an input / output interface. In the vehicle control device 150, the CPU, storage unit, and input / output interface are connected to each other, for example, via an internal bus or an interface circuit. The input / output interface communicates with internal devices such as the drive unit 110 mounted on the host vehicle 10. The input / output interface is communicatively connected to the vehicle communication unit 140. The chassis 160 is a chassis portion that supports the various devices 110 to 150 mounted on the vehicle 10, as well as a body shell 180 and various parts 190 that are assembled in a later manufacturing process. Note that the configuration of the vehicle 10, which is in the form of a platform, is not limited to the above.

[0016] The body shell assembly process is a manufacturing process in which a body shell 180 is assembled to a platform vehicle 101. The platform vehicle 101 and a vehicle 102 in which the body shell 180 is assembled to the platform vehicle 101 (hereinafter referred to as the first assembled vehicle 102) have different exterior shapes, and therefore the appearance of the vehicle 10 differs.

[0017] The part assembly process is a process of assembling exterior parts 190. The first assembly vehicle 102 and a vehicle 103 (hereinafter referred to as the second assembly vehicle 103) on which the exterior part 190 has been assembled to the first assembly vehicle 102 have different exterior shapes, and therefore the appearance of the vehicle 10 is different. Note that at least some of the interior parts may be assembled to the platform vehicle 101 before the body shell assembly process is performed, or may be assembled to the platform vehicle 101 either before or after the part assembly process is performed.

[0018] As described above, the vehicle 10 is classified into a plurality of states due to the different appearances that result from the plurality of manufacturing processes. In this case, the difference in appearance between the platform vehicle 101 and the first assembly vehicle 102 is smaller than, for example, the difference in appearance between the first assembly vehicle 102 and the second assembly vehicle 103. Therefore, in this embodiment, a state in which the vehicle 10 is in the form of a platform, such as the platform vehicle 101, is referred to as a "platform state." A state in which at least the body shell 180 is assembled to the platform vehicle 101, such as the first assembly vehicle 102 and the second assembly vehicle 103, is referred to as an "assembled state." In other words, in this embodiment, the vehicle 10 is classified into a plurality of states depending on the difference in the exterior shape of the vehicle 10. In this case, depending on the magnitude of the difference in the exterior shape of the vehicle 10, the state of the vehicle 10 in one manufacturing process may be classified as one state, or the states of the vehicle 10 in multiple manufacturing processes may be collectively classified as one state. The number and types of manufacturing processes in the manufacturing process of the vehicle 10, and the number and types of states of the vehicle 10 classified according to the manufacturing processes are not limited to those described above.

[0019] In this disclosure, vehicle 10 is at least one of vehicle 10 as a finished product and vehicles 101-105 as semi-finished products and work-in-progress in each manufacturing process of manufacturing the finished product. Hereinafter, when there is no need to distinguish between vehicles 101-105 in multiple states, they will be simply referred to as "vehicle 10."

[0020] 3 is a diagram showing the configuration of the detection device 5 in the first embodiment. The detection device 5 detects at least the outer shape (outline) of the vehicle 10 included in the captured image by identifying a target area, which is an area indicating the vehicle 10, from among the areas in the captured image. The detection device 5 includes a communication unit 51, a storage unit 53, and a CPU 52. In the detection device 5, the communication unit 51, the storage unit 53, and the CPU 52 are connected to each other via, for example, an internal bus and an interface circuit.

[0021] The communication unit 51 of the detection device 5 communicably connects the detection device 5 to other devices such as the vehicle control device 150, the position calculation device 6, the remote control device 7, and the imaging device 9. The communication unit 51 of the detection device 5 is, for example, a wireless communication device.

[0022] The storage unit 53 of the detection device 5 stores various information including various programs for controlling the operation of the detection device 5, a first detection model Md1 as a detection model, a process database Db1, and distortion correction parameters Pa1. The storage unit 53 of the detection device 5 includes, for example, a RAM, a ROM, and a hard disk drive (HDD).

[0023] The detection model is a trained machine learning model used to detect the vehicle 10 included in the captured image. In this embodiment, the storage unit 53 of the detection device 5 stores, as the detection model, a plurality of first detection models Md1 prepared for each state.

[0024] The first detection model Md1 is a machine learning model trained by inputting a first teacher dataset. The first teacher dataset is prepared for each state. The first teacher dataset includes a plurality of training images including the vehicle 10 and region correct labels associated with a plurality of regions constituting each training image. The region correct labels are correct labels indicating whether each region in the training image is a target region indicating the vehicle 10 or a non-target region indicating a region other than the vehicle 10. In this embodiment, each region constituting the training image is a single pixel constituting the training image. Note that each region constituting the training image may be multiple pixels constituting the training image. Each region constituting the training image is classified as either a target region or a non-target region based on, for example, a calculation result obtained by comparing the RGB values ​​of adjacent regions constituting the training image. Each region constituting the training image may be classified as either a target region or a non-target region based on the calculated probability by calculating the probability that the region is a target region. When a region correct label is assigned to each region constituting the training image, information indicating the position of the region to which the region correct label is assigned in the training image is associated with the region. When each region constituting the training image is one pixel, the information indicating the position in the training image is, for example, pixel coordinates indicating the pixel position in the training image.

[0025] The first teacher dataset includes a plurality of training images, namely, M (M is an integer equal to or greater than 2) first training images and N (N is an integer equal to or greater than 0 and less than M) second training images. The first training images are images including vehicles 10 classified into one state. The second training images are images including vehicles 10 classified into another state different from the one state. The number M of first training images included in the first teacher dataset may be, for example, 1.5 times or more the number N of second training images, or may be two or more times the number N of second training images. Furthermore, when there are multiple other states different from the one state, the number of first training images included in the first teacher dataset is, for example, greater than the total number of second training images including vehicles 10 classified into multiple other states different from the one state. In other words, the first detection model Md1 is a machine learning model that preferentially learns the feature quantities of vehicles 10 classified into one state by training more first training images than second training images.

[0026] In the present embodiment, the storage unit 53 of the detection device 5 stores two first detection models Md1a and Md1b suitable for detecting one state determined by the external shape of the vehicle 10. The first first detection model Md1a is a first detection model Md1 that has been trained to train more vehicles 10 classified as a "platform state" than vehicles 10 classified as states other than the "platform state" (for example, "assembled state"). The first first detection model Md1a is used when detecting vehicles 10 classified as a "platform state". The second first detection model Md1b is a first detection model Md1 that has been trained to train more vehicles 10 classified as a "assembled state" than vehicles 10 classified as states other than the "assembled state" (for example, "platform state"). The second first detection model Md1b is used when detecting vehicles 10 classified as a "assembled state". Note that the number and types of first detection models Md1 are not limited to those described above.

[0027] In this embodiment, when a captured image is input, the first detection model Md1 identifies a target region among the regions constituting the input captured image, and then generates a first mask image in which the target region is added with a mask region by masking the target region.

[0028] The algorithm of the first detection model Md1 may be, for example, a deep neural network (hereinafter, DNN) having the structure of a convolutional neural network (hereinafter, CNN) that realizes semantic segmentation and instance segmentation. Note that the configuration of the first detection model Md1 is not limited to the above. The first detection model Md1 may be, for example, a trained machine learning model that uses an algorithm other than a neural network.

[0029] The process database Db1 is a database in which status information is associated with each of a plurality of manufacturing processes. The status information is information that indicates one status of the vehicle 10. In other words, the status information is information that identifies a plurality of statuses in the manufacturing process of the vehicle 10. Note that the configuration of the process database Db1 is not limited to the above.

[0030] The distortion correction parameter Pa1 is a parameter used to correct distortion in a captured image. The distortion correction parameter Pa1 will be described in detail later.

[0031] 4 is a diagram illustrating details of the CPU 52 mounted on the detection device 5 in the first embodiment. The CPU 52 of the detection device 5 executes various programs stored in the storage unit 53 of the detection device 5, thereby functioning as an image acquisition unit 521, a distortion correction unit 522, a rotation processing unit 523, a crop processing unit 524, a state acquisition unit 525, and a model acquisition unit 526. Furthermore, the CPU 52 of the detection device 5 executes various programs stored in the storage unit 53 of the detection device 5, thereby functioning as a detection unit 527 and a first transmission unit 528.

[0032] The image acquisition unit 521 acquires the original image from the imaging device 9 .

[0033] The distortion correction unit 522 corrects distortion in the captured image. In this embodiment, the distortion correction unit 522 generates a corrected image by correcting the distortion of the original image.

[0034] The rotation processing unit 523 rotates the captured image so that the direction of a vector indicating the movement direction of the vehicle 10 (hereinafter referred to as a movement vector) faces a predetermined direction. In this embodiment, the rotation processing unit 523 generates a rotated image by rotating the corrected image.

[0035] The crop processing unit 524 deletes from the captured image, among the regions constituting the captured image, regions other than the region consisting of the vehicle 10 and the region surrounding the vehicle 10 (hereinafter referred to as the necessary region) (hereinafter referred to as the unnecessary region). In this way, the crop processing unit 524 cuts out the necessary region from the captured image. In this embodiment, when the vehicle 10 has moved a distance exceeding a predetermined threshold, the crop processing unit 524 deletes from the rotated image the moved region that corresponds to the distance traveled by the vehicle 10 and that is an unnecessary region. In this way, the crop processing unit 524 generates a processed image in which the unmoved region that includes the vehicle 10 and that is a necessary region has been cut out from the rotated image.

[0036] The status acquisition unit 525 acquires process information related to a manufacturing process being performed on the vehicle 10 included in the captured image and status information indicating a status of the vehicle 10 included in the captured image. The process information is, for example, a unique ID (identifier) ​​assigned to each manufacturing process in order to identify multiple manufacturing processes included in the manufacturing process of the vehicle 10. The status acquisition unit 525, for example, uses information acquired by a sensor mounted on the vehicle 10 (hereinafter referred to as an on-board sensor) to detect feature points that can identify the multiple manufacturing processes, thereby identifying the single manufacturing process being performed on the vehicle 10 included in the captured image. The on-board sensor is, for example, any of an on-board camera, an on-board lidar, and an on-board radar. The on-board camera captures images of the status of the area surrounding the vehicle 10. The on-board lidar and the on-board radar detect objects present in the area surrounding the vehicle 10. The status acquisition unit 525 then acquires status information about the vehicle 10 included in the captured image by identifying status information associated with the single manufacturing process identified by the acquired process information using the process database Db1. The method of acquiring the process information and the state information is not limited to the above.

[0037] The model acquisition unit 526 acquires a first detection model Md1 selected according to one state identified by the state information acquired by the state acquisition unit 525. In the present embodiment, the model acquisition unit 526 selects and acquires a first detection model Md1 according to one state identified by the state information acquired by the state acquisition unit 525 from a plurality of first detection models Md1 prepared for each state stored in the storage unit 53 of the detection device 5. In other words, the model acquisition unit 526 selects a first detection model Md1 that has been trained with more feature amounts of a vehicle 10 classified into one state identified by the state information acquired by the state acquisition unit 525 than feature amounts of vehicles 10 classified into other states different from the one state. For example, if the state information acquired by the state acquisition unit 525 identifies the state of the vehicle 10 included in the captured image as a "platform state," the model acquisition unit 526 acquires a first detection model Md1a that has been trained with more feature amounts of a vehicle 10 classified into the "platform state" than vehicles 10 classified into other states.

[0038] The detection unit 527 detects the vehicle 10 included in the captured image by inputting the captured image to the first detection model Md1 acquired by the model acquisition unit 526 and identifying a target region among the regions constituting the captured image. In this embodiment, the detection unit 527 inputs the processed image to the first detection model Md1 selected by the model acquisition unit 526. As a result, the detection unit 527 identifies a target region among the regions constituting the processed image, thereby generating a first mask image from the processed image.

[0039] The first transmission unit 528 transmits various types of information to devices other than the detection device 5. For example, the first transmission unit 528 transmits the detection result by the detection unit 527 to the position calculation device 6. In this embodiment, the detection result by the detection unit 527 is a first mask image. Note that at least a part of the functions of the detection device 5 may be realized as one function of any of the vehicle control device 150, the position calculation device 6, the remote control device 7, and the imaging device 9.

[0040] FIG. 5 is a diagram showing the configuration of the position calculation device 6. The position calculation device 6 calculates the position of the vehicle 10 included in the captured image using the detection result by the detection device 5. The position calculation device 6 uses the positioning point 10e of the vehicle 10 as the position of the vehicle 10. In this embodiment, the position calculation device 6 calculates the position of the vehicle 10 included in the captured image using the first mask image. The position calculation device 6 includes a communication unit 61, a storage unit 63, and a CPU 62. In the position calculation device 6, the communication unit 61, the storage unit 63, and the CPU 62 are connected to each other via, for example, an internal bus and an interface circuit.

[0041] The communication unit 61 of the position calculation device 6 communicably connects the vehicle control device 150, the detection device 5, the remote control device 7, and the imaging device 9 to the position calculation device 6. The communication unit 61 of the position calculation device 6 is, for example, a wireless communication device.

[0042] The storage unit 63 of the position calculation device 6 stores various information including various programs for controlling the operation of the position calculation device 6, perspective transformation parameters Pa2, and a camera database Db2. The storage unit 63 of the position calculation device 6 includes, for example, a RAM, a ROM, and a hard disk drive (HDD).

[0043] The perspective transformation parameter Pa2 is a parameter used when performing perspective transformation on the first mask image. The perspective transformation parameter Pa2 will be described in detail later.

[0044] The camera database Db2 is a database that indicates, for each imaging device 9, imaging parameters calculated based on the installation position of the imaging device 9 in a global coordinate system. The imaging parameters are parameters related to the distance of the imaging device 9 from a predetermined reference point. In this embodiment, the imaging parameter is the height H of the imaging device 9 from the road surface 20 (see FIG. 11 described below), that is, the height H relative to the road surface 20 on which the vehicle 10 is located. The camera database Db2 is a database that associates, for example, camera identification information with imaging parameters for the imaging device 9 identified by the camera identification information. Note that the configuration of the camera database Db2 is not limited to the above. The camera identification information may also be associated with the imaging range RG, installation position, installation angle, etc. of the imaging device 9.

[0045] The CPU 62 of the position calculation device 6 functions as a data acquisition unit 621, a perspective transformation unit 622, a coordinate point calculation unit 623, a position transformation unit 624, and a second transmission unit 625 by deploying various programs stored in the memory unit 63 of the position calculation device 6.

[0046] The data acquisition unit 621 acquires various types of information. In this embodiment, the data acquisition unit 621 acquires a first mask image from the detection device 5. Furthermore, the data acquisition unit 621 refers to the camera database Db2 stored in the storage unit 63 of the position calculation device 6 to acquire imaging parameters for the imaging device 9 that is the source of the original image to be analyzed.

[0047] The perspective transformation unit 622 generates a second mask image by performing perspective transformation on the first mask image. The coordinate point calculation unit 623 corrects the first coordinate point using the second coordinate point to calculate an image coordinate point. The first coordinate point is a coordinate point in the image coordinate system of a specified vertex of a first circumscribing rectangle set in the mask area in the first mask image. The second coordinate point is a coordinate point in the image coordinate system that indicates the same position as the first coordinate point among the vertices of the second circumscribing rectangle set in the mask area in the second mask image. The image coordinate point is a coordinate point that indicates the position of the vehicle 10 in the image coordinate system. The position transformation unit 624 converts the image coordinate point into a vehicle coordinate point using the imaging parameters acquired by the data acquisition unit 621 and the image coordinate point calculated by the coordinate point calculation unit 623. The vehicle coordinate point is a coordinate point that indicates the position of the vehicle 10 in the global coordinate system.

[0048] The second transmission unit 625 transmits various types of information to devices other than the position calculation device 6. For example, the second transmission unit 625 transmits vehicle coordinate points to the remote control device 7 as information indicating the position of the vehicle 10. Note that at least a part of the functions of the position calculation device 6 may be realized as one function of any of the vehicle control device 150, the detection device 5, the remote control device 7, and the imaging device 9.

[0049] FIG. 6 is a diagram showing the configuration of the remote control device 7. The remote control device 7 uses the position information of the vehicle 10, etc., to create control values ​​that define the driving operation of the vehicle 10 and transmit them to the vehicle 10. In this way, the remote control device 7 remotely controls the operation of the vehicle 10. The remote control device 7 includes a communication unit 71, a storage unit 73, and a CPU 72. The communication unit 71, storage unit 73, and CPU 72 of the remote control device 7 are connected to each other, for example, via an internal bus and an interface circuit. The communication unit 71 of the remote control device 7 communicatively connects the vehicle control device 150, the detection device 5, the position calculation device 6, and the imaging device 9 to the remote control device 7. The communication unit 71 of the remote control device 7 is, for example, a wireless communication device. The storage unit 73 of the remote control device 7 stores various programs that control the operation of the remote control device 7. The storage unit 73 of the remote control device 7 includes, for example, a RAM, a ROM, and a hard disk drive (HDD).

[0050] The CPU 72 of the remote control device 7 functions as an information acquisition unit 721, a control value creation unit 722, and a third transmission unit 723 by executing various programs stored in the storage unit 73 of the remote control device 7.

[0051] The information acquisition unit 721 acquires various types of information. For example, the information acquisition unit 721 acquires information relating to the driving conditions of the vehicle 10 (hereinafter referred to as driving information). The driving information includes, for example, vehicle coordinate points transmitted from the remote control device 7 as information indicating the position of the vehicle 10, the driving speed and actual steering angle of the vehicle 10 transmitted from the vehicle control device 150, and driving route information pre-stored in the storage unit 73 of the remote control device 7. The driving route information is information indicating a target driving route for the vehicle 10 traveling in remote automatic driving mode. Note that the types of information included in the driving information are not limited to those described above.

[0052] The control value creation unit 722 creates control values ​​that define the driving behavior of the vehicle 10 using the driving information acquired by the information acquisition unit 721. Specifically, the control value creation unit 722 creates, for example, a reference control value and a correction control value. The reference control value is a control value for driving the vehicle 10 along a target driving route. The correction control value is a control value that corrects the relative position of the vehicle 10 with respect to the target driving route. The reference control value and the correction control value each include, for example, an acceleration control value that defines the acceleration in the forward direction of the vehicle 10 and a steering angle control value that defines the steering angle of the vehicle 10. Note that the reference control value and the correction control value may each include either a trajectory control value or a destination control value instead of the acceleration control value and the steering angle control value. The trajectory control value is a control value that defines the driving trajectory of the vehicle 10 by chronologically arranging target driving positions of the vehicle 10 for each predetermined time. The destination control value is a control value that indicates a target arrival time of the vehicle 10 at a target destination.

[0053] The third transmission unit 723 transmits various types of information to devices other than the remote control device 7. For example, the third transmission unit 723 transmits the control value created by the control value creation unit 722 to the vehicle 10 to be controlled. Note that at least some of the functions of the remote control device 7 may be realized as one function of any of the vehicle control device 150, the detection device 5, the position calculation device 6, and the imaging device 9.

[0054] A-2. Vehicle detection and position calculation methods: Fig. 7 is a flowchart showing a method for detecting a vehicle 10 and a method for calculating a position of the vehicle 10 in the first embodiment. The method shown in Fig. 7 is executed repeatedly at predetermined time intervals, for example, after the vehicle 10 to be detected starts traveling in the remote autonomous driving mode.

[0055] FIG. 8 is a schematic diagram showing examples of various images obtained when the detection method and position calculation method shown in FIG. 7 are executed. In FIG. 8, step numbers corresponding to the steps in FIG. 7 are assigned. In this embodiment, an example will be described in which the vehicle 10 travels (moves) along the traveling direction on a road surface 20 on which mesh-like grid lines 21 are drawn along an Xg axis parallel to the traveling direction of the vehicle 10 and a Yg axis perpendicular to the Xg axis. The Xg axis and the Yg axis are each coordinate axes in a global coordinate system. In other embodiments, the grid lines 21 may be omitted.

[0056] 7, first, an image acquisition step (step S1) is executed. As shown in FIG. 8, the image acquisition step is a step of acquiring an original image Im1 including the vehicle 10. In this embodiment, in the image acquisition step, the image acquisition unit 521 acquires the original image Im1 from the imaging device 9.

[0057] As shown in FIG. 7, after the image acquisition step, a distortion correction step (step S2) is executed. The distortion correction step is a step of correcting distortion of a captured image. In this embodiment, as shown in FIG. 8, in the distortion correction step, the distortion correction unit 522 corrects distortion of the original image Im1 to generate a corrected image Im2. Specifically, the distortion correction unit 522 corrects distortion of the original image Im1 using, for example, a distortion correction parameter Pa1 stored in advance in the storage unit 53 of the detection device 5. The distortion correction parameter Pa1 is, for example, a parameter related to position information of the grid lines 21 obtained by calibration. Note that the distortion correction method is not limited to the above. The distortion correction parameter Pa1 may be any parameter other than those described above.

[0058] As shown in FIG. 7, a rotation processing step (step S3) is executed after the distortion correction step. The rotation processing step is a step of rotating the captured image so that the direction of the movement vector of the vehicle 10 included in the captured image faces a predetermined direction. In this embodiment, as shown in FIG. 8, in the rotation processing step, the rotation processing unit 523 rotates the corrected image Im2 so that the direction of the movement vector for the vehicle 10 included in the corrected image Im2 faces a predetermined direction. In this way, the rotation processing unit 523 generates a rotated image Im3. Specifically, the rotation processing unit 523 rotates the corrected image Im2 around the center of gravity of the vehicle 10 in the corrected image Im2 as the center of rotation so that the direction of the movement vector of the vehicle 10 faces upward on the screen of a display device that displays the corrected image Im2. The movement of a feature point (e.g., the center of gravity) of the vehicle 10 can be represented as the direction of the movement vector by, for example, an optical flow method. The amount and direction of the movement vector of the vehicle 10 are estimated based on, for example, the change in position of a feature point appropriately set on the corrected image Im2 between image frames. Note that the rotation processing method is not limited to the above.

[0059] As shown in FIG. 7, a cropping process (step S4) is executed after the rotation process. The cropping process is a process of cutting out a necessary area from a captured image. In this embodiment, as shown in FIG. 8, in the cropping process, if the vehicle 10 has moved a distance exceeding a predetermined threshold, the cropping processor 524 deletes a moved area A2 corresponding to the distance moved by the vehicle 10 from the rotated image Im3 as an unnecessary area. As a result, the cropping processor 524 generates a processed image Im4 by cutting out an unmoved area A1 including the vehicle 10 from the rotated image Im3 as a necessary area. At this time, the cropping processor 524 estimates the moved area A2 by, for example, recognizing the distance moved by the vehicle 10 from the estimated movement vector amount of the vehicle 10. Note that, in the method for detecting the vehicle 10, either the rotation process or the cropping process may be executed first. Furthermore, the cropping method is not limited to the above.

[0060] As shown in FIG. 7, after the cropping process, a status acquisition process (step S5) is executed. The status acquisition process is a process of acquiring status information indicating one status of the vehicle 10 included in the captured image. In this embodiment, in the status acquisition process, the status acquisition unit 525 first acquires process information related to one manufacturing process being performed on the vehicle 10 included in the captured image. Then, the status acquisition unit 525 uses the process database Db1 to identify status information associated with the one manufacturing process identified by the acquired process information. In this way, the status acquisition unit 525 acquires status information about the vehicle 10 included in the captured image. Note that the status acquisition process may be executed at any point after the image acquisition process is completed and before the detection process (step S7) is started.

[0061] After the state acquisition step, a model acquisition step (step S6) is executed. The model acquisition step is a step of acquiring a first detection model Md1 selected from a plurality of first detection models Md1 prepared for each state in accordance with the state identified by the state information acquired in the state acquisition step. In this embodiment, in the model acquisition step, the model acquisition unit 526 selects and acquires a first detection model Md1 corresponding to the state identified by the state information acquired in the state acquisition step from a plurality of first detection models Md1 pre-stored in the storage unit 53 of the detection device 5. Note that the model acquisition step may be executed at any point after the state acquisition step is completed and before the detection step is started.

[0062] The detection step is executed after the model acquisition step. As shown in FIG. 8, the detection step is a step of detecting the outline of the vehicle 10 included in the captured image using a detection model such as the first detection model Md1. In this embodiment, in the detection step, the detection unit 527 inputs the processed image Im4 to the first detection model Md1 selected in the model acquisition step. As a result, the detection unit 527 detects the outline of the vehicle 10 included in the processed image Im4. Then, the detection unit 527 masks the target area in the processed image Im4 to generate a first mask image Im5 in which a mask area Ms is added to the target area. The first transmission unit 528 transmits the first mask image Im5 to the position calculation device 6. Note that the method of detecting the vehicle 10 is not limited to the above.

[0063] As shown in FIG. 7, after the detection step, a perspective transformation step (step S8) is executed. The perspective transformation step is a step of performing perspective transformation on a first mask image Im5. In the perspective transformation step, the data acquisition unit 621 acquires the first mask image Im5 from the detection device 5. Then, the perspective transformation unit 622 performs perspective transformation on the first mask image Im5 to generate a second mask image Im6. Specifically, the perspective transformation unit 622 performs perspective transformation on the first mask image Im5 into a bird's-eye view image viewed from a viewpoint above the vehicle 10 (for example, directly above the vehicle 10) that is substantially perpendicular to the road surface 20, using, for example, a perspective transformation parameter Pa2 pre-stored in the storage unit 63 of the position calculation device 6. The perspective transformation parameter Pa2 is, for example, a parameter related to position information and internal parameters of the image capture device 9 obtained by calibration. As a result, the perspective transformation unit 622 generates a second mask image Im6 expressed in the image coordinate system from the first mask image Im5 expressed in the camera coordinate system. The image coordinate system has its origin at a point on the image plane projected by perspective transformation, and has coordinate axes indicated by the Xi axis and the Yi axis perpendicular to the Xi axis. Note that the perspective transformation method is not limited to the above. The perspective transformation parameter Pa2 may be any parameter other than those described above.

[0064] As shown in Fig. 7, after the perspective transformation step, a coordinate point calculation step (step S9) is executed. Fig. 9 is a diagram for explaining the details of the coordinate point calculation step. The coordinate point calculation step is a step for calculating an image coordinate point P3 that indicates the position of the vehicle 10 in the image coordinate system.

[0065] In the coordinate point calculation step, the coordinate point calculation unit 623 calculates a base coordinate point P0 from a first circumscribing rectangle R1 set in a mask area Ms in a first mask image Im5, which is an image before perspective transformation. FIG. 10 is a diagram for explaining a method for calculating the base coordinate point P0. To calculate the base coordinate point P0, the coordinate point calculation unit 623 sets a base circumscribing rectangle R0 for the mask area Ms in the first mask image Im5. Next, the coordinate point calculation unit 623 rotates the first mask image Im5 by a required amount around the center of gravity C of the mask area Ms as the rotation center so that the direction of the movement vector V of the vehicle 10 corresponding to the mask area Ms in the first mask image Im5 faces a predetermined direction. The predetermined direction is, for example, the upward direction on the screen of the display device displaying the first mask image Im5. Next, the coordinate point calculation unit 623 sets a first circumscribing rectangle R1 for the mask area Ms of the rotated first mask image Im5 so that its long side is parallel to the direction of the movement vector V. Next, the coordinate point calculation unit 623 reversely rotates the first mask image Im5 to which the first circumscribing rectangle R1 has been added by the above rotation amount, using the center of gravity C of the mask area Ms as the rotation center. As a result, the coordinate point calculation unit 623 sets the coordinate point of one of the four vertices of the first circumscribing rectangle R1, which has coordinates that are closest to the positioning point 10e of the vehicle 10, as the base coordinate point P0. Then, as shown in FIG. 9 , the coordinate point calculation unit 623 performs perspective transformation on the reverse-rotated first mask image Im5, i.e., the first mask image Im5 after calculating the base coordinate point P0. As a result, the coordinate point calculation unit 623 determines the coordinate point corresponding to the base coordinate point P0 as the first coordinate point P1 in the first circumscribed rectangle R1 transformed by the perspective transformation.

[0066] Furthermore, the coordinate point calculation unit 623 sets a second circumscribing rectangle R2 for the mask area Ms in the second mask image Im6 obtained by perspective transforming the first mask image Im5. Then, the coordinate point calculation unit 623 sets the vertex of the second circumscribing rectangle R2 that indicates the same position as the first coordinate point P1 as the second coordinate point P2. In other words, the first coordinate point P1 and the second coordinate point P2 are coordinate points that indicate the same position, and therefore have a correlation with each other.

[0067] Furthermore, the coordinate point calculation unit 623 corrects the coordinates (Xi1, Yi1) of the first coordinate point P1 by the coordinates (Xi2, Yi2) of the second coordinate point P2 depending on the magnitude relationship between the coordinate values ​​of the first coordinate point P1 and the second coordinate point P2. If the coordinate value Xi1 in the Xi direction of the first coordinate point P1 is greater than the coordinate value Xi2 in the Xi direction of the second coordinate point P2 (Xi1>Xi2), the coordinate point calculation unit 623 replaces the coordinate value Xi1 in the Xi direction of the first coordinate point P1 with the coordinate value Xi2 in the Xi direction of the second coordinate point P2. When the coordinate value Yi1 of the first coordinate point P1 in the Yi direction is greater than the coordinate value Yi2 of the second coordinate point P2 in the Yi direction (Yi1>Yi2), the coordinate point calculation unit 623 replaces the coordinate value Yi1 of the first coordinate point P1 in the Yi direction with the coordinate value Yi2 of the second coordinate point P2 in the Yi direction. In the example shown in FIG. 9, the coordinate value Xi1 of the first coordinate point P1 in the Xi direction is greater than the coordinate value Xi2 of the second coordinate point P2 in the Xi direction. Furthermore, the coordinate value Yi1 of the first coordinate point P1 in the Yi direction is smaller than the coordinate value Yi2 of the second coordinate point P2 in the Yi direction. Therefore, the image coordinate point P3 has coordinates (Xi2, Yi1). In this way, the coordinate point calculation unit 623 calculates the image coordinate point P3 indicating the position of the vehicle 10 in the image coordinate system by correcting the first coordinate point P1 using the second coordinate point P2. The method for calculating the image coordinate point P3 is not limited to the above.

[0068] As shown in FIG. 7, after the coordinate point calculation step, a position conversion step (step S10) is executed. The position conversion step is a step of converting the image coordinate point P3 into a vehicle coordinate point to calculate a vehicle coordinate point that indicates the position of the vehicle 10 in the global coordinate system. In the position conversion step, the position conversion unit 624 converts the image coordinate point P3 into a vehicle coordinate point using the imaging parameters and vehicle parameters acquired by the data acquisition unit 621. The vehicle parameters are parameters related to the distance from a reference point to a positioning point 10e of the vehicle 10. In this embodiment, the vehicle parameter is a height h (see FIG. 11, described later) of the positioning point 10e of the vehicle 10 from the road surface 20.

[0069] The position conversion unit 624 converts the image coordinate point P3 into a vehicle coordinate point using the relational expressions (1) to (3) described below, which use the vehicle coordinate point as a response variable and include the image coordinate point P3, the imaging parameters, and the vehicle parameters as explanatory variables. In this case, the position conversion unit 624 substitutes the coordinate value of the image coordinate point P3 calculated by the coordinate point calculation unit 623 into the relational expressions expressed in the expressions (1) to (3). Furthermore, the position conversion unit 624 substitutes the imaging parameters acquired by the data acquisition unit 621, i.e., the values ​​of the imaging parameters corresponding to the imaging device 9 that acquired the original image Im1, into the relational expressions expressed in the expressions (1) to (3).

[0070] FIG. 11 is a first diagram for explaining the details of the position conversion process. FIG. 11 illustrates the vehicle 10 as viewed from the left side. FIG. 12 is a second diagram for explaining the details of the position conversion process. FIG. 12 illustrates the vehicle 10 as viewed from the roof side. The global coordinate system shown in FIGS. 11 and 12 has a fixed coordinate point Pf indicating an arbitrary reference position on the road surface 20 as its origin, and coordinate axes indicated by the Xg axis and the Yg axis perpendicular to the Xg axis. The imaging coordinate point Pc is the position of the imaging device 9 that captured the original image Im1 used to calculate the image coordinate point P3, and is a coordinate point indicating the position of the imaging device 9 in the global coordinate system. The fixed coordinate point Pf and the imaging coordinate point Pc are pre-stored in the storage unit 63 of the position calculation device 6.

[0071] As shown in FIG. 11, the observation distance on the XgYg plane between the position of the imaging device 9 and the position of the vehicle 10 (image coordinate point P3) is denoted as Do. The observation error is denoted as ΔD. The height [m] of the imaging device 9 from the road surface 20, which is an imaging parameter, is denoted as H. The height [m] of the positioning point 10e of the vehicle 10 from the road surface 20, which is a vehicle parameter, is denoted as h. In this case, the observation error ΔD can be expressed by the following equation (1). ΔD=h / H×Do Equation (1) That is, the larger the observation distance Do, the larger the observation error ΔD.

[0072] Next, when the actual distance between the position of the imaging device 9 and the position of the positioning point 10e of the vehicle 10 (hereinafter referred to as the first distance) is D, the first distance D can be expressed by the following equation (2). D=Do×(1-h / H) Equation (2) That is, the first distance D is determined by the observation distance Do, the height H of the imaging device 9 as an imaging parameter, and the height h of the positioning point 10e of the vehicle 10 as a vehicle parameter.

[0073] Then, as shown in FIG. 12, if the estimated distance between the reference position and the position of vehicle 10 is Dp and the actual distance between the reference position and vehicle 10 (hereinafter referred to as the second distance) is Dt, the second distance Dt can be expressed by the following equation (3): Dt = Dp × (1-h / H) Equation (3)

[0074] Here, the estimated distance Dp can be calculated using the actual distance (hereinafter referred to as the third distance Dc) calculated from the fixed coordinate point Pf and the image coordinate point Pc, the image coordinate point P3, and the fixed coordinate point Pf. Therefore, the position conversion unit 624 can calculate the vehicle coordinate point Pv using the second distance Dt calculated by correcting the estimated distance Dp using the above equation (3) and the fixed coordinate point Pf. At this time, the calculated vehicle coordinate point Pv is a coordinate point indicating the position of the vehicle 10 in the global coordinate system, and therefore corresponds to the position of the vehicle 10 in real space. The second transmission unit 625 transmits the vehicle coordinate point Pv to the remote control device 7.

[0075] A-3. Vehicle operation control method: Fig. 13 is a flowchart showing an operation control method for vehicle 10 traveling in remote automatic driving mode. The operation control method shown in Fig. 13 is repeatedly executed, for example, every time a vehicle coordinate point Pv is received from position calculation device 6 as information indicating the position of vehicle 10.

[0076] The information acquisition unit 721 of the remote control device 7 acquires driving information including the vehicle coordinate point Pv (step S101). Then, the control value creation unit 722 uses the driving information to create a control value that defines the driving operation of the vehicle 10 (step S102). Then, the third transmission unit 723 transmits the control value to the vehicle 10 (step S103). The vehicle control device 150 mounted on the vehicle 10 drives the drive unit 110 and the like in accordance with the received control value (step S104).

[0077] According to the first embodiment, when the vehicle 10 is made to travel automatically by remote control, the position calculation system 1 can calculate the position of the vehicle 10 by using a captured image including the vehicle 10. Specifically, the detection device 5 inputs the captured image to a detection model, which is a machine learning model capable of detecting the vehicle 10 included in the captured image. This allows the detection device 5 to detect the vehicle 10 included in the captured image. Then, the position calculation device 6 can calculate a vehicle coordinate point Pv indicating the position of the vehicle 10 included in the captured image by using a first mask image Im5, which is the detection result by the detection device 5.

[0078] Here, because the vehicle 10 has different appearances due to multiple manufacturing processes, it can be classified into multiple states based on the differences in appearance. As a result, the feature values ​​extracted when training the detection model vary depending on the appearance state of the vehicle 10. Therefore, when a detection model trained using only multiple training images including vehicles 10 classified into different states, with the same number of images for each state, is applied to vehicles 10 classified into all states, the detection accuracy of the vehicle 10 in some states may decrease. If the detection accuracy of the vehicle 10 in some states decreases, it is possible to retrain the detection model by adding training images including the vehicle 10 in the state with the decreased detection accuracy. However, by training more training images including the vehicle 10 in some states than training images including the vehicle 10 in other states, the feature values ​​included in the added training images may be learned more intensively than other feature values. This may result in new differences in detection accuracy between multiple states. If the detection accuracy of the vehicle 10 included in the captured images decreases, the calculation accuracy of the vehicle coordinate point Pv indicating the position of the vehicle 10 decreases. If the calculation accuracy of the vehicle coordinate point Pv decreases, there is a risk that the desired control value cannot be generated.

[0079] Therefore, in the first embodiment, a first detection model Md1 suitable for detecting a vehicle 10 classified into one state is prepared for each state by training more first training images including the vehicle 10 classified into one state than second training images including the vehicle 10 classified into another state. Furthermore, according to the first embodiment, the detection device 5 acquires state information indicating the state of the vehicle 10 included in the captured image and can select a first detection model Md1 corresponding to the state identified by the acquired state information from among the multiple first detection models Md1 prepared for each state. In other words, when calculating the position of a vehicle 10 traveling within a factory, a first detection model Md1 corresponding to the state of the vehicle 10, which is classified into multiple states due to different appearances resulting from multiple manufacturing processes, can be selected. The detection device 5 then inputs the captured image into the first detection model Md1 corresponding to the state of the vehicle 10 included in the captured image. This allows the detection device 5 to accurately detect the vehicle 10 included in the captured image. That is, by selecting the first detection model Md1 suitable for detecting the vehicle 10 classified into one state, each region constituting the captured image can be accurately classified into a target region and a non-target region. As a result, it is possible to prevent the detection accuracy of the vehicle 10 included in the captured image from decreasing due to differences in appearance between multiple manufacturing processes. As a result, it is possible to prevent the calculation accuracy of the vehicle coordinate point Pv from decreasing. This makes it possible to reduce the difference between the position of the vehicle 10 calculated by the position calculation device 6 and the actual position of the vehicle 10. Therefore, it is possible to create more appropriate control values.

[0080] Furthermore, according to the first embodiment, the first teacher data set includes M (M is an integer equal to or greater than 2) first training images and N (N is an integer equal to or greater than 0 and less than M) second training images. In this case, the first teacher data set may include only the first training images without including the second training images. In this case, for example, the detection accuracy of the first detection model Md1 can be improved as the number M of first training images including vehicles 10 classified into one state increases. Furthermore, the first teacher data set may include the first training images and N second training images, the number of which is smaller than the first training images. In this case, including the second training images in the first teacher data set can improve, for example, the detection accuracy of parts of the vehicle 10 that have a common appearance regardless of the state.

[0081] Furthermore, according to the first embodiment, when the form of the vehicle 10 is a platform form, the state of the vehicle 10 can be classified as a platform state. When detecting a vehicle 10 classified as a platform state, the vehicle 10 can be detected by selecting the first detection model Md1 suitable for detecting a vehicle 10 classified as a platform state and inputting a captured image. When detecting a vehicle 10 classified as a platform state, the vehicle 10 can be detected by inputting a captured image associated with the platform state into the second detection model Md2. This improves the detection accuracy of vehicles 10 classified as a platform state.

[0082] Furthermore, according to the first embodiment, when the platform vehicle 101 in the form of a platform is included in the captured image, the external shape of the platform vehicle 101 can be detected. This makes it possible to calculate the position of the vehicle 10 using the captured image not only for the vehicle 10 as a completed vehicle, but also for the vehicle 10 as a semi-finished product or work in progress.

[0083] Furthermore, according to the first embodiment, the platform vehicle 101 has a significantly different appearance from the first assembly vehicle 102 and the second assembly vehicle 103. This allows the state of the vehicle 10 to be classified into a platform state and an assembly state before and after the assembly of the vehicle body, such as the body shell 180. This makes it possible to prevent a decrease in the detection accuracy of the vehicle 10 included in the captured image due to the difference in the external shape of the vehicle 10 before and after the assembly of the vehicle body.

[0084] Furthermore, according to the first embodiment, when detecting a vehicle 10 included in a captured image, it is possible to correct distortion of the original image Im1 and generate a corrected image Im2. In this way, it is possible to improve the detection accuracy of the vehicle 10 included in the captured image. This further improves the calculation accuracy of the position of the vehicle 10.

[0085] Furthermore, according to the first embodiment, when detecting a vehicle 10 included in a captured image, a rotated image Im3 can be generated by rotating the corrected image Im2 so that the direction of the movement vector V of the vehicle 10 faces a predetermined direction. In this way, the vehicle 10 included in the captured image can be detected with the direction of the movement vector V unified. This further improves the detection accuracy of the vehicle 10 included in the captured image.

[0086] Furthermore, according to the first embodiment, when detecting a vehicle 10 included in a captured image, a cropping process is performed to cut out a necessary area including the vehicle 10 from the rotated image Im3, thereby generating a processed image Im4 from which unnecessary areas have been removed. In this way, the area occupied by the vehicle 10 in the captured image can be made larger compared to when the cropping process is not performed. This makes it easier to detect a vehicle 10 that is farther away from the imaging device 9. Therefore, the detection accuracy for a vehicle 10 that is farther away from the imaging device 9 can be improved.

[0087] Furthermore, according to the first embodiment, by inputting the processed image Im4 to the first detection model Md1, the target region among the regions constituting the processed image Im4 is masked. This makes it possible to generate a first mask image Im5 in which a mask region Ms is added to the target region. At this time, according to the first embodiment, a DNN having a CNN structure that realizes semantic segmentation and instance segmentation can be used as the algorithm of the first detection model Md1. This makes it possible to prevent a decrease in the detection accuracy of the vehicle 10 due to the diversity of non-target regions in the captured image.

[0088] Furthermore, according to the first embodiment, the second mask image Im6 can be generated by performing perspective transformation on the first mask image Im5, thereby converting the camera coordinate system into the image coordinate system.

[0089] Furthermore, according to the first embodiment, by setting a first circumscribing rectangle R1 for the mask area Ms before perspective transformation of the first mask image Im5, it is possible to calculate a base coordinate point P0, which is a vertex of the first circumscribing rectangle R1 having coordinates closest to the positioning point 10e of the vehicle 10. Then, by performing perspective transformation on the first mask image Im5 after calculating the base coordinate point P0, it is possible to calculate a first coordinate point P1, which is a coordinate point corresponding to the base coordinate point P0. Furthermore, by setting a second circumscribing rectangle R2 for the mask area Ms of the second mask image Im6, it is possible to calculate a second coordinate point P2, which is a vertex of the second circumscribing rectangle R2 having coordinates closest to the positioning point 10e of the vehicle 10. Then, by correcting the first coordinate point P1 using the second coordinate point P2, it is possible to calculate an image coordinate point P3. In this way, by comparing and correcting the coordinate points before and after perspective transformation, it is possible to more accurately calculate the image coordinate point P3. This further improves the accuracy of calculating the position of the vehicle 10.

[0090] Furthermore, according to the first embodiment, it is possible to acquire process information related to one manufacturing process being performed on the vehicle 10. Then, by using the process database Db1 to identify status information associated with one manufacturing process identified by the acquired process information and acquiring the status information about the vehicle 10, it is possible to identify the status of the vehicle 10 included in the captured image.

[0091] Furthermore, according to the first embodiment, it is possible to acquire imaging parameters related to the imaging device 9 that acquired the original image Im1. Then, by substituting the calculated image coordinate point P3 and the acquired imaging parameter values ​​into a relational expression that uses the vehicle coordinate point Pv as a response variable and includes the image coordinate point P3, the imaging parameters, and the vehicle parameters as explanatory variables, it is possible to convert the image coordinate point P3 into the vehicle coordinate point Pv. This converts the image coordinate system into a global coordinate system, and makes it possible to calculate the position of the positioning point 10e of the vehicle 10 in the global coordinate system as the vehicle coordinate point Pv.

[0092] Furthermore, according to the first embodiment, the imaging parameter is the height H of the imaging device 9 from the road surface 20, which is calculated based on the position of the imaging device 9 in the global coordinate system. Furthermore, the vehicle parameter is the height h of the positioning point 10e of the vehicle 10 from the road surface 20. In this way, the observation error ΔD can be calculated based on the similarity relationship between the imaging parameter and the vehicle parameter. Then, the image coordinate point P3 can be converted to the vehicle coordinate point Pv using the calculated observation error ΔD.

[0093] Furthermore, according to the first embodiment, the position of the vehicle 10 can be calculated without installing any installation objects, such as a marker or a transceiver, on the vehicle 10 to be used for calculating the position of the vehicle 10. Furthermore, the position of the vehicle 10 can be calculated without installing the position calculation device 6 on the vehicle 10. This can improve the versatility of the position calculation system 1.

[0094] B. Second embodiment: FIG. 14 is a diagram showing the manufacturing process of the vehicle 10 in the second embodiment. FIG. 14 shows a representative portion of the multiple manufacturing steps executed in the manufacturing process of the vehicle 10. FIG. 15 is a diagram showing the configuration of the detection device 5a in the second embodiment. In this embodiment, part of the manufacturing process of the vehicle 10 is different from that in the first embodiment. As a result, part of the configuration of the detection device 5a is different from that in the first embodiment. The other configurations are the same as those in the first embodiment. The same reference numerals are used for the same configurations as in the first embodiment, and descriptions thereof will be omitted.

[0095] As shown in FIG. 14 , in this embodiment, the vehicle 10 as a finished product is manufactured by assembling a body shell 180 to a platform vehicle 101, then assembling interior parts and exterior parts 190, and painting the vehicle body (body portion) consisting of the body shell 180, the exterior parts 190, etc. In other words, in this embodiment, the platform manufacturing process, the body shell assembling process, the parts assembling process, and the painting process are performed in this order, thereby manufacturing the vehicle 10 as a finished product. The painting process is a process of painting the vehicle 10. In the painting process, the vehicle 104 before painting (hereinafter referred to as the unpainted vehicle 104) and the vehicle 105 after painting in the painting process (hereinafter referred to as the painted vehicle 105) have different exterior colors, and therefore the appearance of the vehicle 10 is different. Therefore, in this embodiment, the storage unit 53a of the detection device 5a stores three first detection models Md1c to Md1e as detection models suitable for detecting a vehicle 10 that is classified into one state determined by the exterior shape and exterior color of the vehicle 10. Note that, hereinafter, the state of the vehicle 10 before it is painted in the painting process, such as the unpainted vehicle 104, is referred to as the "unpainted state." The state of the vehicle 10 after it has been painted in the painting process, such as the painted vehicle 105, is referred to as the "painted state."

[0096] The first first detection model Md1c is a first detection model Md1 that has been trained on more vehicles 10 classified as "platform state" than on vehicles 10 classified as states other than the "platform state." In the first first detection model Md1c, the first training images are images including vehicles 10 classified as "platform state." The second training images are, for example, images including vehicles 10 classified as "assembled state" and "unpainted state." The first first detection model Md1a is used when detecting vehicles 10 classified as "platform state."

[0097] The second first detection model Md1d is a first detection model Md1 that has been trained on more vehicles 10 classified as "assembled" and "unpainted" than on vehicles 10 classified as states other than "assembled" and "unpainted." In the second first detection model Md1d, the first training images are images including vehicles 10 classified as "assembled" and "unpainted." The second training images are, for example, images including vehicles 10 classified as "assembled" and "painted." The second first detection model Md1d is used when detecting vehicles 10 in an "assembled" and "unpainted" state.

[0098] The third first detection model Md1e is a first detection model Md1 that has been trained on more vehicles 10 classified as "assembled" and "painted" than on vehicles 10 classified as states other than "assembled" and "painted." In the third first detection model Md1e, the first training images are images including vehicles 10 classified as "assembled" and "painted." The second training images are, for example, images including vehicles 10 classified as "assembled" and "unpainted." The third first detection model Md1e is used when detecting vehicles 10 classified as "assembled" and "painted."

[0099] According to the second embodiment, the appearance of the unpainted vehicle 104 and the painted vehicle 105 is significantly different. This allows the state of the vehicle 10 to be classified into an unpainted state and a painted state before and after painting in the painting process. This makes it possible to prevent a decrease in the detection accuracy of the vehicle 10 included in the captured image due to the difference in the exterior color of the vehicle 10 before and after the painting process is performed.

[0100] C. Third embodiment: FIG. 16 is a diagram showing the configuration of a detection device 5b in the third embodiment. In this embodiment, the detection model used when detecting a vehicle 10 included in a captured image is different from that in the first embodiment. As a result, part of the configuration of the detection device 5b and part of the processing of the method for detecting a vehicle 10 are different from those in the first embodiment. The other configurations are the same as those in the first embodiment. The same steps as those in the first embodiment and the same configurations as those in the first embodiment are assigned the same reference numerals and descriptions thereof will be omitted.

[0101] In this embodiment, the storage unit 53b of the detection device 5b stores one second detection model Md2 as a detection model. Like the first detection model Md1, the second detection model Md2 is a trained machine learning model used to detect the vehicle 10 included in the captured image. The second detection model Md2 is a machine learning model trained by inputting a second teacher dataset. The second teacher dataset includes a plurality of training images each including a vehicle 10 classified into different states, region correct labels associated with a plurality of regions constituting each training image, and state correct labels associated with each of the plurality of training images. As in the first embodiment, the region correct labels are correct labels indicating whether each region in the training image is a target region indicating the vehicle 10 or a non-target region indicating something other than the vehicle 10. The state correct labels are correct labels indicating one state of the vehicle 10 included in the training image. In this embodiment, the second detection model Md2 executes the following processing when a captured image is input together with one state identified by the state information acquired by the state acquisition unit 525. In this case, the second detection model Md2 identifies a target area included in the input captured image. Then, the second detection model Md2 masks the target area to generate a first mask image Im5 in which a mask area Ms is added to the target area. For example, CNN is used as the algorithm of the second detection model Md2. Note that the configuration of the second detection model Md2 is not limited to the above. For example, the second detection model Md2 may be a trained machine learning model that uses an algorithm other than a neural network.

[0102] 17 is a diagram illustrating details of a CPU 52b mounted on a detection device 5b according to the third embodiment. The CPU 52b of the detection device 5b according to the third embodiment includes a model acquisition unit 526b instead of the model acquisition unit 526 in the first embodiment. The model acquisition unit 526b acquires a second detection model Md2. The CPU 52b of the detection device 5b also includes a detection unit 527b instead of the detection unit 527 in the first embodiment. The detection unit 527b inputs a captured image and a state identified by the state information acquired by the state acquisition unit 525 into the second detection model Md2, and identifies a target region among the regions constituting the input captured image, thereby detecting a vehicle 10 included in the captured image.

[0103] Fig. 18 is a flowchart showing a method for detecting a vehicle 10 and a method for calculating the position of the vehicle 10 in the third embodiment. The method shown in Fig. 18 is executed repeatedly at predetermined time intervals, for example, after the vehicle 10 to be detected starts traveling in the remote automated driving mode.

[0104] In this embodiment, in a model acquisition step (step S6b), the model acquisition unit 526b acquires a second detection model Md2. Then, in a detection step (step S7b), the detection unit 527b inputs the processed image Im4 and one state identified by the state information acquired by the state acquisition unit 525 to the second detection model Md2. As a result, the detection unit 527b detects the outline of the vehicle 10 included in the processed image Im4. Then, the detection unit 527b masks the target area in the processed image Im4 to generate a first mask image Im5 in which a mask area Ms is added to the target area.

[0105] According to the third embodiment, a second detection model Md2 is prepared as a detection model trained by associating a plurality of training images including the vehicle 10 with state labels indicating the state of the vehicle 10 included in each training image. As a result, by inputting captured images associated with the state of the vehicle 10 into the second detection model Md2, the vehicle 10 included in the input captured image can be accurately detected. That is, each region constituting the captured image input to the second detection model Md2 can be accurately classified into a target region and a non-target region. As a result, it is possible to prevent a decrease in the detection accuracy of the vehicle 10 included in the captured image due to differences in appearance between multiple manufacturing processes. As a result, it is possible to prevent a decrease in the calculation accuracy of the vehicle coordinate point Pv. This reduces the difference between the position of the vehicle 10 calculated by the position calculation device 6 and the actual position of the vehicle 10. Therefore, it is possible to generate more appropriate control values.

[0106] D. Fourth embodiment: FIG. 19 is a diagram showing the configuration of a detection device 5c in the fourth embodiment. In this embodiment, the method of acquiring state information is different from that of the first embodiment. As a result, part of the configuration of the detection device 5c is different from that of the first embodiment. The other configurations are the same as those of the first embodiment. The same reference numerals are used for the same configurations as those of the first embodiment, and descriptions thereof will be omitted.

[0107] In this embodiment, the storage unit 53c of the detection device 5c stores an identification model Md3 instead of the process database Db1. The identification model Md3 is a trained machine learning model used to identify one state of the vehicle 10 included in a captured image and acquire state information indicating the identified one state. Specifically, the identification model Md3 is a trained machine learning model that is trained to output state information about the vehicle 10 included in the captured image when the captured image is input. The identification model Md3 learns feature quantities corresponding to the state of the vehicle 10 in order to identify one state of the vehicle 10. For example, a CNN is used as the algorithm for the identification model Md3. Note that the configuration of the identification model Md3 is not limited to the above. For example, the identification model Md3 may be a trained machine learning model that uses an algorithm other than a neural network.

[0108] 20 is a diagram illustrating details of a CPU 52c mounted on a detection device 5c according to the fourth embodiment. In this embodiment, a state acquisition unit 525c inputs a captured image to a specific model Md3, thereby acquiring state information about the vehicle 10 included in the captured image.

[0109] According to the fourth embodiment, the captured image is input to the specific model Md3 that has been trained to output status information about the vehicle 10 included in the captured image when the captured image is input, thereby making it possible to acquire status information about the vehicle 10 included in the captured image. In other words, it is possible to acquire status information about the vehicle 10 using machine learning.

[0110] E. Fifth embodiment: FIG. 21 is a diagram showing the configuration of a detection device 5d in the fifth embodiment. In this embodiment, the method of acquiring state information is different from that in the first embodiment. As a result, part of the configuration of the detection device 5d is different from that in the first embodiment. The other configurations are the same as those in the first embodiment. A description of the same configuration as in the first embodiment will be omitted.

[0111] In this embodiment, the storage unit 53d of the detection device 5d stores a status database Db3 instead of the process database Db1. The status database Db3 is a database that associates imaging information and status information related to the imaging device 9. The imaging information is information for identifying the imaging device 9 that acquired the original image Im1. The imaging information is, for example, camera identification information. In this case, the status database Db3 is a database that associates, for example, the camera identification information, the imaging range RG of the imaging device 9 identified by the camera identification information, and status information that indicates the status of the vehicle 10 according to the manufacturing process performed in the imaging range RG. Note that the configurations of the status database Db3 and the imaging information are not limited to those described above. The imaging information may include, for example, the imaging range RG instead of the camera identification information.

[0112] 22 is a diagram illustrating details of a CPU 52d mounted on a detection device 5d in the fifth embodiment. In this embodiment, the status acquisition unit 525d acquires, as imaging information related to the imaging device 9 that acquired the original image Im1, camera identification information indicating the imaging device 9 that acquired the original image Im1. Then, the status acquisition unit 525d acquires, using a status database Db3, status information associated with the imaging device 9 identified by the acquired imaging information, thereby acquiring status information about the vehicle 10 included in the captured image.

[0113] According to the fifth embodiment, it is possible to acquire imaging information related to the imaging device 9. Then, by using the state database Db3 in which imaging information and state information are associated with each other, it is possible to identify state information associated with one imaging device 9 identified by the acquired imaging information, thereby acquiring state information indicating the state of the vehicle 10 included in the captured image. This makes it possible to acquire state information about the vehicle 10 included in the captured image without performing image analysis, thereby reducing the processing load on the state acquisition unit 525d in the state acquisition step.

[0114] F. Other Embodiments: F-1. Alternative embodiment 1: The finished vehicle 10 may be manufactured by assembling multiple painted units. In this case, the finished vehicle 10 may be manufactured by assembling, for example, a front unit, a rear unit, a left unit, a right unit, a ceiling unit, and a bottom unit. The front unit is a semi-finished product formed by integrally molding a group of parts that form the front side of the vehicle 10. The front unit includes, for example, a front lamp, a front bumper, a front grille, and a hood. The rear unit is a semi-finished product formed by integrally molding a group of parts that form the rear side of the vehicle 10. The rear unit includes, for example, a rear lamp, a rear bumper, and a trunk. The left unit is a semi-finished product formed by integrally molding a group of parts that form the left side of the vehicle 10. The right unit is a semi-finished product formed by integrally molding a group of parts that form the right side of the vehicle 10. The left unit and the right unit each include, for example, a side sill and a pillar. The ceiling unit is a semi-finished product formed by integrally molding a group of parts that form the upper side of the vehicle 10. The ceiling unit includes, for example, a roof. The bottom unit is a semi-finished product formed by integrally molding a group of parts that form the bottom side of the vehicle 10. The bottom unit includes, for example, a chassis 160 and a seat. Even in this configuration, the state of the vehicle 10 can be classified into a plurality of states depending on the appearance of the vehicle 10. This makes it possible to prevent a decrease in the detection accuracy of the vehicle 10 depending on the appearance of the vehicle 10 by inputting a captured image into the first detection model Md1 that is suitable for one state determined by the appearance of the vehicle 10, or inputting a captured image associated with the state of the vehicle 10 into the second detection model Md2.

[0115] F-2. Alternative embodiment 2: In a captured image, the lower the contrast between the vehicle 10 and the road 2, the smaller the color difference between the target area and the non-target area. Therefore, there is a risk that the detection accuracy of the vehicle 10 will decrease depending on the exterior color of the vehicle 10 and the color of the road 2. Therefore, a plurality of first detection models Md1 used to detect a vehicle 10 classified as having a painted state may be prepared according to the exterior color of the vehicle 10. For example, when detecting a vehicle 10 classified as having a painted state, the following two first detection models Md1 may be prepared. In this case, a first detection model Md1 for detecting a vehicle 10 painted with a light-colored paint having a brightness equal to or greater than a predetermined threshold, and a first detection model Md1 for detecting a vehicle 10 painted with a dark-colored paint having a brightness less than the threshold may be prepared. This configuration allows the state of the vehicle 10 to be classified into a plurality of states depending on the color of the paint applied in the painting process. This prevents a decrease in the detection accuracy of the vehicle 10 in a captured image depending on the color of the paint applied in the painting process, i.e., the exterior color of the vehicle 10.

[0116] F-3. Alternative embodiment 3: The training images included in the first and second teacher data sets may be images that have not been subjected to processing such as distortion correction, rotation, or cropping, like the original image Im1. In this case, no image processing is required when preparing the first and second teacher data sets. This reduces the processing load when training the first and second detection models Md1 and Md2.

[0117] F-4. Alternative embodiment 4: The training images included in the first teacher data set and the second teacher data set may be any one of the corrected image Im2, the rotated image Im3, and the processed image Im4. In this configuration, when a captured image obtained by processing the original image Im1 is input to either the first detection model Md1 or the second detection model Md2, as in the first and third embodiments, the detection accuracy of the vehicle 10 can be improved.

[0118] F-5. Alternative Embodiment 5: In the vehicle 10 detection methods shown in Figures 7 and 18, the distortion correction step is not an essential step. For example, if the rotation processing step and subsequent steps are performed without performing the distortion correction step, the rotation processing unit 523 rotates the original image Im1 instead of the corrected image Im2 in the rotation processing step. Even in this configuration, the image coordinate point P3 can be calculated.

[0119] F-6. Alternative Embodiment 6: 7 and 18, the rotation process is not an essential step. For example, if the distortion correction process and the rotation process are not performed, the cropping processor 524 performs cropping on the original image Im1 instead of the rotated image Im3 in the cropping process. Even in this case, the image coordinate point P3 can be calculated.

[0120] F-7. Alternative Embodiment 7: In the vehicle 10 detection methods shown in Figures 7 and 18, the cropping process is not an essential step. For example, if the distortion correction process, rotation process, and cropping process are not performed, the detection units 527 and 527b generate a first mask image Im5 by adding a mask area Ms to the original image Im1 in place of the processed image Im4 in the detection process. Even in this configuration, the image coordinate point P3 can be calculated.

[0121] F-8. Alternative Embodiment 8: The captured image may include multiple vehicles 10. In this case, the CPUs 52, 52b, 52c, 52d of the detection devices 5, 5a, 5b, 5c, 5d may include, for example, a deletion unit that deletes, from the first mask image Im5, mask regions Ms of the vehicles 10 that are not to be subject to position calculation. For example, the deletion unit determines that, among the mask regions Ms generated in the detection process, a mask region Ms that exists outside the recognition target region is a mask region of the vehicle 10 that is not to be subject to position calculation, and deletes the mask region Ms from the first mask image Im5. The recognition target region is, for example, a predetermined region in the first mask image Im5 through which the vehicle 10 moves. The predetermined region through which the vehicle 10 moves is, for example, a region corresponding to the region of the grid lines 21. The recognition target region is pre-stored in the storage units 53, 53a, 53d of the detection devices 5, 5a, 5b, 5c, 5d. With this configuration, when a captured image includes multiple vehicles 10, it is possible to eliminate the influence of the vehicles 10 that are not to be subject to position calculation. This improves the accuracy of calculating the position of the vehicle 10.

[0122] F-9. Alternative Embodiment 9: The captured image may include multiple vehicles 10. In this case, a DNN that performs instance segmentation may be used as the algorithm for the first detection model Md1 and the second detection model Md2. In this manner, the multiple vehicles 10 included in the captured image can be classified, and a first mask image Im5 in which each vehicle 10 is masked can be generated. As a result, when the captured image includes multiple vehicles 10, it is possible to select a vehicle 10 to be the target of position calculation, and calculate the position of the selected vehicle 10.

[0123] F-10. Alternative Embodiment 10: The position calculation system 1 may calculate the position of a stationary vehicle 10. When calculating the position of a stationary vehicle 10, the position calculation system 1 calculates the position of the vehicle 10 by using, for example, an initial vector direction of the vehicle 10 estimated from the original image Im1 acquired first after startup of the position calculation system 1, instead of the direction of the movement vector V of the vehicle 10 while it is moving. In this manner, even when the vehicle 10 is stopped, the position of the vehicle 10 can be calculated by using the captured image.

[0124] F-11. Other Embodiment 11: At least one of the detection devices 5, 5a to 5d, the position calculation device 6, and the remote control device 7 may be configured as an integrated unit. Furthermore, each of the detection devices 5, 5a to 5d, the position calculation device 6, and the remote control device 7 may be realized by, for example, cloud computing configured by one or more computers. In this configuration, the configurations of the detection devices 5, 5a to 5d, the position calculation device 6, and the remote control device 7 can be changed as appropriate.

[0125] F-12. Other Embodiment 12: The image acquisition unit 521 of the detection devices 5, 5a to 5d may acquire the original image Im1 via an external device (for example, the remote control device 7) other than the imaging device 9, rather than acquiring the original image Im1 directly from the imaging device 9. Even in this configuration, the image acquisition unit 521 of the detection devices 5, 5a to 5d can acquire the original image Im1 acquired by the imaging device 9.

[0126] F-13. Other Embodiment 13: The detection devices 5, 5a to 5d may detect the vehicle 10 using the first detection model Md1 that is stored in a device other than the detection devices 5, 5a to 5d and selected by the device other than the detection devices 5, 5a to 5d. Even in this configuration, the first detection model Md1 according to the state of the vehicle 10 can be acquired.

[0127] The present disclosure is not limited to the above-described embodiments and can be realized in various configurations without departing from the spirit thereof. For example, the technical features of the embodiments corresponding to the technical features in each aspect described in the Summary of the Invention section can be appropriately replaced or combined to solve some or all of the above-described problems or achieve some or all of the above-described effects. Furthermore, if a technical feature is not described as essential in this specification, it can be appropriately deleted. [Explanation of symbols]

[0128] 1...position calculation system, 2...roadway, 5, 5a to 5d...detection device, 6...position calculation device, 7...remote control device, 9...imaging device, 10...vehicle, 10e...positioning point, 20...road surface, 21...grid line, 51...communication unit of detection device, 52, 52b to 52d...CPU of detection device, 53, 53a to 53d...storage unit of detection device, 61...communication unit of position calculation device, 62...CPU of position calculation device, 63...storage unit of position calculation device, 71...communication unit of remote control device, 72...CPU of remote control device, 73...storage unit of remote control device, 101...platform vehicle, 102...first assembly vehicle, 103... Second assembled vehicle, 104... unpainted vehicle, 105... painted vehicle, 110... drive unit, 120... steering unit, 130... braking unit, 140... vehicle communication unit, 150... vehicle control device, 160... chassis, 170... wheels, 180... body shell, 190... exterior parts, 521... image acquisition unit, 522... distortion correction unit, 523... rotation processing unit, 524... crop processing unit, 525, 525c, 525d... status acquisition unit, 526, 526b... model acquisition unit, 527, 527b... detection unit, 528... first transmission unit, 621... data acquisition unit, 622... perspective transformation unit, 623... coordinate point calculation unit, 624... position position conversion unit, 625...second transmission unit, 721...information acquisition unit, 722...control value creation unit, 723...third transmission unit, 901...first imaging device, 902...second imaging device, A1...unmoved area, A2...moved area, C...center of gravity, ΔD...observation error, D...first distance, Db1...process database, Db2...camera database, Db3...state database, Dc...third distance, Do...observed distance, Dp...estimated distance, Dt...second distance, h...height of vehicle positioning point from road surface, H...height of imaging device from road surface, Im1...original image, Im2...corrected image, Im3...rotated image, Im4...processed image, I m5...first mask image, Im6...second mask image, Md1, Md1a to Md1e...first detection model, Md2...second detection model, Md3...specific model, Ms...mask area, Nt...network, P0...base coordinate point, P1...first coordinate point, P2...second coordinate point, P3...image coordinate point, Pa1...distortion correction parameter, Pa2...perspective transformation parameter, Pc...imaging coordinate point, Pf...fixed coordinate point, Pv...vehicle coordinate point, R0...base circumscribing rectangle, R1...first circumscribing rectangle, R2...second circumscribing rectangle, RG...imaging range, RG1...first imaging range, RG2...second imaging range, V...movement vector

Claims

1. A detection device for detecting a vehicle included in a captured image, The vehicle travels within a factory where a plurality of manufacturing processes are performed to manufacture and ship the vehicle; The vehicle is classified into a plurality of states by having different appearances according to the plurality of manufacturing processes; The detection device includes: an image acquisition unit that acquires the captured image; a state acquisition unit that acquires state information indicating one of the states of the vehicle included in the captured image; a model acquisition unit that acquires a first detection model selected from a plurality of first detection models, which are machine learning models prepared for each of the states, in accordance with the state identified by the state information acquired by the state acquisition unit; a detection unit that detects the vehicle included in the captured image by inputting the captured image to the first detection model acquired by the model acquisition unit and identifying a target area in the captured image that indicates the vehicle.

2. A detection device for detecting a vehicle included in a captured image, The vehicle travels within a factory where a plurality of manufacturing processes are performed to manufacture and ship the vehicle; The vehicle is classified into a plurality of states by having different appearances according to the plurality of manufacturing processes; the plurality of states include a platform state in which the vehicle is in the form of a platform including at least wheels, a chassis, a drive unit that accelerates the vehicle, a steering unit that changes the traveling direction of the vehicle, a braking unit that decelerates the vehicle, a vehicle control unit that controls the operation of the vehicle, and a vehicle communication unit that communicates with other devices other than the vehicle itself; The detection device includes: an image acquisition unit that acquires the captured image; a state acquisition unit that acquires state information indicating one of the states of the vehicle included in the captured image; a model acquisition unit that acquires a first detection model selected from a plurality of first detection models, which are machine learning models prepared for each of the states, in accordance with the state identified by the state information acquired by the state acquisition unit; a detection unit that detects the vehicle included in the captured image by inputting the captured image to the first detection model acquired by the model acquisition unit and identifying a target area in the captured image that indicates the vehicle.

3. A detection device for detecting a vehicle included in a captured image, The vehicle travels within a factory where a plurality of manufacturing processes are performed to manufacture and ship the vehicle; The vehicle is classified into a plurality of states by having different appearances according to the plurality of manufacturing processes; The detection device includes: an image acquisition unit that acquires the captured image; a state acquisition unit that acquires state information indicating one of the states of the vehicle included in the captured image; a detection unit that detects the vehicle included in the captured image by inputting the captured image and the one state identified by the state information acquired by the state acquisition unit into a second detection model that is a machine learning model, and identifying a target area in the captured image that indicates the vehicle.

4. 2. The detection device according to claim 1, A detection device, wherein each of the plurality of first detection models has been pre-trained to identify the target region by inputting a plurality of training images including M (M is an integer of 2 or more) first training images each including the vehicle classified into one of the states, and N (N is an integer of 0 or more and less than M) second training images each including the vehicle classified into another of the states different from the one state.

5. 3. The detection device according to claim 2, A detection device, wherein each of the plurality of first detection models has been pre-trained to identify the target region by inputting a plurality of training images including M (M is an integer of 2 or more) first training images each including the vehicle classified into one of the states, and N (N is an integer of 0 or more and less than M) second training images each including the vehicle classified into another of the states different from the one state.

6. 4. The detection device according to claim 3, the second detection model is pre-trained to identify the target region by inputting a plurality of training images including the vehicle and state answer labels associated with each of the plurality of training images, the state answer labels indicating the state of the vehicle included in the training images.

7. 7. The detection device according to claim 4, wherein A detection device in which each region in the training image is associated with a region correct answer label indicating whether the region is the target region or a non-target region indicating a region other than the vehicle.

8. 4. The detection device according to claim 1, wherein: The status acquisition unit acquires process information related to one of the manufacturing processes being performed on the vehicle, and acquires the status information by using a process database in which the status information is associated with each of the multiple manufacturing processes to identify the status information associated with the one manufacturing process identified by the acquired process information.

9. 4. The detection device according to claim 1, wherein: The state acquisition unit acquires the state information by inputting the captured image to a specific model, which is a machine learning model trained to output the state information when the captured image is input.

10. 4. The detection device according to claim 1, wherein: The status acquisition unit acquires imaging information related to an imaging device, and acquires the status information by identifying the status information associated with one of the imaging devices identified by the acquired imaging information using a status database in which the imaging information and the status information are associated.

11. 4. The detection device according to claim 1, wherein: the plurality of manufacturing processes includes a painting process of painting the vehicle; The detection device, wherein the multiple states include (iii) an unpainted state indicating the state of the vehicle before it is painted in the painting process, and (iv) a painted state indicating the state of the vehicle after it has been painted in the painting process.

12. The detection device according to any one of claims 1 to 3, further comprising: The detection device includes a distortion correction unit that corrects distortion in the captured image.

13. The detection device according to any one of claims 1 to 3, further comprising: a rotation processing unit that rotates the captured image so that the moving direction of the vehicle faces a predetermined direction.

14. A position calculation system for calculating a position of a vehicle included in a captured image, A detection device according to any one of claims 1 to 3; a position calculation device that calculates the position of the vehicle, The detection device further generates a first mask image by masking a target area, which is an area indicating the vehicle, in the captured image and adding a mask area to the target area; a position calculation device that calculates a position of the vehicle using the first mask image, The position calculation device a perspective transformation unit that performs perspective transformation on the first mask image to generate a second mask image; a coordinate point calculation unit that calculates an image coordinate point that indicates a position of the vehicle in an image coordinate system by setting a specified vertex of a first circumscribing rectangle set in the mask area in the first mask image as a first coordinate point, setting a vertex of a second circumscribing rectangle set in the mask area in the second mask image that indicates the same position as the first coordinate point as a second coordinate point, and correcting the first coordinate point using the second coordinate point; a position conversion unit that converts the image coordinate points into vehicle coordinate points that indicate the position of the vehicle in the global coordinate system, using a distance from a reference point of the imaging device that is calculated based on the position of the imaging device in the global coordinate system and a distance from the reference point of a predetermined positioning point of the vehicle.

15. A detection method for detecting a vehicle included in a captured image, comprising: The vehicle travels within a factory where a plurality of manufacturing processes are performed to manufacture and ship the vehicle; The vehicle is classified into a plurality of states by having different appearances according to the plurality of manufacturing processes; The detection method includes: an image acquisition step of acquiring the captured image; a state acquisition step of acquiring state information indicating one of the states of the vehicle included in the captured image; a model acquisition step of acquiring a first detection model selected from a plurality of first detection models, which are machine learning models prepared for each of the states, in accordance with the state identified by the state information acquired in the state acquisition step; a detection step of detecting the vehicle included in the captured image by inputting the captured image to the first detection model acquired in the model acquisition step and identifying a target area in the captured image that indicates the vehicle.

16. A detection method for detecting a vehicle included in a captured image, comprising: The vehicle travels within a factory where a plurality of manufacturing processes are performed to manufacture and ship the vehicle; The vehicle is classified into a plurality of states by having different appearances according to the plurality of manufacturing processes; The detection method includes: an image acquisition step of acquiring the captured image; a state acquisition step of acquiring state information indicating one of the states of the vehicle included in the captured image; a detection process for detecting the vehicle included in the captured image by inputting the captured image and the one state identified by the state information acquired in the state acquisition process into a second detection model, which is a machine learning model, and identifying a target area in the captured image that indicates the vehicle.

Images