control device

Abstract

To provide a technique capable of preventing detection accuracy of a position and an orientation of a vehicle from being lowered, even when the vehicle has an assembly failure of components assembled in the vehicle.SOLUTION: A control device for remotely controlling a vehicle that is an object for a plurality of steps, the vehicle traveling inside a factory in which the plurality of steps of manufacturing the vehicle are executed, comprises a determination unit and a calculation unit. The determination unit determines whether or not components are properly assembled to the vehicle. The calculation unit acquires at least one of a position and an orientation of the vehicle using a first method, based on detection data acquired by an external sensor when the components are properly assembled to the vehicle, and finds at least one of the position and the orientation using a second method different from the first method, based on the detection data when the components are not properly assembled to the vehicle.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present disclosure relates to a control device.

Background Art

[0002] Patent Document 1 describes a vehicle that is an object of manufacture and travels by remote control in a manufacturing process for producing the vehicle.

Prior Art Literature

Patent Literature

[0003]

Patent Document 1

Summary of the Invention

Problem to be Solved by the Invention

[0004] In order to remotely control the traveling of a vehicle, it is necessary to detect the position and orientation of the vehicle. For this reason, the position and orientation of the vehicle are sometimes detected based on the external shape of the vehicle. In this case, there has been a demand for a technique that can suppress a decrease in detection accuracy of the position and orientation of the vehicle even if there is an assembly defect of a component to be assembled to the vehicle.

Means for Solving the Problem

[0005] The present disclosure can be implemented in the following modes.

[0006] (1) According to a first embodiment of the present disclosure, a control device is provided. The control device is a vehicle that travels within a factory where a plurality of processes are carried out to manufacture a vehicle, and is a control device for remotely controlling the vehicle which is the subject of the plurality of processes, and comprises a determination unit that determines whether or not parts have been correctly assembled to the vehicle, and a calculation unit that, when parts have been correctly assembled to the vehicle, obtains at least one of the position and orientation of the vehicle using a first method based on detection data obtained by an external sensor, and when parts have not been correctly assembled to the vehicle, determines at least one of the position and orientation using a second method different from the first method based on the detection data. In this configuration, different methods are used to calculate at least one of the vehicle's position and orientation depending on whether there are assembly defects in the parts or not. Therefore, even if there are assembly defects in the parts installed on the vehicle, a decrease in the accuracy of detecting the vehicle's position and orientation can be suppressed. (2) In the control device of the above embodiment, the first method is a method for determining at least one of the position and orientation using a first machine learning model generated by machine learning using a training dataset that includes images of the vehicle with the parts correctly assembled, based on images of the vehicle as detection data acquired by a camera as an external sensor, and the second method is a method for determining at least one of the position and orientation using a second machine learning model generated by machine learning using a training dataset that includes images of the vehicle with the parts not correctly assembled, based on images of the vehicle as detection data. In this configuration, different machine learning models are used to calculate at least one of the vehicle's position and orientation using images of the vehicle acquired by a camera acting as an external sensor, depending on whether there are assembly defects in the parts or not. Therefore, even if there are assembly defects in the parts installed on the vehicle, the decrease in the accuracy of detecting the vehicle's position and orientation can be suppressed. (3) In the control device of the above embodiment, the first method is a method for determining at least one of the position and orientation using a third machine learning model generated by machine learning using a training dataset that includes images of the vehicle taken in the state after the target process has been performed, based on the image of the vehicle as detection data acquired by the camera as an external sensor, and the second method is a method for determining at least one of the position and orientation using a fourth machine learning model generated by machine learning using a training dataset that includes images of the vehicle taken in the state before the target process has been performed, based on the image of the vehicle as detection data. In this configuration, different machine learning models are used to calculate at least one of the vehicle's position and orientation using images of the vehicle acquired by a camera acting as an external sensor, depending on whether there are assembly defects in the parts or not. Therefore, even if there are assembly defects in the parts installed on the vehicle, the decrease in the accuracy of detecting the vehicle's position and orientation can be suppressed. (4) In the control device of the above form, the first method is a method for determining at least one of the position and orientation using distance point data as detection data acquired by the distance measuring device as an external sensor and first reference data which is 3D CAD representing the shape of the vehicle with the parts correctly assembled, and the second method is a method for determining at least one of the position and orientation using distance point data as detection data and second reference data which is 3D CAD data representing the shape of the vehicle with the parts not correctly assembled. In this configuration, different methods are used to calculate the vehicle's position and orientation using distance measurement point data acquired by the distance measuring device as an external sensor, depending on whether there are assembly defects in the parts or not. Therefore, even if there are assembly defects in the parts installed on the vehicle, a decrease in the accuracy of detecting the vehicle's position and orientation can be suppressed. (5) In the control device of the above embodiment, the first method is a method for determining at least one of the position and orientation using distance point data as detection data acquired by the distance measuring device as an external sensor and third reference data which is 3D CAD data representing the shape of the vehicle in the state after the target process has been performed, and the second method is a method for determining at least one of the position and orientation using the distance point data as detection data and fourth reference data which is 3D CAD data representing the shape of the vehicle in the state before the target process has been performed. In this configuration, different methods are used to calculate the vehicle's position and orientation using distance measurement point data acquired by the distance measuring device as an external sensor, depending on whether there are assembly defects in the parts or not. Therefore, even if there are assembly defects in the parts installed on the vehicle, a decrease in the accuracy of detecting the vehicle's position and orientation can be suppressed.

[0007] Furthermore, this disclosure can be implemented in various forms, for example, as a remote control system, a mobile device control device, a remote automatic driving method, and a method for manufacturing a mobile device. [Brief explanation of the drawing]

[0008] [Figure 1] This is a conceptual diagram showing the system configuration in the embodiment. [Figure 2] This is a block diagram showing the system configuration. [Figure 3] This is a flowchart showing the processing procedure for vehicle driving control in the embodiment. [Figure 4] Figure 3 is a flowchart showing the detailed processing steps for step S1. [Figure 5] This diagram illustrates the advantages of template matching using 3D CAD data in another embodiment (B4). [Figure 6] This is an explanatory diagram regarding the procedure for moving a vehicle into an evacuation area. [Modes for carrying out the invention]

[0009] A. Embodiments: A1. Remote control system configuration: Figure 1 is a conceptual diagram showing the configuration of system 50 in an embodiment. System 50 is used in a factory fuel cell (FC) that manufactures vehicles 100. Vehicle 100 is an electric vehicle (BEV: Battery Electric Vehicle). System 50 comprises one or more vehicles 100 as mobile units, a server 200, and a plurality of external sensors 300. The server 200 is also called a "control device". The external sensors 300 are cameras that photograph the vehicles 100. In this disclosure, "mobile unit" means an object that can move, such as a vehicle or an electric vertical take-off and landing aircraft (so-called flying car). A vehicle may be a wheeled vehicle or a tracked vehicle, such as a passenger car, truck, bus, motorcycle, car, tank, or construction vehicle. Vehicles include electric vehicles (BEV: Battery Electric Vehicle), gasoline vehicles, hybrid vehicles, and fuel cell vehicles. If the moving object is not a vehicle, the terms "vehicle" and "car" in this disclosure may be replaced with "moving object" as appropriate, and the term "driving" may be replaced with "moving" as appropriate.

[0010] Vehicle 100 is capable of unmanned operation. "Unmanned operation" means operation without the operation of a passenger. Operation refers to operations related to at least one of the following: "going," "turning," or "stopping" of vehicle 100. Unmanned operation is achieved by automatic or manual remote control using a device located outside vehicle 100, or by autonomous control of vehicle 100. Vehicle 100 operating unmanned may have passengers on board who do not perform operation. Passengers who do not perform operation include, for example, people simply sitting in the seats of vehicle 100, or people performing tasks other than operation, such as assembly, inspection, or operating switches, while on board vehicle 100. Operation by a passenger is sometimes called "manned operation."

[0011] Vehicle 100 is in a manufacturing state and travels autonomously within the factory FC where vehicle 100 is manufactured. The reference coordinate system of factory FC is the global coordinate system GC. That is, any position within factory FC is represented by X, Y, Z coordinates in the global coordinate system GC. Factory FC comprises a first location PL1 and a second location PL2. The first location PL1 and the second location PL2 are connected by a track TR on which vehicle 100 can travel. Multiple external sensors 300 are installed along the track TR in factory FC. The positions of each external sensor 300 in factory FC are pre-adjusted. Vehicle 100 moves autonomously from the first location PL1 to the second location PL2 via the track TR.

[0012] The first location PL1 is where the assembly of the vehicle 100 is carried out. For example, at the first location PL1, the assembly of parts is carried out by an assembly robot (not shown). The vehicle 100 assembled at the first location PL1 is in a state where it can be driven autonomously, in other words, it is in a state where it can perform the three functions of "driving," "turning," and "stopping" autonomously. In this embodiment, the vehicle 100 assembled at the first location PL1 travels autonomously from the first location PL1 to the second location PL2 in the form of a platform having the configuration described below. Specifically, in order for the vehicle 100 to perform the three functions of "driving," "turning," and "stopping" autonomously, it is sufficient to have at least a vehicle control device 110 and an actuator group 120. When the vehicle 100 acquires information from the outside for autonomous driving, the vehicle 100 may further be equipped with a communication device 130. In other words, the autonomously operated vehicle 100 may not have at least some of its interior components, such as the driver's seat and dashboard, or at least some of its exterior components, such as the bumper and fenders, or it may not have a body shell. In this case, the remaining components, such as the body shell, may be attached to the vehicle 100 before it is shipped from the factory fuel cell, or the remaining components, such as the body shell, may be attached to the vehicle 100 after it has been shipped from the factory fuel cell, while the remaining components, such as the body shell, are not attached to the vehicle 100. Each component may be attached from any direction, such as the top, bottom, front, rear, right, or left side of the vehicle 100, and they may be attached from the same direction or from different directions.

[0013] At location PL2, additional parts are assembled onto vehicle 100 by an assembly robot (not shown).

[0014] FIG. 2 is a block diagram showing the configuration of system 50. A vehicle 100 includes a vehicle control device 110 for controlling each part of the vehicle 100, an actuator group 120 including one or more actuators driven under the control of the vehicle control device 110, and a communication device 130 for performing wireless communication with an external device such as a server 200. The actuator group 120 includes an actuator for a driving device that accelerates the vehicle 100, an actuator for a steering device that changes the traveling direction of the vehicle 100, and an actuator for a braking device that decelerates the vehicle 100.

[0015] The vehicle control device 110 is configured by a computer including a processor 111, a memory 112, an input / output interface 113, and an internal bus 114. The processor 111, the memory 112, and the input / output interface 113 are communicatively connected bidirectionally via the internal bus 114. The actuator group 120 and the communication device 130 are connected to the input / output interface 113. The processor 111 implements various functions including the function as a vehicle control unit 115 by executing a program PG1 stored in the memory 112.

[0016] The vehicle control unit 115 can cause the vehicle 100 to travel by controlling the actuator group 120 using a travel control signal received from the server 200. The travel control signal is a control signal for causing the vehicle 100 to travel. In the present embodiment, the travel control signal includes the acceleration and steering angle of the vehicle 100 as parameters. Alternatively, the travel control signal may include the speed of the vehicle 100 as a parameter instead of or in addition to the acceleration of the vehicle 100.

[0017] The server 200 is configured by a computer comprising a processor 201, a memory 202, an input / output interface 203, and an internal bus 204. The processor 201, the memory 202, and the input / output interface 203 are bidirectionally communicably connected via the internal bus 204. A communication device 205 for communicating with various devices external to the server 200 is connected to the input / output interface 203. The communication device 205 can communicate with the vehicle 100 via wireless communication, and can communicate with each external sensor 300 via wired communication or wireless communication.

[0018] The memory 202 stores in advance a program PG2, a first detection model DM1 and a second detection model DM2 which will be described later, a reference route RR indicating a route on which the vehicle 100 should travel, and the like. The processor 201 executes the program PG2 stored in the memory 202, thereby realizing various functions including functions as a sensor acquisition unit 210, an estimation unit 220, a defect determination unit 230, a method selection unit 240, and a remote control unit 250.

[0019] The sensor acquisition unit 210 acquires detection results output from an external sensor 300 which will be described later. In the present embodiment, the sensor acquisition unit 210 acquires a captured image of the vehicle 100 captured by a camera that is the external sensor 300. The captured image is also referred to as "detection data". The estimation unit 220 estimates the position and orientation of the vehicle 100 using the detection result output from the external sensor 300. Alternatively, the estimation unit 220 may estimate only one of the position and the orientation of the vehicle 100 using the detection result output from the external sensor 300. In this case, for example, the other of the position and the orientation of the vehicle 100 is determined using the travel history or the like of the vehicle 100. The defect determination unit 230 determines whether there is a defect in assembling components to the vehicle 100. The defect determination unit 230 is also referred to as a "determination unit". The method selection unit 240 selects a method for estimating the position and orientation of the vehicle 100 in accordance with the presence or absence of a component assembly defect in the vehicle 100. The estimation unit 220 and the method selection unit 240 are also collectively referred to as a "calculation unit".

[0020] The remote control unit 250 acquires detection results from sensors, generates a driving control signal to control the actuator group 120 of the vehicle 100 using the detection results, and transmits the driving control signal to the vehicle 100, thereby driving the vehicle 100 by remote control. The remote control unit 250 may generate and output not only driving control signals, but also control signals to operate various auxiliary equipment and actuators that operate various devices such as wipers, power windows, and lamps, which are provided on the vehicle 100. In other words, the remote control unit 250 may operate these various devices and auxiliary equipment by remote control. In this specification, "remote control" includes "fully remote control," in which all operations of the vehicle 100 are completely determined from outside the vehicle 100, and "partial remote control," in which some operations of the vehicle 100 are determined from outside the vehicle 100.

[0021] The external sensor 300 is a sensor located outside the vehicle 100. The external sensor 300 is a sensor that detects the vehicle 100 from outside the vehicle 100. The external sensor 300 is equipped with a communication device (not shown) and can communicate with other devices such as the server 200 via wired or wireless communication. Specifically, the external sensor 300 consists of a camera installed on the premises of the factory fuel cell. The camera as the external sensor 300 captures an image including the vehicle 100 and outputs the captured image as the detection result.

[0022] Figure 3 is a flowchart showing the processing procedure for controlling the movement of vehicle 100. The processing shown in Figure 3 is executed by the processor 201 of the server 200, which functions as a remote control unit 250, and the processor 111 of vehicle 100, which functions as a vehicle control unit 115. The processing shown in Figure 3 is repeatedly executed at predetermined time intervals, for example, from the moment vehicle 100 starts moving under remote control.

[0023] In step S1, the processor 201 of the server 200 acquires vehicle position information of the vehicle 100 using the detection results output from the external sensor 300. The vehicle position information is the position information that forms the basis for generating the driving control signal. In this embodiment, the vehicle position information includes the position and orientation of the vehicle 100 in the global coordinate system GC of the factory FC. Specifically, in step S1, the processor 201 acquires vehicle position information using the detection results acquired by the external sensor 300, i.e., the captured image acquired by the camera. Details of the method for detecting the position and orientation of the vehicle 100 will be described later.

[0024] In step S2, the processor 201 of the server 200 determines the next target location that the vehicle 100 should head to. In this embodiment, the target location is represented by X, Y, Z coordinates in the global coordinate system GC. The memory 202 of the server 200 pre-stores a reference route RR, which is the path that the vehicle 100 should travel. The route is represented by a node indicating the starting point, nodes indicating waypoints, a node indicating the destination, and links connecting each node. The processor 201 uses the vehicle position information and the reference route RR to determine the next target location that the vehicle 100 should head to. The processor 201 determines the target location on the reference route RR beyond the current location of the vehicle 100.

[0025] In step S3, the processor 201 of the server 200 generates a driving control signal to drive the vehicle 100 toward the determined target position. The processor 201 calculates the vehicle's speed from the change in the vehicle's position and compares the calculated speed with the target speed. Overall, the processor 201 determines the acceleration so that the vehicle 100 accelerates if the speed is lower than the target speed, and determines the acceleration so that the vehicle 100 decelerates if the speed is higher than the target speed. Furthermore, if the vehicle 100 is located on the reference path RR, the processor 201 determines the steering angle and acceleration so that the vehicle 100 does not deviate from the reference path RR, and if the vehicle 100 is not located on the reference path RR, in other words, if the vehicle 100 has deviated from the reference path RR, the processor 201 determines the steering angle and acceleration so that the vehicle 100 returns to the reference path RR.

[0026] In step S4, the processor 201 of the server 200 transmits the generated driving control signal to the vehicle 100. The processor 201 repeats the process of acquiring the position of the vehicle 100, determining the target position, generating the driving control signal, and transmitting the driving control signal at predetermined intervals.

[0027] In step S5, the processor 111 of the vehicle 100 receives a driving control signal transmitted from the server 200. In step S6, the processor 111 of the vehicle 100 controls the actuator group 120 using the received driving control signal, thereby driving the vehicle 100 at the acceleration and steering angle indicated in the driving control signal. The processor 111 repeats the reception of the driving control signal and the control of the actuator group 120 at predetermined intervals. According to the system 50 in this embodiment, the vehicle 100 can be driven by remote control, and the vehicle 100 can be moved without using transport equipment such as cranes or conveyors.

[0028] Figure 4 is a flowchart showing the specific processing of step S1 in Figure 3. The processing shown in Figure 4 is executed by the processor 201 of the server 200, which functions as a sensor acquisition unit 210, an estimation unit 220, a defect determination unit 230, and a method selection unit 240. In this embodiment, if there is an assembly defect in the parts to be assembled to the vehicle 100, the method for calculating the position and orientation of the vehicle 100 is changed.

[0029] In step S11, an image of the vehicle 100 acquired by the external sensor 300 is used to determine whether the parts to be installed in the target process have been correctly assembled to the vehicle 100. The target process is, for example, the process performed immediately before. For the sake of ease of understanding the technology, this specification assumes that the processes prior to the target process have been performed correctly. For example, if the process performed immediately before is called process P2, and the process before process P2 is called process P1, then the vehicle 100 targeted by process P2 is a vehicle in which the parts were correctly installed in process P1. For example, a vehicle 100 in which there was a defect in the assembly of parts in process P1 is removed from the line before process P2 is performed.

[0030] For example, a pattern matching method using pre-prepared reference images can be used to determine whether the parts are correctly assembled based on whether the captured image and the reference image are similar. As a pre-prepared reference image, for example, an image of a vehicle 100 in which the parts were correctly assembled in the immediately preceding process can be used, captured by an external sensor 300. The reference image and the captured image are acquired, for example, by the same external sensor 300. If the similarity between the captured image and the reference image is above a predetermined threshold, it is determined that the parts are correctly assembled.

[0031] Furthermore, for example, it may be determined whether or not the parts are correctly assembled by a pattern matching method using multiple images acquired by the same external sensor 300 over a predetermined period. The multiple images acquired over a predetermined period may be multiple images acquired consecutively in a time series. If the determination that the parts are not correctly assembled is obtained based on a predetermined number of images from the multiple images, it may be determined that the parts are not correctly assembled.

[0032] Furthermore, for example, whether or not the parts are correctly assembled may be determined by a pattern matching method using multiple images acquired by multiple external sensors 300 at the same time. If the determination that the parts are not correctly assembled is obtained based on a predetermined number of images from the multiple images, then it may be determined that the parts are not correctly assembled.

[0033] If the parts are correctly assembled to the vehicle 100 (step S11; YES), the process in step S12 is executed. If the parts are not correctly assembled to the vehicle 100 (step S11; NO), the process in step S13 is executed.

[0034] In step S12, the position of vehicle 100 is estimated using the first detection model DM1. The method of estimating the position of vehicle 100 using the first detection model DM1 is also called the "first method." The first detection model DM1 is also called the "first machine learning model." Specifically, the external shape of vehicle 100 is detected by first inputting the captured image into the first detection model DM1, which utilizes artificial intelligence. As the first detection model DM1, for example, a trained machine learning model that has been trained to achieve either semantic segmentation or instance segmentation can be used. As this machine learning model, for example, a convolutional neural network (CNN) trained by supervised learning using a training dataset can be used. The training dataset DS1 used to generate the first detection model DM1 has multiple training images including vehicle 100 in which the parts have been correctly assembled in the immediately preceding process, and labels indicating whether each region in the training image is a region indicating vehicle 100 or a region indicating something other than vehicle 100. During training, it is preferable that parameters be updated using backpropagation to reduce the error between the output result of the first detection model DM1 and the label. Furthermore, the coordinates in the image coordinate system representing the external shape detected using the first detection model DM1 are converted to coordinates in the global coordinate system GC. In this way, the position of the vehicle 100 in the global coordinate system GC is obtained.

[0035] Furthermore, the orientation of vehicle 100 is estimated based on the direction of a vector relating to the movement of vehicle 100, calculated using, for example, the optical flow method. In the optical flow method, the direction of the vector relating to the movement of vehicle 100 is estimated from the change in the position of the feature points of the moving object between frames of the captured image.

[0036] In step S13, the position and orientation of vehicle 100 are estimated using the second detection model DM2. The method of estimating the position of vehicle 100 using the second detection model DM2 is also called the "second method." The second detection model DM2 is also called the "second machine learning model." Specifically, the captured image is first input into the second detection model DM2, which utilizes artificial intelligence, to detect the external shape of vehicle 100. The difference from the processing in step S12 is that the second detection model DM2 is used as the machine learning model to detect the external shape of vehicle 100.

[0037] The training dataset DS2, used to generate the second detection model DM2, includes multiple training images of a vehicle 100 in which parts were not properly assembled in the immediately preceding process, and labels indicating whether each region in the training image represents a vehicle 100 or a region other than vehicle 100. Poor assembly of parts includes cases where a part is not attached to the vehicle 100 at all, where a part is attached to the vehicle 100 but in an incorrect position, or where a part is not sufficiently secured to the vehicle 100, resulting in looseness. In cases where multiple parts are assembled in the immediately preceding process, incorrect assembly of parts includes cases where one or more parts are not attached in the correct position.

[0038] Since various forms of incorrect assembly of parts are conceivable, for example, the assembly defects of parts that may occur in the target process may be classified into typical patterns, and multiple machine learning models generated using training sets containing training images corresponding to each pattern may be included in the second detection model DM2.

[0039] Furthermore, the coordinates in the image coordinate system representing the external shape detected using the second detection model DM2 are converted to coordinates in the global coordinate system GC. In this way, the position of vehicle 100 in the global coordinate system GC is obtained. The estimation of the orientation of vehicle 100 is the same as in step S12. The above describes the position and estimation process in step S1 shown in Figure 3.

[0040] According to this embodiment, different methods are used to determine the vehicle's position depending on whether there are assembly defects in the parts or not. Therefore, even if there are assembly defects in the parts assembled to the vehicle 100, a decrease in the accuracy of detecting the vehicle 100's position can be suppressed.

[0041] B. Other embodiments: (B1) The machine learning models used in cases where there are no assembly defects in the parts and cases where there are assembly defects in the parts are not limited to those described in the above embodiments.

[0042] If there are no assembly defects of the parts, the processor 201 may determine the position of the vehicle 100 using a third detection model DM3 generated by machine learning using a training dataset that includes images of the vehicle taken in the state after the target process has been performed, based on images of the vehicle as detection data acquired by a camera as an external sensor 300.

[0043] The training dataset DS3 used to generate the third detection model DM3 includes multiple training images containing the vehicle 100 in the state after the target process has been performed, and labels indicating whether each region in the training images represents the vehicle 100 or a region other than the vehicle 100. If the target process is the most recently performed process, the multiple training images containing the vehicle 100 in the state after the target process has been performed will be, as in the embodiment, multiple training images containing the vehicle 100 with parts correctly assembled in the most recently performed process.

[0044] Furthermore, if there is a defect in the assembly of parts, the processor 201 may determine the position of the vehicle 100 using a fourth detection model DM4 generated by machine learning using a training dataset that includes images of the vehicle taken before the target process was performed, based on the image of the vehicle as detection data.

[0045] The training dataset DS4 used to generate the fourth detection model DM4 consists of multiple training images containing vehicle 100 in the state before the target process is performed, and labels indicating whether each region in the training images represents vehicle 100 or something other than vehicle 100. If the target process is the most recently performed process, the multiple training images containing vehicle 100 in the state before the target process is performed are, for example, multiple training images containing vehicle 100 in the process immediately preceding the most recently performed process.

[0046] In another embodiment (B1), the method of estimating the position of vehicle 100 using the third detection model DM3 is also referred to as the "first method." The third detection model DM3 is also referred to as the "third machine learning model." The method of estimating the position of vehicle 100 using the fourth detection model DM4 is also referred to as the "second method." The fourth detection model DM4 is also referred to as the "fourth machine learning model." In this case, the effort of preparing training images of vehicles with faulty assembly of parts can be eliminated, and the training dataset can be easily prepared.

[0047] (B2) In addition, if there is a defect in the assembly of the parts, the processor 201 may use two or more machine learning models to determine the position of the vehicle 100 based on the image of the vehicle. For example, suppose two machine learning models are used. In this case, the processor 201 may use the fourth detection model DM4, which is generated in the same manner as described in (B1) above, as one of the two machine learning models.

[0048] Furthermore, the processor 201 may determine the position of the vehicle 100 using a fifth detection model DM5, which is generated by machine learning using a training dataset that includes images of the vehicle taken after the target process has been performed, based on images of the vehicle as detection data, as the other of the two machine learning models.

[0049] The training dataset DS5 used to generate the fifth detection model DM5 contains multiple training images including the vehicle 100 in the state after the target process has been performed, and labels indicating whether each region in the training images represents the vehicle 100 or a region other than the vehicle 100. If the target process is the most recent process, the multiple training images including the vehicle 100 in the state after the target process has been performed will be the same as the training images included in the training dataset DS3 in (D1) above. Therefore, the same third detection model DM3, which was generated as described in (D1) above, can be used as the fifth detection model DM5. This simplifies the effort required to generate machine learning models.

[0050] In the event of a faulty assembly of parts, the processor 201 may adopt the position of the vehicle 100 estimated using the fourth detection model DM4 and the position of the vehicle 100 estimated using the fifth detection model DM5, whichever has a higher confidence level, as the final estimated result.

[0051] Furthermore, if there are no assembly defects of the parts, the processor 201 can determine the position of the vehicle 100 using the third detection model DM3, similar to (B1) above.

[0052] (B3) In the above embodiment, the external sensor 300 is a camera, but the external sensor 300 may be a distance measuring device using LiDAR (Light Detection And Ranging) technology. In this case, the external sensor 300 acquires 3D point cloud data of the vehicle 100. 3D point cloud data is data that indicates the 3D position of the point cloud. 3D point cloud data is also called "detection data". 3D point cloud data is also called "distance measurement point data".

[0053] Furthermore, different methods are used to estimate at least one of the position and orientation of the vehicle 100 depending on whether or not there is a defect in the assembly of the parts.

[0054] If there are no assembly defects of the parts, the processor 201 estimates at least one of the position and orientation of the vehicle 100 by performing template matching using 3D point cloud data as detection data acquired by the distance measuring device as an external sensor 300 and the first vehicle point cloud data VP1, which is 3D CAD data representing the shape of the vehicle 100 with the parts correctly assembled in the target process.

[0055] On the other hand, in the event of a faulty assembly of parts, the processor 201 estimates at least one of the position and orientation of the vehicle 100 by performing template matching using 3D point cloud data as detection data acquired by a distance measuring device acting as an external sensor 300, and second vehicle point cloud data VP2, which is 3D CAD data representing the shape of the vehicle 100 in which the faulty assembly of parts occurred in the target process. The memory 112 pre-stores the first vehicle point cloud data VP1 and the second vehicle point cloud data VP2. Furthermore, the first vehicle point cloud data VP1 and the second vehicle point cloud data VP2 may also contain information for identifying the orientation of the vehicle 100.

[0056] Furthermore, in the relevant process, for example, if multiple parts are assembled, there are multiple patterns of assembly defects. For this reason, the second vehicle point cloud data VP2 may include multiple 3D CAD data representing multiple assembly defect patterns. For example, if three parts are assembled in the relevant process, the second vehicle point cloud data VP2 may include 3D CAD data representing the shape of vehicle 100 with only one part assembled, and 3D CAD data representing the shape of vehicle 100 with two parts assembled.

[0057] The method for estimating the position and orientation of vehicle 100 using the first vehicle point cloud data VP1 is also called the "first method." The first vehicle point cloud data VP1 is also called the "first reference data." The method for estimating the position and orientation of vehicle 100 using the second vehicle point cloud data VP2 is also called the "second method." The second vehicle point cloud data VP2 is also called the "second reference data." In this specification, 3D CAD data representing the shape of a vehicle, such as the first vehicle point cloud data VP1 used in template matching, may be referred to as reference data.

[0058] (B4) When the external sensor 300 is a distance measuring device using LiDAR technology, the position and orientation of the vehicle 100 may be estimated as follows.

[0059] If there are no assembly defects in the parts, the processor 201 estimates at least one of the position and orientation of the vehicle 100 by performing template matching using the 3D point cloud data as detection data acquired by the distance measuring device as an external sensor 300 and the third vehicle point cloud data VP3, which is 3D CAD data representing the shape of the vehicle 100 after the target process has been performed. The target process is, for example, the process performed immediately before. The third vehicle point cloud data VP3, which is 3D CAD data representing the shape of the vehicle 100 after the target process has been performed, may be the same as the first vehicle point cloud data VP1, which is 3D CAD data representing the shape of the vehicle 100 with the parts correctly assembled in the target process, as described in (B3) above.

[0060] On the other hand, if there is a defect in the assembly of the parts, the processor 201 estimates at least one of the position and orientation of the vehicle 100 by performing template matching using the 3D point cloud data as detection data acquired by the distance measuring device as an external sensor 300 and the fourth vehicle point cloud data VP4, which is 3D CAD data representing the shape of the vehicle 100 in the state before the target process is performed. The memory 112 pre-stores the third vehicle point cloud data VP3 and the fourth vehicle point cloud data VP4. In addition, the third vehicle point cloud data VP3 and the fourth vehicle point cloud data VP4 may contain information for identifying the orientation of the vehicle 100.

[0061] The method of estimating the position and orientation of vehicle 100 using the third vehicle point cloud data VP3 is also called the "first method." The third vehicle point cloud data VP3 is also called the "third reference data." The method of estimating the position and orientation of vehicle 100 using the fourth vehicle point cloud data VP4 is also called the "second method." The fourth vehicle point cloud data VP4 is also called the "fourth reference data." In this case, the effort of preparing CAD data for vehicles with faulty assembly of parts can be avoided.

[0062] Figure 5 illustrates the advantages of performing template matching using 3D CAD data representing the shape of vehicle 100 after the target process has been performed, and 3D CAD data representing the shape of vehicle 100 before the target process is performed. For example, assume that processes A, B, and C are performed in that order, and that parts are assembled to vehicle 100 in each process. Furthermore, assume that there are no other processes between process A and process B, and no other processes between process B and process C.

[0063] In process A, if there are no assembly defects of the parts, the 3D CAD data representing the shape of vehicle 100 after process A is performed is used as reference data in template matching.

[0064] In process B, if there are assembly defects in the parts, 3D CAD data representing the shape of vehicle 100 before process B is performed is used as reference data. As shown in the figure, the 3D CAD data representing the shape of vehicle 100 before process B is performed is the same as the 3D CAD data representing the shape of vehicle 100 after process A is performed.

[0065] In process C, if there are assembly defects in the parts, 3D CAD data representing the shape of vehicle 100 before process C is performed is used as reference data. As shown in the figure, the 3D CAD data representing the shape of vehicle 100 before process C is performed is the same as the 3D CAD data representing the shape of vehicle 100 after process B is performed.

[0066] In this way, 3D CAD data representing the shape of vehicle 100 after the target process has been performed and 3D CAD data representing the shape of vehicle 100 before the target process has been performed can be shared as reference data. Therefore, the preparation of the 3D CAD data used as reference data can be simplified.

[0067] (B5) In addition, if there is a defect in the assembly of the parts, the processor 201 may use two or more reference data to determine the position and orientation of the vehicle 100. For example, suppose two reference data are used. In this case, the processor 201 may use the fourth vehicle point cloud data VP4, which is 3D CAD data representing the shape of the vehicle 100 in the state before the target process is performed, as one of the two reference data, as described in (B4) above.

[0068] Furthermore, the processor 201 can use 3D CAD data representing the shape of the vehicle 100 after the target process has been performed as the other of the two reference data. This 3D CAD data is the same as the third vehicle point cloud data VP3 described in (B4) above.

[0069] If there is a defect in the assembly of the parts, the processor 201 may adopt as the final estimation result the one with a higher similarity to the reference data between the estimation result of the position and orientation of the vehicle 100 estimated using the fourth vehicle point cloud data VP4 and the estimation result of the position and orientation of the vehicle 100 estimated using the third vehicle point cloud data VP3.

[0070] Furthermore, if there are no assembly defects of the parts, the processor 201 can estimate the position and orientation of the vehicle 100 in the same manner as in (B4) above.

[0071] (B6) In this embodiment, an example was described in which the position of the vehicle 100 is estimated using an image acquired by a camera, which is an external sensor 300. Alternatively, a 3D point cloud representing a 3D space can be reconstructed using multiple images acquired by multiple external sensors 300. The position and orientation of the vehicle 100 may be estimated by template matching using the reconstructed 3D point cloud data and pre-prepared vehicle point cloud data. In this case, a first vehicle point cloud data VP1, which is 3D CAD data representing the shape of the vehicle 100 with parts correctly assembled in the target process, and a second vehicle point cloud data VP2, which is 3D CAD data representing the shape of the vehicle 100 with assembly defects in the target process, are prepared in advance. The first vehicle point cloud data VP1 and the second vehicle point cloud data VP2 contain information for identifying the orientation of the vehicle 100.

[0072] Therefore, if the parts are correctly assembled to the vehicle 100 in the target process, the position and orientation of the vehicle 100 can be estimated by performing template matching using the first vehicle point cloud data V1. On the other hand, if there is a defect in the assembly of the parts in the target process, the position and orientation of the vehicle 100 can be estimated by performing template matching using the second vehicle point cloud data V2. Furthermore, if, for example, multiple parts are assembled in the target process, there are multiple patterns of defective part assembly. For this reason, multiple 3D CAD data representing multiple assembly defect patterns may be prepared as the second vehicle point cloud data VP2.

[0073] Alternatively, as explained in (B4) above, if there are no assembly defects of the parts, template matching is performed using 3D CAD data representing the shape of the vehicle 100 after the target process has been carried out. If there are assembly defects of the parts, template matching is performed using 3D CAD data representing the shape of the vehicle 100 before the target process has been carried out.

[0074] (B7) In the embodiment, after determining whether or not there is a defect in the assembly of the parts (see step S11 in Figure 4), if there is no defect in the assembly of the parts, the position of the vehicle 100 is estimated using the first detection model DM1, and if there is a defect in the assembly of the parts, the position of the vehicle 100 is estimated using the second detection model DM2.

[0075] Alternatively, the position of vehicle 100 may be estimated first using the first detection model DM1 without determining whether there are any assembly defects in the parts. If the score value indicating the reliability of the estimation result is below a predetermined threshold, the position of vehicle 100 is estimated using the second detection model DM2. This is because if the score value indicating the reliability of the estimation result using the first detection model DM1 is below the threshold, it is assumed that the parts were not properly assembled in the previous process. On the other hand, if the score value indicating the reliability of the estimation result of the position of vehicle 100 using the first detection model DM1 is above the threshold, it is assumed that the parts were properly assembled in the previous process. In this case, the estimation process using the second detection model DM2 is not performed.

[0076] (B8) In this embodiment, an example was described in which the presence or absence of assembly defects in the parts is determined using images (see step S11 in Figure 4). After each process is completed, for example, if the worker outputs whether or not there are assembly defects in the parts to the upper-level server using a terminal device, the processor 201 can determine whether or not there are assembly defects in the parts by querying the upper-level server about the presence or absence of assembly defects in the parts in the immediately preceding process.

[0077] (B9) After estimating the location of the vehicle 100 where a component assembly defect occurred, the processor 201 of the server 200 can use the vehicle location information to determine the evacuation area EA as the target location to which the vehicle 100 should next go. The memory 202 of the server 200 is assumed to have the evacuation area EA, which is the area to which the vehicle 100 should be evacuated, stored in advance.

[0078] Figure 6 is an explanatory diagram of the case where a vehicle is moved to the evacuation area EA. The processor 201 generates a driving control signal to drive the vehicle 100 toward the target position, the evacuation area EA. Furthermore, the processor 201 may notify at least one worker WO of the instruction to retrieve the vehicle 100, along with information identifying the vehicle 100, either before moving the vehicle 100 to the evacuation area EA or after moving the vehicle 100 to the evacuation area EA. For example, the processor 201 notifies the worker WO via a terminal device T1 owned by the worker WO and a terminal device T2 installed in the waiting area OF. The waiting area OF is a place where the worker WO waits or rests. At least one worker WO who receives the notification performs the retrieval work of the vehicle 100 that has been moved to the evacuation area EA. By removing the vehicle 100 that has a faulty assembly of parts from the production line, it is possible to suppress the occurrence of interference with the movement of other vehicles 100 and delays in subsequent processes.

[0079] (B10) In the above embodiment, the server 200 automatically generates a driving control signal to be transmitted to the vehicle 100. Alternatively, the server 200 may generate a driving control signal to be transmitted to the vehicle 100 in accordance with the operation of an external operator located outside the vehicle 100. For example, the external operator may operate a control device that includes a display for displaying captured images output from the external sensor 300, a steering wheel for remotely controlling the vehicle 100, an accelerator pedal, a brake pedal, and a communication device for communicating with the server 200 via wired or wireless communication, and the server 200 may generate a driving control signal in accordance with the operation applied to the control device.

[0080] (B11) Vehicle 100 may be manufactured by combining multiple modules. A module means a unit composed of multiple parts grouped together according to the part or function of the vehicle 100. For example, the platform of vehicle 100 may be manufactured by combining a front module that constitutes the front part of the platform, a central module that constitutes the central part of the platform, and a rear module that constitutes the rear part of the platform. The number of modules that constitute the platform is not limited to three, but may be two or fewer, or four or more. In addition to, or instead of, the parts that constitute the platform may be modularized, as well as parts that constitute parts of the vehicle 100 that are different from the platform. Various modules may also include any exterior parts such as bumpers and grilles, or any interior parts such as seats and consoles. Furthermore, not limited to vehicle 100, any type of mobile body may be manufactured by combining multiple modules. Such modules may be manufactured, for example, by joining multiple parts by welding or fasteners, or by integrally molding at least a part of the parts that constitute the module as a single part by casting. A molding technique for integrally molding a single component, especially a relatively large component, is also called gigacast or megacast. For example, the front module, central module, and rear module mentioned above may be manufactured using gigacast.

[0081] (B12) Transporting vehicle 100 using the unmanned operation of vehicle 100 is also called "autonomous transport." The configuration for realizing autonomous transport is also called a "vehicle remote control autonomous driving transport system." Furthermore, a production method that uses autonomous transport to produce vehicle 100 is also called "autonomous production." In autonomous production, for example, at a factory FC that manufactures vehicle 100, at least a portion of the transport of vehicle 100 is realized by autonomous transport.

[0082] (B13) The means of realizing the functions of server 200 are not limited to software, and some or all of them may be realized by dedicated hardware. For example, circuits such as FPGA (Field Programmable Gate Array) or ASIC (Application Specific Integrated Circuit) may be used as dedicated hardware.

[0083] This disclosure is not limited to the embodiments described above, and can be implemented in various configurations without departing from its spirit. For example, the technical features in the embodiments corresponding to the technical features in each form described in the summary of the invention can be replaced or combined as appropriate in order to solve the problems described above or to achieve some or all of the effects described above. Furthermore, if a technical feature is not described as essential in this specification, it can be deleted as appropriate. [Explanation of Symbols]

[0084] 50...System, 100...Vehicle, 110...Vehicle control device, 111...Processor, 112...Memory, 113...Input / Output interface, 114...Internal bus, 115...Vehicle control unit, 120...Actuator group, 130...Communication device, 200...Server, 201...Processor, 202...Memory, 203...Input / Output interface, 204...Internal bus, 205...Communication device, 210...Sensor acquisition unit, 220...Estimation unit, 230...Defect discrimination unit, 240...Method selection unit, 250...Remote control unit, 300...External sensor, DM1...First detection model, DM2...Second detection model, EA...Evacuation area, FC...Factory, GC...Global coordinate system, OF...Waiting area, PG1...Program, PG2...Program, PL1...First location, PL2...Second location, RR...Reference path, T1, T2...Terminal devices, TR...Track, WO...Worker

Claims

1. A vehicle that travels within a factory where a plurality of processes are carried out for the manufacture of a vehicle, and a control device for remotely controlling the vehicle which is the subject of the plurality of processes, A determination unit for determining whether or not the parts have been correctly assembled to the vehicle, It is an arithmetic unit, When the parts are correctly assembled to the vehicle, the position and orientation of the vehicle are obtained using the first method based on detection data acquired by an external sensor. If the parts are not properly assembled to the vehicle, the position and orientation are determined using a second method different from the first method, based on the detection data. The calculation unit and Equipped with, The first method is a method for determining at least one of the position and orientation using a first machine learning model generated by machine learning using a training dataset that includes images of the vehicle with the parts correctly assembled, based on images of the vehicle as detection data acquired by the camera as an external sensor. The second method is a method for determining at least one of the position and orientation using a second machine learning model generated by machine learning using a training dataset that includes images of the vehicle in which the parts are not properly assembled, based on the images of the vehicle as detection data. Control device.

2. A vehicle that travels within a factory where a plurality of processes are carried out for the manufacture of a vehicle, and a control device for remotely controlling the vehicle which is the subject of the plurality of processes, A determination unit for determining whether or not the parts have been correctly assembled to the vehicle, It is an arithmetic unit, When the parts are correctly assembled to the vehicle, the position and orientation of the vehicle are obtained using the first method based on detection data acquired by an external sensor. If the parts are not properly assembled to the vehicle, the position and orientation are determined using a second method different from the first method, based on the detection data. The calculation unit and Equipped with, The first method is a method for determining at least one of the position and orientation using a third machine learning model generated by machine learning using a training dataset that includes images of the vehicle taken in the state after the target process has been performed, based on the image of the vehicle as detection data acquired by the camera as an external sensor. The second method is a method for determining at least one of the position and orientation using a fourth machine learning model generated by machine learning using a training dataset that includes images of the vehicle taken in the state before the target process is performed, based on the images of the vehicle as detection data. Control device.

3. A vehicle that travels within a factory where a plurality of processes are carried out for the manufacture of a vehicle, and a control device for remotely controlling the vehicle which is the subject of the plurality of processes, A determination unit for determining whether or not the parts have been correctly assembled to the vehicle, It is an arithmetic unit, When the parts are correctly assembled to the vehicle, the position and orientation of the vehicle are obtained using the first method based on detection data acquired by an external sensor. If the parts are not properly assembled to the vehicle, the position and orientation are determined using a second method different from the first method, based on the detection data. The calculation unit and Equipped with, The first method is a method for determining at least one of the position and orientation using distance point data as detection data acquired by the distance measuring device as an external sensor, and first reference data which is a three-dimensional CAD representing the shape of the vehicle with the parts correctly assembled. The second method is a method for determining at least one of the position and orientation using the distance measurement point data as detection data and second reference data which is three-dimensional CAD data representing the shape of the vehicle in which the parts are not properly assembled. Control device.

4. A vehicle that travels within a factory where a plurality of processes are carried out for the manufacture of a vehicle, and a control device for remotely controlling the vehicle which is the subject of the plurality of processes, A determination unit for determining whether or not the parts have been correctly assembled to the vehicle, It is an arithmetic unit, When the parts are correctly assembled to the vehicle, the position and orientation of the vehicle are obtained using the first method based on detection data acquired by an external sensor. If the parts are not properly assembled to the vehicle, the position and orientation are determined using a second method different from the first method, based on the detection data. The calculation unit and Equipped with, The first method is a method for determining at least one of the position and orientation using distance point data as detection data acquired by the distance measuring device as an external sensor, and third reference data which is three-dimensional CAD data representing the shape of the vehicle in the state after the target process has been performed. The second method is a method for determining at least one of the position and orientation using the distance measurement point data as detection data and a fourth reference data which is three-dimensional CAD data representing the shape of the vehicle in the state before the target process is performed. Control device.

Images