Control device

Abstract

To provide a technique capable of preventing detection accuracy of a position and an orientation of a vehicle from being lowered, even when the vehicle has an assembly failure of components assembled in the vehicle.SOLUTION: A control device for remotely controlling a vehicle that is an object for a plurality of steps, the vehicle traveling inside a factory in which the plurality of steps of manufacturing the vehicle are executed, comprises a determination unit and a calculation unit. The determination unit determines whether or not components are properly assembled to the vehicle. The calculation unit acquires at least one of a position and an orientation of the vehicle using a first method, based on detection data acquired by an external sensor when the components are properly assembled to the vehicle, and finds at least one of the position and the orientation using a second method different from the first method, based on the detection data when the components are not properly assembled to the vehicle.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present disclosure relates to a control device.

Background Art

[0002] Patent Document 1 describes a vehicle that is an object of manufacturing and travels by remote control in a manufacturing process for manufacturing a vehicle.

Prior Art Document

Patent Document

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In order to remotely control the running of a vehicle, it is necessary to detect the position and orientation of the vehicle. For this reason, the position and orientation of the vehicle may be detected based on the external shape of the vehicle. In this case, even if there is a defective assembly of the parts assembled to the vehicle, a technique capable of suppressing a decrease in the detection accuracy of the position and orientation of the vehicle has been demanded.

Means for Solving the Problems

[0005] The present disclosure can be realized in the following forms.

[0006] (1) According to the first aspect of the present disclosure, a control device is provided. This control device is a vehicle that travels within a factory where a plurality of processes are carried out to manufacture a vehicle, and is a control device that remotely controls the vehicle that is the object of the plurality of processes. The control device includes a determination unit that determines whether parts are correctly assembled to the vehicle, and a calculation unit. When the parts are correctly assembled to the vehicle, based on detection data acquired by an external sensor, using a first method, at least one of the position and orientation of the vehicle is acquired. When the parts are not correctly assembled to the vehicle, based on the detection data, using a second method different from the first method, at least one of the position and orientation is obtained. According to this aspect, different methods are used to calculate at least one of the position and orientation of the vehicle when there is no defective assembly of parts and when there is a defective assembly of parts. Therefore, even if there is a defective assembly of parts assembled to the vehicle, a decrease in the detection accuracy of the position and orientation of the vehicle can be suppressed. (2) In the control device of the above aspect, the first method may be a method of obtaining at least one of the position and orientation using a first machine learning model generated by machine learning using a learning data set including an image of the vehicle in which the parts are correctly assembled, based on the image of the vehicle as the detection data acquired by a camera as the external sensor. The second method may be a method of obtaining at least one of the position and orientation using a second machine learning model generated by machine learning using a learning data set including an image of the vehicle in which the parts are not correctly assembled, based on the image of the vehicle as the detection data. According to this aspect, different machine learning models are used to calculate at least one of the position and orientation of the vehicle using an image of the vehicle acquired by a camera as an external sensor when there is no defective assembly of parts and when there is a defective assembly of parts. Therefore, even if there is a defective assembly of parts assembled to the vehicle, a decrease in the detection accuracy of the position and orientation of the vehicle can be suppressed. (3) In the control device of the above-described embodiment, the first method is a method of obtaining at least one of the position and the orientation by using a third machine learning model generated by machine learning using a learning data set including an image of the vehicle in a state after a target process is performed, based on the image of the vehicle as the detection data acquired by a camera as the external sensor. The second method may be a method of obtaining at least one of the position and the orientation by using a fourth machine learning model generated by machine learning using a learning data set including an image of the vehicle in a state before the target process is performed, based on the image of the vehicle as the detection data. According to this embodiment, different machine learning models are used to calculate at least one of the position and the orientation of the vehicle by using an image of the vehicle acquired by a camera as an external sensor, in the case where there is no defective assembly of parts and in the case where there is a defective assembly of parts. Therefore, even if there is a defective assembly of parts assembled to the vehicle, a decrease in the detection accuracy of the position and the orientation of the vehicle can be suppressed. (4) In the control device of the above-described embodiment, the first method is a method of obtaining at least one of the position and the orientation by using the distance measurement point data as the detection data acquired by a distance measurement device as the external sensor and first reference data which is a 3D CAD representing the shape of the vehicle with the parts correctly assembled. The second method may be a method of obtaining at least one of the position and the orientation by using the distance measurement point data as the detection data and second reference data which is 3D CAD data representing the shape of the vehicle with the parts not correctly assembled. According to this embodiment, different methods are used to calculate the position and the orientation of the vehicle by using the distance measurement point data acquired by a distance measurement device as an external sensor, in the case where there is no defective assembly of parts and in the case where there is a defective assembly of parts. Therefore, even if there is a defective assembly of parts assembled to the vehicle, a decrease in the detection accuracy of the position and the orientation of the vehicle can be suppressed. (5) In the control device of the above-described embodiment, the first method may be a method of obtaining at least one of the position and the orientation using the distance measurement point data as the detection data acquired by the distance measurement device as the external sensor and the third reference data which is the 3D CAD data representing the shape of the vehicle in the state after the target process is performed. The second method may be a method of obtaining at least one of the position and the orientation using the distance measurement point data as the detection data and the fourth reference data which is the 3D CAD data representing the shape of the vehicle in the state before the target process is performed. According to this embodiment, different methods are used to calculate the position and orientation of the vehicle using the distance measurement point data acquired by the distance measurement device as the external sensor, depending on whether there is a defective assembly of components or not. Therefore, even if there is a defective assembly of components to be assembled to the vehicle, a decrease in the detection accuracy of the position and orientation of the vehicle can be suppressed.

[0007] Note that the present disclosure can be realized in various forms. For example, it can be realized in forms such as a remote operation system, a movement control device, a remote automatic driving method, and a manufacturing method of a moving body.

Brief Description of Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Mode for Carrying Out the Invention

[0009] A. Embodiment: A1. Remote operation system configuration: FIG. 1 is a conceptual diagram showing the configuration of a system 50 in an embodiment. The system 50 is used in a factory FC that manufactures a vehicle 100. The vehicle 100 is a battery electric vehicle (BEV). The system 50 includes one or more vehicles 100 as moving bodies, a server 200, and a plurality of external sensors 300. The server 200 is also referred to as a "control device". The external sensor 300 is a camera that photographs the vehicle 100. In the present disclosure, a "moving body" means an object that can move, and for example, a vehicle or an electric vertical take-off and landing aircraft (so-called flying car). The vehicle may be a vehicle that travels on wheels or a vehicle that travels on an endless track, and for example, a passenger car, a truck, a bus, a two-wheeled vehicle, a four-wheeled vehicle, a tank, a construction vehicle, etc. The vehicle includes a battery electric vehicle (BEV), a gasoline vehicle, a hybrid vehicle, and a fuel cell vehicle. When the moving body is other than a vehicle, the expressions "vehicle" and "car" in the present disclosure can be appropriately replaced with "moving body", and the expression "travel" can be appropriately replaced with "move".

[0010] Vehicle 100 is capable of traveling through driverless operation. "Driverless operation" means operation without relying on the driving operation of a passenger. The driving operation means an operation related to at least any one of "running", "turning", and "stopping" of vehicle 100. Driverless operation is realized by automatic or manual remote control using a device located outside vehicle 100, or by autonomous control of vehicle 100. A passenger who does not perform a driving operation may board vehicle 100 while it is traveling through driverless operation. Passengers who do not perform a driving operation include, for example, a person simply sitting on the seat of vehicle 100, or a person performing work different from the driving operation, such as assembly, inspection, and operation of switches, while boarding vehicle 100. Note that the operation by the passenger's driving operation may be called "drivered operation".

[0011] Vehicle 100 is in a state during manufacturing and travels through driverless operation within the factory FC where vehicle 100 is manufactured. The reference coordinate system of factory FC is the global coordinate system GC. That is, any position within factory FC is expressed by the coordinates of X, Y, and Z in the global coordinate system GC. Factory FC includes a first location PL1 and a second location PL2. The first location PL1 and the second location PL2 are connected by a road TR on which vehicle 100 can travel. A plurality of external sensors 300 are installed along the road TR in factory FC. The positions of the respective external sensors 300 in factory FC are adjusted in advance. Vehicle 100 moves from the first location PL1 to the second location PL2 through the road TR by driverless operation.

[0012] The first location PL1 is a place where the operation of assembling the vehicle 100 is carried out. For example, at the first location PL1, the operation of assembling parts is carried out by an assembly robot (not shown). The vehicle 100 assembled at the first location PL1 is in a state where it can run by autonomous driving, in other words, it can exhibit the three functions of "running", "turning", and "stopping" by autonomous driving. In the present embodiment, the vehicle 100 assembled at the first location PL1 travels from the first location PL1 to the second location PL2 by autonomous driving in the form of a platform having the configuration described below. Specifically, the vehicle 100 only needs to include at least a vehicle control device 110 and an actuator group 120 in order to exhibit the three functions of "running", "turning", and "stopping" by autonomous driving. When the vehicle 100 acquires information from the outside for autonomous driving, the vehicle 100 only needs to further include a communication device 130. That is, the vehicle 100 that can move by autonomous driving does not necessarily have at least a part of the interior parts such as the driver's seat and the dashboard installed, and does not necessarily have at least a part of the exterior parts such as the bumper and the fender installed, and does not necessarily have the body shell installed. In this case, before the vehicle 100 is shipped from the factory FC, the remaining parts such as the body shell may be installed on the vehicle 100, or the vehicle 100 may be shipped from the factory FC in a state where the remaining parts such as the body shell are not installed on the vehicle 100, and then the remaining parts such as the body shell may be installed on the vehicle 100. Each part may be installed from any direction such as the upper side, the lower side, the front side, the rear side, the right side, or the left side of the vehicle 100, and they may be installed from the same direction or from different directions respectively.

[0013] At the second location PL2, further parts are assembled to the vehicle 100 by an assembly robot (not shown).

[0014] Figure 2 is a block diagram showing the configuration of system 50. Vehicle 100 includes a vehicle control device 110 for controlling each part of vehicle 100, an actuator group 120 including one or more actuators driven under the control of vehicle control device 110, and a communication device 130 for communicating with an external device such as server 200 by wireless communication. Actuator group 120 includes an actuator of a driving device for accelerating vehicle 100, an actuator of a steering device for changing the traveling direction of vehicle 100, and an actuator of a braking device for decelerating vehicle 100.

[0015] Vehicle control device 110 is constituted by a computer including a processor 111, a memory 112, an input / output interface 113, and an internal bus 114. Processor 111, memory 112, and input / output interface 113 are connected so as to be communicable bidirectionally via internal bus 114. Actuator group 120 and communication device 130 are connected to input / output interface 113. Processor 111 realizes various functions including the function as vehicle control unit 115 by executing program PG1 stored in memory 112.

[0016] Vehicle control unit 115 can run vehicle 100 by controlling actuator group 120 using the running control signal received from server 200. The running control signal is a control signal for running vehicle 100. In the present embodiment, the running control signal includes the acceleration and steering angle of vehicle 100 as parameters. Alternatively, the running control signal may include the speed of vehicle 100 as a parameter instead of or in addition to the acceleration of vehicle 100.

[0017] Server 200 is composed of a computer including a processor 201, a memory 202, an input / output interface 203, and an internal bus 204. The processor 201, the memory 202, and the input / output interface 203 are communicably connected bidirectionally via the internal bus 204. A communication device 205 for communicating with various external devices outside the server 200 is connected to the input / output interface 203. The communication device 205 can communicate with the vehicle 100 by wireless communication and can communicate with each external sensor 300 by wired communication or wireless communication.

[0018] The memory 202 stores in advance a program PG2, a first detection model DM1 and a second detection model DM2 described later, a reference route RR indicating a route along which the vehicle 100 should travel, and the like. By executing the program PG2 stored in the memory 202, the processor 201 realizes various functions including functions as a sensor acquisition unit 210, an estimation unit 220, a defect determination unit 230, a method selection unit 240, and a remote control unit 250.

[0019] The sensor acquisition unit 210 acquires a detection result output from an external sensor 300 described later. In the present embodiment, the sensor acquisition unit 210 acquires a captured image of the vehicle 100 captured by a camera which is the external sensor 300. The captured image is also referred to as "detection data". The estimation unit 220 estimates the position and orientation of the vehicle 100 using the detection result output from the external sensor 300. Alternatively, the estimation unit 220 may estimate only one of the position and orientation of the vehicle 100 using the detection result output from the external sensor 300. In this case, for example, the other of the position and orientation of the vehicle 100 is determined using the driving history of the vehicle 100 or the like. The defect determination unit 230 determines whether there is a defective assembly of parts to the vehicle 100. The defect determination unit 230 is also referred to as a "determination unit". The method selection unit 240 selects a method for estimating the position and orientation of the vehicle 100 according to whether there is a defective assembly of parts to the vehicle 100. The estimation unit 220 and the method selection unit 240 are also referred to as a "calculation unit".

[0020] The remote control unit 250 acquires the detection results from the sensors, generates a driving control signal for controlling the actuator group 120 of the vehicle 100 using the detection results, and transmits the driving control signal to the vehicle 100, thereby driving the vehicle 100 by remote control. The remote control unit 250 may generate and output not only the driving control signal but also, for example, control signals for controlling actuators that operate various auxiliary machines provided in the vehicle 100 and various equipment such as wipers, power windows, and lamps. That is, the remote control unit 250 may operate such various equipment and various auxiliary machines by remote control. In this specification, "remote control" includes "complete remote control" in which all operations of the vehicle 100 are completely determined from outside the vehicle 100 and "partial remote control" in which part of the operations of the vehicle 100 are determined from outside the vehicle 100.

[0021] The external sensor 300 is a sensor located outside the vehicle 100. The external sensor 300 is a sensor that captures the vehicle 100 from outside the vehicle 100. The external sensor 300 includes a communication device (not shown) and can communicate with other devices such as the server 200 by wired communication or wireless communication. Specifically, the external sensor 300 is constituted by a camera installed within the premises of the factory FC. The camera as the external sensor 300 captures a captured image including the vehicle 100 and outputs the captured image as a detection result.

[0022] FIG. 3 is a flowchart showing the processing procedure for driving control of the vehicle 100. The processing in FIG. 3 is executed by the processor 201 of the server 200 that functions as the remote control unit 250 and the processor 111 of the vehicle 100 that functions as the vehicle control unit 115. The processing shown in FIG. 3 is repeatedly executed, for example, at predetermined time intervals from the time when the vehicle 100 starts driving by remote control.

[0023] In step S1, the processor 201 of the server 200 acquires the vehicle position information of the vehicle 100 by using the detection result output from the external sensor 300. The vehicle position information is the position information that serves as the basis for generating the driving control signal. In this embodiment, the vehicle position information includes the position and orientation of the vehicle 100 in the global coordinate system GC of the factory FC. Specifically, in step S1, the processor 201 acquires the vehicle position information by using the detection result acquired by the external sensor 300, that is, the captured image acquired by the camera. Details of the method for detecting the position and orientation of the vehicle 100 will be described later.

[0024] In step S2, the processor 201 of the server 200 determines the target position to which the vehicle 100 should next head. In this embodiment, the target position is represented by the coordinates of X, Y, and Z in the global coordinate system GC. In the memory 202 of the server 200, a reference route RR, which is the route that the vehicle 100 should travel, is stored in advance. The route is represented by a node indicating the departure point, a node indicating the passing point, a node indicating the destination, and links connecting the respective nodes. The processor 201 determines the target position to which the vehicle 100 should next head by using the vehicle position information and the reference route RR. The processor 201 determines the target position on the reference route RR ahead of the current position of the vehicle 100.

[0025] In step S3, the processor 201 of the server 200 generates a driving control signal for causing the vehicle 100 to travel toward the determined target position. The processor 201 calculates the traveling speed of the vehicle 100 from the change in the position of the vehicle 100, and compares the calculated traveling speed with the target speed. Overall, when the traveling speed is lower than the target speed, the processor 201 determines the acceleration so that the vehicle 100 accelerates, and when the traveling speed is higher than the target speed, the processor 201 determines the acceleration so that the vehicle 100 decelerates. Further, when the vehicle 100 is located on the reference route RR, the processor 201 determines the steering angle and the acceleration so that the vehicle 100 does not deviate from the reference route RR, and when the vehicle 100 is not located on the reference route RR, in other words, when the vehicle 100 has deviated from the reference route RR, the processor 201 determines the steering angle and the acceleration so that the vehicle 100 returns to the reference route RR.

[0026] In step S4, the processor 201 of the server 200 transmits the generated driving control signal to the vehicle 100. The processor 201 repeats acquiring the position of the vehicle 100, determining the target position, generating the driving control signal, and transmitting the driving control signal at a predetermined cycle.

[0027] In step S5, the processor 111 of the vehicle 100 receives the driving control signal transmitted from the server 200. In step S6, the processor 111 of the vehicle 100 controls the actuator group 120 using the received driving control signal, and causes the vehicle 100 to travel at the acceleration and the steering angle represented by the driving control signal. The processor 111 repeats receiving the driving control signal and controlling the actuator group 120 at a predetermined cycle. According to the system 50 in the present embodiment, the vehicle 100 can be caused to travel by remote control, and the vehicle 100 can be moved without using conveying equipment such as a crane or a conveyor.

[0028] FIG. 4 is a flowchart showing the specific process of step S1 in FIG. 3. The process shown in FIG. 4 is executed by the processor 201 of the server 200 that functions as the sensor acquisition unit 210, the estimation unit 220, the defect determination unit 230, and the method selection unit 240. In the present embodiment, when there is an assembly defect in the parts assembled to the vehicle 100, the method for calculating the position and orientation of the vehicle 100 is changed.

[0029] In step S11, using the image of the vehicle 100 acquired by the external sensor 300, it is determined whether the part to be attached in the target process is correctly assembled to the vehicle 100. The target process is, for example, the process immediately preceding. For the sake of easy understanding of the technology, in this specification, it is assumed that the processes before the target process are correctly executed. For example, when the process immediately preceding is process P2 and the process before process P2 is process P1, the vehicle 100 targeted in process P2 is a vehicle in which the parts are correctly attached in process P1. For example, for the vehicle 100 with an assembly defect in part in process P1, it is taken out of the line before the execution of process P2.

[0030] For example, by the method of pattern matching using a previously prepared reference image, it is determined whether the part is correctly assembled according to whether the captured image and the reference image are similar. As the previously prepared reference image, for example, an image captured by the external sensor 300 of the vehicle 100 in which the parts are correctly assembled in the process immediately preceding can be used. The reference image and the captured image are acquired by, for example, the same external sensor 300. When the similarity between the captured image and the reference image is equal to or greater than a predetermined threshold, it is determined that the part is correctly assembled.

[0031] Further, for example, by using a plurality of captured images acquired by the same external sensor 300 during a predetermined period, it may be determined whether the parts are correctly assembled by a pattern matching method. The plurality of captured images acquired during the predetermined period may be a plurality of captured images continuously acquired in time series. When a determination result that the parts are not correctly assembled is obtained based on a predetermined number of captured images among the plurality of captured images, it may be determined that the parts are not correctly assembled.

[0032] Further, for example, by a pattern matching method using a plurality of captured images acquired by a plurality of external sensors 300 at the same timing, it may be determined whether the parts are correctly assembled. When a determination result that the parts are not correctly assembled is obtained based on a predetermined number of captured images among the plurality of captured images, it may be determined that the parts are not correctly assembled.

[0033] When the parts are correctly assembled in the vehicle 100 (step S11; YES), the process of step S12 is executed. When the parts are not correctly assembled in the vehicle 100 (step S11; NO), the process of step S13 is executed.

[0034] In step S12, the position of the vehicle 100 is estimated using the first detection model DM1. The method of estimating the position of the vehicle 100 using the first detection model DM1 is also referred to as the "first method". The first detection model DM1 is also referred to as the "first machine learning model". Specifically, first, by inputting the captured image into the first detection model DM1 that utilizes artificial intelligence, the external shape of the vehicle 100 is detected. As the first detection model DM1, for example, a trained machine learning model trained to implement either semantic segmentation or instance segmentation can be mentioned. As this machine learning model, for example, a convolutional neural network (hereinafter, CNN) trained by supervised learning using a training dataset can be used. The training dataset DS1 used to generate the first detection model DM1 includes a plurality of training images including the vehicle 100 in which the parts are correctly assembled in the process immediately preceding, and a label indicating whether each region in the training image is a region indicating the vehicle 100 or a region indicating other than the vehicle 100. During learning, it is preferable that the parameters are updated so as to reduce the error between the output result by the first detection model DM1 and the label by backpropagation (error backpropagation method). Further, the coordinates in the image coordinate system representing the external shape detected using the first detection model DM1 are converted into the coordinates in the global coordinate system GC. In this way, the position of the vehicle 100 in the global coordinate system GC is acquired.

[0035] Also, for example, based on the direction of the vector regarding the movement of the vehicle 100 calculated using the optical flow method, the direction of the vehicle 100 is estimated. In the optical flow method, the direction of the vector regarding the movement of the vehicle 100 is estimated from the position change of the feature points of the moving object between the frames of the captured image.

[0036] In step S13, the position and orientation of the vehicle 100 are estimated using the second detection model DM2. The method of estimating the position of the vehicle 100 using the second detection model DM2 is also referred to as the "second method". The second detection model DM2 is also referred to as the "second machine learning model". Specifically, first, the captured image is input into the second detection model DM2 that utilizes artificial intelligence, thereby detecting the external shape of the vehicle 100. The difference from the process in step S12 is that the second detection model DM2 is used as the machine learning model for detecting the external shape of the vehicle 100.

[0037] The learning dataset DS2 used to generate the second detection model DM2 includes a plurality of training images including the vehicle 100 in which the parts are not correctly assembled in the immediately preceding process, and a label indicating whether each region in the training image is a region indicating the vehicle 100 or a region indicating other than the vehicle 100. Defects in part assembly include cases where the part is not attached to the vehicle 100 at all, cases where the part is assembled to the vehicle 100 but the assembled position is incorrect, cases where the part is not sufficiently fixed to the vehicle 100 and there is play in the part, etc. When a plurality of parts are assembled in the immediately preceding process, incorrect assembly of the parts includes cases where one or more parts are not attached in the correct position.

[0038] Since various modes of incorrect part assembly are assumed, for example, a plurality of machine learning models respectively generated using a learning set including training images corresponding to each pattern may be included in the second detection model DM2 by classifying typical patterns of part assembly defects that may occur in the target process.

[0039] Furthermore, the coordinates in the image coordinate system representing the appearance shape detected using the second detection model DM2 are converted into the coordinates in the global coordinate system GC. In this way, the position of the vehicle 100 in the global coordinate system GC is obtained. The estimation of the orientation of the vehicle 100 is the same as in step S12. The above is the processing related to the position and estimation in step S1 shown in FIG. 3.

[0040] According to the present embodiment, different methods are used to obtain the position of the vehicle depending on whether there is a defective component assembly or not. Therefore, even if there is a defective component assembly in the components assembled to the vehicle 100, a decrease in the detection accuracy of the position of the vehicle 100 can be suppressed.

[0041] B. Other Embodiments: (B1) The machine learning models respectively used when there is no defective component assembly and when there is a defective component assembly are not limited to those described in the above embodiment.

[0042] When there is no defective component assembly, the processor 201 may obtain the position of the vehicle 100 using a third detection model DM3 generated by machine learning using a learning data set including an image of the vehicle taken after the target process has been performed, based on an image of the vehicle as detection data acquired by a camera as the external sensor 300.

[0043] The learning data set DS3 used when generating the third detection model DM3 has a plurality of training images including the vehicle 100 in a state after the target process has been performed, and a label indicating whether each region in the training image is a region indicating the vehicle 100 or a region indicating other than the vehicle 100. When the target process is the process immediately preceding, the plurality of training images including the vehicle 100 in a state after the target process has been performed are, as in the embodiment, a plurality of training images including the vehicle 100 in which the components have been correctly assembled in the process immediately preceding.

[0044] In addition, in the case of defective component assembly, the processor 201 may determine the position of the vehicle 100 by using a fourth detection model DM4 generated by machine learning using a learning dataset including an image of the vehicle taken in a state before the target process is performed, based on the image of the vehicle as detection data.

[0045] The learning dataset DS4 used when generating the fourth detection model DM4 includes a plurality of training images including the vehicle 100 in a state before the target process is performed, and a label indicating whether each region in the training image is a region indicating the vehicle 100 or a region indicating other than the vehicle 100. When the target process is the process immediately preceding, the plurality of training images including the vehicle 100 in a state before the target process is performed are, for example, a plurality of training images including the vehicle 100 in which the process one step before the immediately preceding process was performed.

[0046] In another embodiment (B1), the method of estimating the position of the vehicle 100 using the third detection model DM3 may also be referred to as the "first method". The third detection model DM3 may also be referred to as the "third machine learning model". The method of estimating the position of the vehicle 100 using the fourth detection model DM4 may also be referred to as the "second method". The fourth detection model DM4 may also be referred to as the "fourth machine learning model". In this case, the labor of preparing training images of vehicles with defective component assembly can be saved, and the learning dataset can be easily prepared.

[0047] (B2) In addition, in the case of defective component assembly, the processor 201 may determine the position of the vehicle 100 based on an image of the vehicle using two or more machine learning models. For example, it is assumed that two machine learning models are used. In this case, the processor 201 can use the fourth detection model DM4 generated in the same manner as described in (B1) above as one of the two machine learning models.

[0048] Furthermore, as the other of the two machine learning models, the processor 201 may determine the position of the vehicle 100 by using a fifth detection model DM5 generated by machine learning using a learning dataset including an image of the vehicle taken in the state after the target process has been performed, based on an image of the vehicle as detection data.

[0049] The learning dataset DS5 used when generating the fifth detection model DM5 includes a plurality of training images including the vehicle 100 in the state where the target process has been performed, and a label indicating whether each region in the training image is a region indicating the vehicle 100 or a region indicating other than the vehicle 100. When the target process is the process immediately preceding, the plurality of training images including the vehicle 100 in the state after the target process has been performed are the same as the training images included in the learning dataset DS3 of (D1) above. For this reason, as the fifth detection model DM5, the same one as the third detection model DM3 generated as described in (D1) above can be used. Thereby, the labor of generating the machine learning model can be made simple.

[0050] In the case where there is a defective component assembly, the processor 201 can adopt, as the final estimation result, the one with a higher reliability for each estimation result among the position of the vehicle 100 estimated using the fourth detection model DM4 and the position of the vehicle 100 estimated using the fifth detection model DM5, respectively.

[0051] Also, in the case where there is no defective component assembly, the processor 201 can determine the position of the vehicle 100 by using the third detection model DM3 in the same manner as in (B1) above.

[0052] (B3) In the above-described embodiment, the external sensor 300 is a camera, but the external sensor 300 may be a distance measuring device using LiDAR (Light Detection And Ranging) technology. In this case, the external sensor 300 acquires 3D point cloud data of the vehicle 100. The 3D point cloud data is data indicating the 3D positions of the point cloud. The 3D point cloud data is also referred to as "detection data". The 3D point cloud data is also referred to as "distance measurement point data".

[0053] Also, different methods are used to estimate at least one of the position and orientation of the vehicle 100 depending on whether there is a defective component assembly or not.

[0054] When there is no defective component assembly, the processor 201 performs template matching using the 3D point cloud data as the detection data acquired by the distance measuring device as the external sensor 300 and the first vehicle point cloud data VP1 which is 3D CAD data representing the shape of the vehicle 100 in which the components are correctly assembled in the target process, thereby estimating at least one of the position and orientation of the vehicle 100.

[0055] On the other hand, when there is a defective component assembly, the processor 201 performs template matching using the 3D point cloud data as the detection data acquired by the distance measuring device as the external sensor 300 and the second vehicle point cloud data VP2 which is 3D CAD data representing the shape of the vehicle 100 in which a defective component assembly has occurred in the target process, thereby estimating at least one of the position and orientation of the vehicle 100. The first vehicle point cloud data VP1 and the second vehicle point cloud data VP2 are pre-stored in the memory 112. Also, the first vehicle point cloud data VP1 and the second vehicle point cloud data VP2 may include information for specifying the orientation of the vehicle 100.

[0056] In addition, in the target process, for example, when a plurality of components are assembled, there are a plurality of patterns of assembly defects of the components. Therefore, as the second vehicle point cloud data VP2, a plurality of 3D CAD data representing a plurality of patterns of assembly defects may be prepared. For example, in the target process, when three components are assembled, the second vehicle point cloud data VP2 may include 3D CAD data representing the shape of the vehicle 100 in which only one component is assembled and 3D CAD data representing the shape of the vehicle 100 in which two components are assembled.

[0057] The method of estimating the position and orientation of the vehicle 100 using the first vehicle point cloud data VP1 may also be referred to as the "first method". The first vehicle point cloud data VP1 may also be referred to as the "first reference data". The method of estimating the position and orientation of the vehicle 100 using the second vehicle point cloud data VP2 may also be referred to as the "second method". The second vehicle point cloud data VP2 may also be referred to as the "second reference data". In this specification, the 3D CAD data representing the shape of a vehicle such as the first vehicle point cloud data VP1 used in template matching may sometimes be referred to as reference data.

[0058] (B4) When the external sensor 300 is a distance measuring device using LiDAR technology, the position and orientation of the vehicle 100 may be estimated as follows.

[0059] When there are no assembly defects in the components, the processor 201 performs template matching using the 3D point cloud data as the detection data acquired by the distance measuring device as the external sensor 300 and the third vehicle point cloud data VP3 which is 3D CAD data representing the shape of the vehicle 100 in the state after the target process is performed, thereby estimating at least one of the position and orientation of the vehicle 100. The target process is, for example, the process performed immediately before. The third vehicle point cloud data VP3 which is 3D CAD data representing the shape of the vehicle 100 in the state after the target process is performed may be the same as the first vehicle point cloud data VP1 which is 3D CAD data representing the shape of the vehicle 100 in which the components are correctly assembled in the target process described in (B3) above.

[0060] On the other hand, when there is a defective assembly of parts, the processor 201 performs template matching using the three-dimensional point cloud data as detection data acquired by the distance measuring device as the external sensor 300 and the fourth vehicle point cloud data VP4 which is the three-dimensional CAD data representing the shape of the vehicle 100 in the state before the target process is performed, thereby estimating at least one of the position and orientation of the vehicle 100. The third vehicle point cloud data VP3 and the fourth vehicle point cloud data VP4 are stored in advance in the memory 112. Further, the third vehicle point cloud data VP3 and the fourth vehicle point cloud data VP4 may include information for specifying the orientation of the vehicle 100.

[0061] The method of estimating the position and orientation of the vehicle 100 using the third vehicle point cloud data VP3 is also referred to as the "first method". The third vehicle point cloud data VP3 is also referred to as the "third reference data". The method of estimating the position and orientation of the vehicle 100 using the fourth vehicle point cloud data VP4 is also referred to as the "second method". The fourth vehicle point cloud data VP4 is also referred to as the "fourth reference data". In this case, the labor of preparing the CAD data of the vehicle with defective part assembly can be saved.

[0062] FIG. 5 is an explanatory diagram of the advantages of performing template matching using the three-dimensional CAD data representing the shape of the vehicle 100 in the state where the target process has been performed and the three-dimensional CAD data representing the shape of the vehicle 100 in the state before the target process is performed. For example, it is assumed that the processes A, B, and C are performed in this order, and parts are assembled to the vehicle 100 in each process. Further, it is assumed that there is no other process between process A and process B, and there is no other process between process B and process C.

[0063] In process A, when there is no defective assembly of parts, in the template matching, the three-dimensional CAD data representing the shape of the vehicle 100 after process A is performed is used as the reference data.

[0064] In Process B, when there is a defective component assembly, 3D CAD data representing the shape of the vehicle 100 before Process B is used as reference data. As shown in the figure, the 3D CAD data representing the shape of the vehicle 100 before Process B is the same as the 3D CAD data representing the shape of the vehicle 100 after Process A is carried out.

[0065] In Process C, when there is a defective component assembly, 3D CAD data representing the shape of the vehicle 100 before Process C is used as reference data. As shown in the figure, the 3D CAD data representing the shape of the vehicle 100 before Process C is the same as the 3D CAD data representing the shape of the vehicle 100 after Process B is carried out.

[0066] In this way, the 3D CAD data representing the shape of the vehicle 100 in the state after the target process is carried out and the 3D CAD data representing the shape of the vehicle 100 in the state before the target process is carried out can be shared as reference data. Therefore, the preparation of the 3D CAD data as reference data can be made simple.

[0067] (B5) Also, when there is a defective component assembly, the processor 201 may use two or more reference data to obtain the position and orientation of the vehicle 100. For example, assume that two reference data are used. In this case, the processor 201 can use, as one of the two reference data, the fourth vehicle point cloud data VP4, which is 3D CAD data representing the shape of the vehicle 100 in the state before the target process, in the same way as described in (B4) above.

[0068] Furthermore, the processor 201 can use, as the other of the two reference data, 3D CAD data representing the shape of the vehicle 100 in the state after the target process is carried out. This 3D CAD data is the same as the third vehicle point cloud data VP3 described in (B4) above.

[0069] When there is a defective component assembly, the processor 201 can adopt, as the final estimation result, the estimation result of the position and orientation of the vehicle 100 with a higher similarity to the reference data, out of the estimation result of the position and orientation of the vehicle 100 estimated using the fourth vehicle point cloud data VP4 and the estimation result of the position and orientation of the vehicle 100 estimated using the third vehicle point cloud data VP3.

[0070] Also, when there is no defective component assembly, the processor 201 can estimate the position and orientation of the vehicle 100 in the same manner as in the above (B4).

[0071] (B6) In the embodiment, an example of estimating the position of the vehicle 100 using an image acquired by a camera which is an external sensor 300 has been described. Alternatively, a three-dimensional point cloud representing a three-dimensional space can be reproduced using a plurality of images acquired by a plurality of external sensors 300. The position and orientation of the vehicle 100 may be estimated by template matching using the reproduced three-dimensional point cloud data and vehicle point cloud data prepared in advance. In this case, first vehicle point cloud data VP1 which is three-dimensional CAD data representing the shape of the vehicle 100 with components correctly assembled in the target process, and second vehicle point cloud data VP2 which is three-dimensional CAD data representing the shape of the vehicle 100 with a defective component assembly in the target process are prepared in advance. The first vehicle point cloud data VP1 and the second vehicle point cloud data VP2 include information for specifying the orientation of the vehicle 100.

[0072] Therefore, when the parts are correctly assembled to the vehicle 100 in the target process, the position and orientation of the vehicle 100 are estimated by performing template matching using the first vehicle point cloud data V1. On the other hand, when there is an assembly defect of the parts in the target process, the position and orientation of the vehicle 100 are estimated by performing template matching using the second vehicle point cloud data V2. Also, in the target process, for example, when a plurality of parts are assembled, there are a plurality of patterns of assembly defects of the parts. For this reason, as the second vehicle point cloud data VP2, a plurality of 3D CAD data representing a plurality of patterns of assembly defects may be prepared.

[0073] Alternatively, as described in (B4) above, when there is no assembly defect of the parts, template matching is performed using the 3D CAD data representing the shape of the vehicle 100 in the state after the target process is performed. Also, when there is an assembly defect of the parts, template matching is performed using the 3D CAD data representing the shape of the vehicle 100 in the state before the target process is performed.

[0074] (B7) In the embodiment, after determining the presence or absence of an assembly defect of the parts (see step S11 in FIG. 4), when there is no assembly defect of the parts, the position of the vehicle 100 is estimated using the first detection model DM1, and when there is an assembly defect of the parts, the position of the vehicle 100 is estimated using the second detection model DM2.

[0075] Alternatively, without determining the presence or absence of defective component assembly, first, the position of the vehicle 100 may be estimated using the first detection model DM1. When the score value indicating the reliability of the estimation result is less than a predetermined threshold value, the position of the vehicle 100 is estimated using the second detection model DM2. This is because when the score value indicating the reliability of the estimation result using the first detection model DM1 is less than the threshold value, it is assumed that the components were not assembled correctly in the previous process. On the other hand, when the score value indicating the reliability of the estimation result of the position of the vehicle 100 using the first detection model DM1 is equal to or greater than the threshold value, it is assumed that the components were assembled correctly in the previous process. In this case, the estimation process using the second detection model DM2 is not performed.

[0076] (B8) In the embodiment, an example in which the presence or absence of defective component assembly is determined using an image has been described (see step S11 in FIG. 4). After each process is performed, for example, when an operator outputs the presence or absence of defective component assembly to the upper server using the terminal device, the processor 201 can determine the presence or absence of defective component assembly by querying the upper server about the presence or absence of defective component assembly in the previous process.

[0077] (B9) Further, after estimating the position of the vehicle 100 in which defective component assembly has occurred, the processor 201 of the server 200 can determine the evacuation area EA as the target position to which the vehicle 100 should next travel using the vehicle position information. It is assumed that the evacuation area EA, which is the area where the vehicle 100 should be evacuated, is stored in advance in the memory 202 of the server 200.

[0078] FIG. 6 is an explanatory diagram of the case where the vehicle is retracted to the retraction area EA. The processor 201 generates a driving control signal for driving the vehicle 100 toward the retraction area EA which is the target position. Further, the processor 201 may notify at least one operator WO of an instruction to collect the vehicle 100 together with information for identifying the vehicle 100 before moving the vehicle 100 to the retraction area EA or after moving the vehicle 100 to the retraction area EA. For example, the processor 201 notifies the operator WO via the terminal device T1 possessed by the operator WO and the terminal device T2 installed at the waiting place OF. The waiting place OF is a place where the operator WO waits or takes a break. At least one operator WO who has received the notification performs a collection operation or the like of the vehicle 100 retracted to the retraction area EA. For the vehicle 100 in which a defective component assembly has occurred, by once taking it out of the production line, it is possible to suppress the occurrence of interference with the running of other vehicles 100 and the occurrence of delays in subsequent processes.

[0079] (B10) In the above embodiment, the server 200 automatically generates a driving control signal to be transmitted to the vehicle 100. Alternatively, the server 200 may generate a driving control signal to be transmitted to the vehicle 100 according to the operation of an external operator located outside the vehicle 100. For example, an external operator operates a control device including a display for displaying a captured image output from the external sensor 300, a steering wheel for remotely operating the vehicle 100, an accelerator pedal, a brake pedal, and a communication device for communicating with the server 200 by wired communication or wireless communication, and the server 200 may generate a driving control signal corresponding to the operation applied to the control device.

[0080] (B11) The vehicle 100 may be manufactured by combining a plurality of modules. A module means a unit composed of a plurality of parts grouped according to the parts and functions of the vehicle 100. For example, the platform of the vehicle 100 may be manufactured by combining a front module that constitutes the front part of the platform, a center module that constitutes the central part of the platform, and a rear module that constitutes the rear part of the platform. Note that the number of modules constituting the platform is not limited to three, and may be two or less or four or more. In addition to, or instead of, the parts constituting the platform, the parts constituting a part of the vehicle 100 different from the platform may be modularized. Further, each type of module may include any exterior parts such as bumpers and grills, and any interior parts such as seats and consoles. Further, not limited to the vehicle 100, any type of moving body may be manufactured by combining a plurality of modules. Such modules may be manufactured, for example, by joining a plurality of parts by welding, fixtures, or the like, or by integrally molding at least a part of the parts constituting the module as one part by casting. The molding method of integrally molding one part, particularly a relatively large part, is also called gigacasting or megacasting. For example, the above-mentioned front module, center module, and rear module may be manufactured using gigacasting.

[0081] (B12) Using the running of the vehicle 100 by autonomous driving to transport the vehicle 100 is also called "self-propelled transport". Further, the configuration for realizing self-propelled transport is also called "vehicle remote control autonomous driving transport system". Further, the production method of producing the vehicle 100 using self-propelled transport is also called "self-propelled production". In self-propelled production, for example, in the factory FC that manufactures the vehicle 100, at least a part of the transport of the vehicle 100 is realized by self-propelled transport.

[0082] (B13) The means for realizing the functions of the server 200 is not limited to software, and part or all of it may be realized by dedicated hardware. For example, as the dedicated hardware, a circuit represented by FPGA (Field Programmable Gate Array) or ASIC (Application Specific Integrated Circuit) may be used.

[0083] The present disclosure is not limited to the above-described embodiments, and can be realized in various configurations without departing from the gist thereof. For example, the technical features in the embodiments corresponding to the technical features in each of the embodiments described in the summary of the invention can be appropriately replaced or combined in order to solve the above-described problems or to achieve part or all of the above-described effects. Further, if the technical feature is not described as essential in this specification, it can be appropriately deleted.

Description of Reference Numerals

[0084] 50… System, 100… Vehicle, 110… Vehicle control device, 111… Processor, 112… Memory, 113… Input / output interface, 114… Internal bus, 115… Vehicle control unit, 120… Actuator group, 130… Communication device, 200… Server, 201… Processor, 202… Memory, 203… Input / output interface, 204… Internal bus, 205… Communication device, 210… Sensor acquisition unit, 220… Estimation unit, 230… Defect discrimination unit, 240… Method selection unit, 250… Remote control unit, 300… External sensor, DM1… First detection model, DM2… Second detection model, EA… Evacuation area, FC… Factory, GC… Global coordinate system, OF… Waiting location, PG1… Program, PG2… Program, PL1… First location, PL2… Second location, RR… Reference path, T1, T2… Terminal device, TR… Track, WO… Operator

Claims

1. A vehicle that travels within a factory where a plurality of processes are performed to manufacture a vehicle, and a control device that remotely controls the vehicle that is the subject of the plurality of processes, a determination unit that determines whether parts are correctly assembled to the vehicle, an arithmetic unit, when the parts are correctly assembled to the vehicle, based on detection data acquired by an external sensor, using a first method, at least one of the position and orientation of the vehicle is acquired, when the parts are not correctly assembled to the vehicle, based on the detection data, using a second method different from the first method, at least one of the position and orientation is obtained, an arithmetic unit, A control device comprising.

2. The control device according to claim 1, wherein the first method is a method of obtaining at least one of the position and orientation using a first machine learning model generated by machine learning using a learning data set including an image of the vehicle in which the parts are correctly assembled, based on the image of the vehicle as the detection data acquired by a camera as the external sensor, The second method is a method of obtaining at least one of the position and orientation using a second machine learning model generated by machine learning using a learning data set including an image of the vehicle in which the parts are not correctly assembled, based on the image of the vehicle as the detection data. Control device.

3. The control device according to claim 1, wherein the first method is a method of obtaining at least one of the position and orientation using a third machine learning model generated by machine learning using a learning data set including an image of the vehicle taken in a state after the target process is performed, based on the image of the vehicle as the detection data acquired by a camera as the external sensor, The second method is a method of obtaining at least one of the position and orientation using a fourth machine learning model generated by machine learning using a learning data set including an image of the vehicle taken in a state before the target process is performed, based on the image of the vehicle as the detection data. Control device.

4. The control device according to claim 1, The first method is a method for obtaining at least one of the position and the orientation, using the distance measurement point data as the detection data acquired by a distance measurement device as the external sensor and first reference data which is a 3D CAD representing the shape of the vehicle in which the parts are correctly assembled. The second method is a method for obtaining at least one of the position and the orientation, using the distance measurement point data as the detection data and second reference data which is 3D CAD data representing the shape of the vehicle in which the parts are not correctly assembled. Control device.

5. The control device according to claim 1, The first method is a method for obtaining at least one of the position and the orientation, using the distance measurement point data as the detection data acquired by a distance measurement device as the external sensor and third reference data which is 3D CAD data representing the shape of the vehicle in a state after a target process is performed. The second method is a method for obtaining at least one of the position and the orientation, using the distance measurement point data as the detection data and fourth reference data which is 3D CAD data representing the shape of the vehicle in a state before the target process is performed. Control device.

Images