Systems, apparatus, and methods

Abstract

To suppress the decrease in the accuracy of vehicle position estimation caused by a decrease in the accuracy of detecting the vehicle's external shape. [Solution] The system comprises a camera that captures an imaging area including the path of a moving object; a distance measuring device that measures at least a part of the imaging area; an acquisition unit that acquires image data captured by the camera and 3D point cloud data measured by the distance measuring device; a position estimation unit that estimates the position of a moving object using the shape of the moving object obtained by inputting the image data of the moving object into a trained machine learning model; and a learning unit that retrains the machine learning model using the current image data when there is a difference between past 3D point cloud data and current 3D point cloud data.

Description

Technical Field

[0001] The present disclosure relates to systems, devices, and methods.

Background Art

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When an object not shown in the image used for learning the machine learning model is placed around the traveling path of the vehicle, the detection accuracy of the outer shape of the vehicle using the machine learning model may decrease. And when the detection accuracy of the outer shape of the vehicle decreases, the accuracy of the estimated position of the vehicle decreases. This problem is common not only to vehicles but also to moving bodies.

Means for Solving the Problems

[0005] The present disclosure can be realized in the following forms.

[0006] (1) According to the first embodiment of this disclosure, a system is provided. This system comprises a camera that images an imaging area including the path of a moving object; a distance measuring device that measures at least a portion of the imaging area; an acquisition unit that acquires image data captured by the camera and 3D point cloud data measured by the distance measuring device; a position estimation unit that estimates the position of the moving object using the shape of the moving object obtained by inputting the image data of the moving object into a trained machine learning model; and a learning unit that retrains the machine learning model using the current image data when there is a difference between past 3D point cloud data and the current 3D point cloud data. According to this embodiment, the learning unit can detect when an object is placed in the imaging area. When an object is placed in the imaging area, the learning unit retrains the machine learning model using the current image data. As a result, the machine learning model can recognize the newly placed object in the imaging area and distinguish between the moving object and the object, thus suppressing a decrease in the detection accuracy of the moving object's shape. Therefore, a decrease in the accuracy of the estimated position of the vehicle can be suppressed. (2) In the system of the above form, the learning unit may set the past 3D point cloud data as reference data, perform a judgment process multiple times to determine whether there is a difference between the reference data and the current 3D point cloud data, and retrain the machine learning model if the judgment results in the judgment process consecutively determine that there is a difference. With this form, retraining can be performed when there is a high probability that the object is permanently placed. Therefore, unnecessary retraining can be avoided. (3) In the system of the above configuration, the track is provided on the manufacturing line, and the learning unit may determine whether there is a difference between past 3D point cloud data and current 3D point cloud data before manufacturing on the manufacturing line begins. Rearranging objects on the manufacturing line is often done while manufacturing is stopped. Therefore, with this configuration, the probability of relearning being performed after the rearrangement of objects is confirmed increases, so the relearning process can be performed efficiently. (4) According to a second embodiment of the present disclosure, an apparatus is provided. This apparatus includes: an acquisition unit that acquires image data of an imaging area including the track of a moving object and 3D point cloud data measured by a distance measuring device that measures at least a portion of the imaging area; a position estimation unit that estimates the position of a moving object using the shape of the moving object obtained by inputting the image data of the moving object into a trained machine learning model; and a learning unit that retrains the machine learning model using the current image data when there is a difference between past 3D point cloud data and current 3D point cloud data. According to this embodiment, a decrease in the detection accuracy of the shape of the moving object can be suppressed and a decrease in the accuracy of the estimated position of the vehicle can be suppressed. (5) A third embodiment of the present disclosure provides a method. This method comprises an acquisition step of acquiring three-dimensional point cloud data measured by a distance measuring device that measures at least a portion of an imaging area including the path of a moving object, and a learning step of retraining a previously trained machine learning model that determines the outline of the moving object using the current image data of the imaging area including the moving object as input, when there is a difference between past three-dimensional point cloud data and current three-dimensional point cloud data. This embodiment makes it possible to suppress a decrease in the accuracy of detecting the outline of the moving object. In addition to the system described above, this disclosure can also be implemented in the form of a tangible, non-temporary recording medium in which a control device, control method, or control program is recorded in a way that is readable by a computer. [Brief explanation of the drawing]

[0007] [Figure 1] This is a conceptual diagram showing the system configuration in the first embodiment. [Figure 2] This is a block diagram showing the system configuration. [Figure 3] This is a flowchart showing the processing procedure for vehicle driving control. [Figure 4] This diagram illustrates the relationship between the imaging area and the measurement area. [Figure 5] This is a flowchart showing the processing steps for the retraining process. [Figure 6] This is a block diagram showing the configuration of the vehicle in the second embodiment. [Figure 7] This is a flowchart showing the processing procedure for vehicle driving control in the second embodiment. [Modes for carrying out the invention]

[0008] A. First Embodiment: Figure 1 is a conceptual diagram showing the configuration of the system 50 in the first embodiment. The system 50 comprises one or more vehicles 100 as mobile bodies, a server 200, and one or more external sensors 300.

[0009] In this disclosure, “mobile object” means an object that can move, such as a vehicle or an electric vertical take-off and landing aircraft (so-called flying car). A vehicle may be a wheeled vehicle or a tracked vehicle, such as a passenger car, truck, bus, motorcycle, car, or construction vehicle. Vehicles include electric vehicles (BEVs: Battery Electric Vehicles), gasoline vehicles, hybrid vehicles, and fuel cell vehicles. If the mobile object is not a vehicle, the terms “vehicle” and “car” in this disclosure may be replaced with “mobile object” as appropriate, and the term “driving” may be replaced with “moving” as appropriate.

[0010] Vehicle 100 is configured to operate autonomously. "Autonomous operation" means operation without the operation of a passenger. Operation refers to operations related to at least one of the following: "going," "turning," or "stopping" of vehicle 100. Autonomous operation is achieved by automatic or manual remote control using a device located outside vehicle 100, or by autonomous control of vehicle 100. Vehicle 100 operating autonomously may have passengers on board who do not perform operation. Passengers who do not perform operation include, for example, people simply sitting in the seats of vehicle 100, or people performing tasks other than operation, such as assembly, inspection, or operating switches, while on board vehicle 100. Operation by a passenger is sometimes called "manned operation."

[0011] In this specification, "remote control" includes "fully remote control," in which all operations of the vehicle 100 are completely determined from outside the vehicle 100, and "partial remote control," in which some operations of the vehicle 100 are determined from outside the vehicle 100. Furthermore, "autonomous control" includes "fully autonomous control," in which the vehicle 100 autonomously controls its own operations without receiving any information from external devices, and "partial autonomous control," in which the vehicle 100 autonomously controls its own operations using information received from external devices.

[0012] In this embodiment, system 50 is used in a factory FC where vehicle 100 is manufactured. The reference coordinate system of the factory FC is the global coordinate system GC, and any position within the factory FC can be represented by X, Y, Z coordinates in the global coordinate system GC. The factory FC comprises a first location PL1 and a second location PL2. The first location PL1 and the second location PL2 are connected by a track TR on which vehicle 100 can travel. Multiple external sensors 300 are installed along the track TR in the factory FC. The position of each external sensor 300 in the factory FC is pre-adjusted. Vehicle 100 moves from the first location PL1 to the second location PL2 via the track TR by unmanned operation.

[0013] Figure 2 is a block diagram showing the configuration of system 50. The vehicle 100 includes a vehicle control device 110 for controlling various parts of the vehicle 100, an actuator group 120 including one or more actuators driven under the control of the vehicle control device 110, and a communication device 130 for communicating wirelessly with external devices such as a server 200. The actuator group 120 includes actuators for a drive system to accelerate the vehicle 100, actuators for a steering system to change the direction of travel of the vehicle 100, and actuators for a braking system to decelerate the vehicle 100.

[0014] The vehicle control device 110 is composed of a computer comprising a processor 111, a memory 112, an input / output interface 113, and an internal bus 114. The processor 111, the memory 112, and the input / output interface 113 are connected via the internal bus 114 to enable bidirectional communication. The input / output interface 113 is connected to an actuator group 120 and a communication device 130. The processor 111 implements various functions, including those of a vehicle control unit 115, by executing a program PG1 stored in the memory 112.

[0015] The vehicle control unit 115 drives the vehicle 100 by controlling the actuator group 120. The vehicle control unit 115 can drive the vehicle 100 by controlling the actuator group 120 using the driving control signal received from the server 200. The driving control signal is a control signal for driving the vehicle 100. In this embodiment, the driving control signal includes the acceleration and steering angle of the vehicle 100 as parameters. In other embodiments, the driving control signal may include the speed of the vehicle 100 as a parameter instead of, or in addition to, the acceleration of the vehicle 100.

[0016] Server 200 is composed of a computer including a processor 201, a memory 202, an input / output interface 203, and an internal bus 204. The processor 201, the memory 202, and the input / output interface 203 are connected in a bidirectional communicable manner via the internal bus 204. A communication device 205 for communicating with various external devices outside the server 200 is connected to the input / output interface 203. The communication device 205 can communicate with the vehicle 100 by wireless communication and can communicate with each external sensor 300 by wired communication or wireless communication. The processor 201 realizes various functions including the function as a remote control unit 210 by executing the program PG2 stored in the memory 202.

[0017] The remote control unit 210 acquires the detection result by a sensor, generates a driving control signal for controlling the actuator group 120 of the vehicle 100 using the detection result, and transmits the driving control signal to the vehicle 100, thereby driving the vehicle 100 by remote control. The remote control unit 210 may generate and output not only the driving control signal but also a control signal for controlling an actuator that operates various auxiliary machines provided in the vehicle 100, various equipment such as a wiper, a power window, and a lamp. That is, the remote control unit 210 may operate such various equipment and various auxiliary machines by remote control.

[0018] In addition to the above configuration, the processor 201 includes an acquisition unit 211, a position estimation unit 212, and a learning unit 213. The acquisition unit 211, the position estimation unit 212, and the learning unit 213 are functional units realized by executing the program PG2. The acquisition unit 211 acquires the image data captured by a camera 300A described later and the three-dimensional point cloud data measured by a distance measuring device 300B described later. The position estimation unit 212 estimates the position of the vehicle 100 using the outer shape of the vehicle 100 obtained by inputting the image data into a detection model DM that is a learned machine learning model. The learning unit 213 re-learns the detection model DM using the current image data when there is a difference between the past three-dimensional point cloud data and the current three-dimensional point cloud data.

[0019] The external sensor 300 is a sensor located outside the vehicle 100. The external sensor 300 in the present embodiment is a sensor that captures the vehicle 100 from outside the vehicle 100. The external sensor 300 includes a communication device (not shown) and can communicate with other devices such as the server 200 by wired communication or wireless communication.

[0020] Specifically, the external sensor 300 is composed of a camera 300A or a distance measuring device 300B. The camera 300A as the external sensor 300 images the vehicle 100 and outputs image data as a detection result. The distance measuring device 300B as the external sensor 300 images the vehicle 100 and outputs three-dimensional point cloud data as a detection result. The distance measuring device is, for example, LiDAR (Light Detection And Ranging).

[0021] FIG. 3 is a flowchart showing the processing procedure of the travel control of the vehicle 100 in the first embodiment. In the processing procedure of FIG. 3, the processor 201 of the server 200 functions as the remote control unit 210 by executing the program PG2. Also, the processor 111 of the vehicle 100 functions as the vehicle control unit 115 by executing the program PG1.

[0022] As will be described later in other embodiments, the data used for obtaining the vehicle position information to be described next may be three-dimensional point cloud data instead of image data. However, in the present embodiment, image data is used for obtaining the vehicle position information. In the following description, the image data may be described as a captured image. Also, the step S1 to be described next is performed by the position estimation unit 212.

[0023] In step S1, the processor 201 of the server 200 acquires vehicle position information using the detection results output from the external sensor 300. The vehicle position information is the position information that forms the basis for generating the driving control signal. In this embodiment, the vehicle position information includes the position and orientation of the vehicle 100 in the global coordinate system GC of the factory FC. Specifically, in step S1, the processor 201 acquires vehicle position information using the captured image acquired from the camera 300A, which is the external sensor 300.

[0024] In detail, in step S1, the processor 201 detects the outline of the vehicle 100 from the captured image, calculates the coordinates of the vehicle 100's positioning point in the coordinate system of the captured image, i.e., the local coordinate system, and obtains the position of the vehicle 100 by converting the calculated coordinates to coordinates in the global coordinate system GC. The outline of the vehicle 100 included in the captured image can be detected, for example, by inputting the captured image into a detection model DM that utilizes artificial intelligence. The detection model DM is prepared, for example, within or outside the system 50 and pre-stored in the memory 202 of the server 200. Examples of the detection model DM include a pre-trained machine learning model that has been trained to implement either semantic segmentation or instance segmentation. As this machine learning model, for example, a convolutional neural network (CNN) trained by supervised learning using a training dataset can be used. The training dataset includes, for example, multiple training images including the vehicle 100, and labels indicating whether each region in the training image represents the vehicle 100 or a region other than the vehicle 100. During CNN training, it is preferable that the CNN parameters be updated using backpropagation to reduce the error between the output result of the detection model DM and the label. Furthermore, the processor 201 can obtain the orientation of vehicle 100 by, for example, using the optical flow method, estimating the orientation of the vehicle 100's movement vector calculated from the positional changes of the vehicle 100's feature points between frames of the captured image.

[0025] In this embodiment, the training dataset used to train the detection model DM is labeled for each region in the training images. For example, each region such as vehicle 100, track TR, and equipment used for manufacturing is labeled.

[0026] In step S2, the processor 201 of the server 200 determines the next target location that the vehicle 100 should head to. In this embodiment, the target location is represented by X, Y, Z coordinates in the global coordinate system GC. The memory 202 of the server 200 pre-stores a reference route RR, which is the path that the vehicle 100 should travel. The route is represented by a node indicating the starting point, nodes indicating waypoints, a node indicating the destination, and links connecting each node. The processor 201 uses the vehicle position information and the reference route RR to determine the next target location that the vehicle 100 should head to. The processor 201 determines the target location on the reference route RR beyond the vehicle 100's current location.

[0027] In step S3, the processor 201 of the server 200 generates a driving control signal to drive the vehicle 100 toward the determined target position. The processor 201 calculates the vehicle's speed from the change in the vehicle's position and compares the calculated speed with the target speed. Overall, the processor 201 determines the acceleration so that the vehicle 100 accelerates if the speed is lower than the target speed, and determines the acceleration so that the vehicle 100 decelerates if the speed is higher than the target speed. Furthermore, if the vehicle 100 is located on the reference path RR, the processor 201 determines the steering angle and acceleration so that the vehicle 100 does not deviate from the reference path RR, and if the vehicle 100 is not located on the reference path RR, in other words, if the vehicle 100 has deviated from the reference path RR, the processor 201 determines the steering angle and acceleration so that the vehicle 100 returns to the reference path RR.

[0028] In step S4, the processor 201 of the server 200 transmits the generated driving control signal to the vehicle 100. The processor 201 repeats the acquisition of vehicle position information, determination of target position, generation of driving control signal, and transmission of driving control signal at predetermined intervals.

[0029] In step S5, the processor 111 of the vehicle 100 receives a driving control signal transmitted from the server 200. In step S6, the processor 111 of the vehicle 100 controls the actuator group 120 using the received driving control signal, thereby driving the vehicle 100 at the acceleration and steering angle indicated in the driving control signal. The processor 111 repeats the reception of the driving control signal and the control of the actuator group 120 at predetermined intervals. According to the system 50 in this embodiment, the vehicle 100 can be driven by remote control, and the vehicle 100 can be moved without using transport equipment such as cranes or conveyors.

[0030] Figure 4 illustrates the relationship between the imaging area IA captured by camera 300A and the measurement area MA measured by distance measuring device 300B. As shown in Figure 4, camera 300A is positioned so that the vehicle 100 traveling on the track TR is included in the imaging area IA.

[0031] Incidentally, there are cases where objects OBJ are placed around the track TR by workers, for example. For example, in a manufacturing line within a factory FC where tasks such as assembling and inspecting parts are performed on a moving vehicle 100, workers may place objects OBJ such as shelves or marker cones for their work. If an object OBJ is placed between the vehicle 100 and the camera 300A, or if an object OBJ is placed behind the vehicle 100, the positional accuracy of the vehicle 100 estimated in step S1 may decrease. This is because the data of the object OBJ, which is not included in the image data used in the training dataset used to train the detection model DM, is included in the image data, which may reduce the accuracy of segmentation. As a specific example, the detection model DM may identify the region in the image where this object OBJ and the vehicle 100 are merged as the vehicle 100. Note that the decrease in segmentation accuracy can occur not only when an object OBJ is placed, but also, for the same reason, when an object OBJ included in the image data used in the training dataset is removed. The placement of new object OBJs or the removal of existing object OBJs by workers or other personnel is also called rearrangement.

[0032] Therefore, in this embodiment, when it is detected that the object OBJ has been rearranged, the detection model DM is retrained. This suppresses the decrease in the position accuracy of the vehicle 100 estimated in step S1. This embodiment is applied to an area within the factory FC where the camera 300A and the distance measuring device 300B are redundantly arranged on the track TR. In this embodiment, the track TR on which the camera 300A and the distance measuring device 300B are arranged is located on the manufacturing line.

[0033] As shown in Figure 4, the measurement area MA includes at least a portion of the imaging area IA. In other words, the distance measuring device 300B is positioned to measure at least a portion of the imaging area IA. Note that the relationship between the measurement area MA and the measurement area MA is not limited to the example shown in Figure 4. For example, the measurement area MA may be approximately the same as the imaging area IA. Also, the measurement area MA may be set so that the distance measuring device 300B measures from an angle different from the angle at which the camera 300A images the vehicle 100. By positioning the distance measuring device 300B to measure at least a portion of the imaging area IA, it is possible to detect any rearrangement of the object OBJ that has occurred in at least a portion of the imaging area IA.

[0034] Figure 5 is a flowchart showing the processing procedure for the retraining process. Server 200 performs the retraining process before manufacturing begins on the production line. Specifically, before manufacturing begins on the production line is before the continuous operation of multiple vehicles 100 for manufacturing begins on a day when the factory FC is in operation. Typically, the retraining process is performed in the morning on a day when the factory FC is in operation. Workers are likely to rearrange object OBJs while manufacturing using the production line is stopped. Therefore, by performing the retraining process before manufacturing begins, the retraining process can be performed after the placement of object OBJs has been finalized, thus making the retraining process more efficient.

[0035] The acquisition unit 211 stores one 3D point cloud data set per day in the memory 202. Then, in the retraining process, the learning unit 213 sets the previous day's 3D point cloud data as the reference data 221. In this embodiment, the reference data 221 is data from a day when the factory FC is in operation, after manufacturing has finished and the vehicle 100 is not running. In this embodiment, the server 200 performs the retraining process before the vehicle 100 starts running at the factory FC. In other words, neither the image data nor the 3D point cloud data used in the retraining process contains data of the vehicle 100.

[0036] In step S11, which is a decision-making process, the learning unit 213 determines whether there is a difference between the current 3D point cloud data and the reference data 221 stored the previous day, which is past 3D point cloud data. Specifically, the learning unit 213 determines that there is a difference if the difference is outside a predetermined reference range. The reference range is larger than the measurement error. In other words, the learning unit 213 determines that there is a difference if the difference is larger than the measurement error. In another embodiment, the learning unit 213 may also process the data to determine that there is a difference if multiple points with differences are detected together, that is, if the points with differences form a region. In this case, false detections can be suppressed and the presence or absence of rearrangement can be detected with high accuracy.

[0037] In step S11, if it is determined that there is no difference, there is no need to relearn, and the learning unit 213 terminates this processing routine. In step S11, if it is determined that there is a difference, a rearrangement has occurred, and in step S12, the learning unit 213 determines whether a predetermined inspection time has elapsed since the start of this processing routine. In this embodiment, the inspection time is about 10 minutes. In step S12, if it is determined that the inspection time has not elapsed, in step S13, the learning unit 213 waits for a predetermined waiting time after performing step S12, and then returns to the processing step S11. In this embodiment, the waiting time is about 3 minutes.

[0038] In step S12, if the learning unit 213 determines that the inspection time has elapsed, it proceeds to step S14. In other words, in this embodiment, step S14 is performed if the judgment result that a difference is found in step S11 occurs three times in a row. There are cases where an operator temporarily places an object OBJ. Therefore, by performing step S14 when the judgment result that a difference is found in step S11 occurs consecutively, step S14 is performed only when the object OBJ is permanently placed, thus avoiding unnecessary retraining. The waiting time should be long enough for the object OBJ to move, preferably several minutes.

[0039] In step S14, the learning unit 213 retrains the detection model DM using the current image data transmitted from the camera 300A. This allows retraining to be performed using the image data after the rearrangement. Specifically, the learning unit 213 first defines the regions in the image data corresponding to the regions where differences were detected in the 3D point cloud data by assigning an unused label, such as "Unknown Object A". Next, the learning unit 213 retrains the detection model DM using the training dataset containing the labeled image data. As a result, the detection model DM can recognize object OBJ in the image data.

[0040] After performing step S14, the learning unit 213 terminates this processing routine. When manufacturing begins, the position estimation unit 212 acquires vehicle position information using the retrained detection model DM. Therefore, the remote control unit 210 can reduce the discrepancy between the actual position of the vehicle 100 and the estimated position, and control the vehicle 100 with high accuracy.

[0041] The process step performed by the acquisition unit 211 to acquire image data and 3D point cloud data is also called the acquisition step. The process step performed by the position estimation unit 212 to estimate the position of the vehicle 100 using the outline of the vehicle 100 obtained by inputting image data into the detection model DM is also called the position estimation step. Step S14 is also called the learning step.

[0042] According to the first embodiment described above, the system 50 comprises a camera 300A, a distance measuring device 300B, an acquisition unit 211, a position estimation unit 212, a remote control unit 210 as a control unit, and a learning unit 213. The acquisition unit 211 acquires image data captured by the camera 300A and 3D point cloud data measured by the distance measuring device 300B. In step S11, if there is a difference between past 3D point cloud data and current 3D point cloud data, the learning unit 213 retrains the detection model DM using the current image data in step S13. This improves the detection accuracy of the vehicle 100's outline using the detection model DM. Therefore, a decrease in the accuracy of the position estimation unit 212's estimation of the vehicle 100's position can be suppressed.

[0043] Furthermore, the learning unit 213 sets the previous day's 3D point cloud data as the reference data 221. Then, in the retraining process, the learning unit 213 performs step S11 multiple times to determine whether or not there is a difference between the reference data 221 and the current 3D point cloud data. If the determination results indicate a difference consecutively, the detection model DM is retrained in step S14. This allows step S14 to be performed only when the object OBJ is permanently placed, thus avoiding unnecessary retraining.

[0044] Furthermore, the learning unit 213 determines in step S11 whether there are any differences between past 3D point cloud data and current 3D point cloud data before manufacturing begins on the production line. This allows the retraining process to be performed after the placement of the object OBJ is finalized, thus enabling efficient retraining.

[0045] B. Second Embodiment: Figure 6 is an explanatory diagram showing the schematic configuration of system 50v in the second embodiment. In this embodiment, system 50v differs from the first embodiment in that it does not have a server 200. Also, in this embodiment, vehicle 100v can be driven by autonomous control of vehicle 100v. The other configurations are the same as in the first embodiment unless otherwise specified.

[0046] In this embodiment, the processor 111v of the vehicle control device 110v functions as a vehicle control unit 115v by executing the program PG1 stored in memory 112v. The vehicle control unit 115v acquires the output results from the sensors, generates a driving control signal using the output results, and outputs the generated driving control signal to operate the actuator group 120, thereby enabling the vehicle 100v to be driven autonomously. In this embodiment, in addition to the program PG1, the detection model DM and the reference path RR are pre-stored in memory 112v.

[0047] In this embodiment, the processor 111v includes, in addition to the above configuration, an acquisition unit 116, a position estimation unit 117, and a learning unit 118. The memory 112v stores reference data 121 in addition to the above configuration. The acquisition unit 116, the position estimation unit 117, and the learning unit 118 are functional units realized by the processor 111v executing the program PG1. Each of the acquisition unit 116, the position estimation unit 117, and the learning unit 118 functions in the same way as the acquisition unit 211, the position estimation unit 212, and the learning unit 213 of the first embodiment. As a result, when a rearrangement occurs, the learning unit 118 can retrain the detection model DM using the current image data.

[0048] Figure 7 is a flowchart showing the processing procedure for vehicle 100V's driving control in the second embodiment. In the processing procedure shown in Figure 7, the vehicle 100V's processor 111V functions as a vehicle control unit 115V by executing program PG1.

[0049] In step S901, the processor 111v of the vehicle control device 110v acquires vehicle position information using the detection result output from the camera, which is an external sensor 300. In step S902, the processor 111v determines the target position to which the vehicle 100v should next go. In step S903, the processor 111v generates a driving control signal to drive the vehicle 100v toward the determined target position. In step S904, the processor 111v controls the actuator group 120 using the generated driving control signal to drive the vehicle 100v according to the parameters expressed in the driving control signal. The processor 111v repeats the acquisition of vehicle position information, determination of the target position, generation of the driving control signal, and control of the actuators at a predetermined cycle. According to the system 50v in this embodiment, the vehicle 100v can be driven by autonomous control of the vehicle 100v without remote control of the vehicle 100v by the server 200.

[0050] Similar to the first embodiment, the position estimation unit 117 estimates the position of the vehicle 100v using the retrained detection model DM. Therefore, a decrease in the accuracy of the position estimation unit 117's estimation of the vehicle 100v's position can be suppressed.

[0051] C. Other embodiments (other embodiments related to relearning): (C1) In the first embodiment described above, in step S11, it is determined whether or not there is a difference between the reference data 221 and the current 3D point cloud data. In other embodiments, in the second and subsequent determination processes, if there is no difference between the 3D point cloud data used in the previous determination process and the newly acquired current 3D point cloud data, it may be determined that there is a difference between the reference data 221 and the current 3D point cloud data. In the first embodiment described above, the determination process in step S11 is performed three times, and if a difference is determined in any of the three steps of step S11, the retraining in step S14 is performed. In other embodiments, the number of times step S11 is performed may be more than three. If the number of times step S11 is performed is three or more, the retraining in step S14 may be performed if a difference is determined at least two consecutive times in the most recent determination process. In this case as well, it can be determined that the object OBJ is permanently placed.

[0052] (C2) In the first embodiment described above, the 3D point cloud data does not include data for the vehicle 100. In other embodiments, the 3D point cloud data may include data for the vehicle 100. In this case, the learning unit 213 focuses on the region of the 3D point cloud data excluding the region corresponding to the vehicle 100, and performs retraining if there is a difference between the past 3D point cloud data and the current 3D point cloud data. For example, the learning unit 213 estimates the region of the vehicle 100 in the 3D point cloud data by template matching using pre-prepared reference point cloud data for both the past 3D point cloud data and the current 3D point cloud data. The reference point cloud data is data that represents the shape of the vehicle 100. Next, the learning unit 213 creates two sets of data by merging the region of the vehicle 100 in the past 3D point cloud data and the region of the vehicle 100 in the current 3D point cloud data, and deleting that region from both the past 3D point cloud data and the current 3D point cloud data. The learning unit 213 then uses the difference data between these two sets of data to determine whether or not an object OBJ exists. With this method, the learning unit 213 can determine whether or not there is a difference in the area of ​​the imaging region IA excluding the vehicle 100, even if the 3D point cloud data includes data for the vehicle 100. Note that the image data used for retraining may or may not include data for the vehicle 100. If the image data used for retraining includes data for the vehicle 100, a label indicating the vehicle is assigned to the area of ​​the vehicle 100.

[0053] (C3) In the first embodiment described above, in step S14, the learning unit 213 defines the region in the image data corresponding to the region in the 3D point cloud data where a difference has been detected by assigning an unused temporary label. In another embodiment, a person may look at the image data displayed on the display screen and define the region corresponding to the rearranged object OBJ and assign a specific label. This allows for accurate definition of regions and labels. Furthermore, in this case, the person may look at the image data displayed on the display screen and determine whether retraining is necessary for the region in the 3D point cloud data where a difference has been detected, and if it is determined that retraining is necessary, the process may be to perform retraining. By involving human judgment, unnecessary retraining due to false detections can be avoided, for example. In addition, after the learning unit 213 assigns a temporary label (e.g., "Unknown Object A"), a person may change the label defined by the learning unit 213 to a specific label (e.g., "Shelf"). This improves the accuracy of segmentation. In this case, the timing of a person changing to a specific label does not have to be the same as the timing of when the learning unit 213 assigns a provisional label such as "unknown object". Specifically, after multiple object OBJs that the learning unit 213 has assigned provisional labels to have been accumulated, a person may assign specific labels to multiple object OBJs all at once.

[0054] (C4) In the first embodiment described above, step S11 is repeated during the relearning process. In another embodiment, the relearning process may be configured so that step S11 is performed only once. This reduces the processing load of repeatedly performing step S11. In addition, in the first embodiment described above, the learning unit 213 performs the relearning process before manufacturing begins on the production line. The timing of the relearning process is not limited to before the start of manufacturing. For example, the relearning process may be performed in the middle of manufacturing. Regardless of the timing, performing the relearning process can suppress a decrease in the detection accuracy of the vehicle 100's external shape when rearrangement occurs.

[0055] D. Other embodiments (other embodiments related to self-propelled transport): (D1) In each of the above embodiments, the external sensor 300 is not limited to the camera 300A, but may be, for example, a distance measuring device 300B. The distance measuring device 300B is, for example, LiDAR (Light Detection And Ranging). In this case, the detection result output by the external sensor 300 may be 3D point cloud data representing the vehicle 100. In this case, the server 200 and the vehicle 100 may acquire vehicle position information by template matching using the 3D point cloud data as the detection result and pre-prepared reference point cloud data.

[0056] (D2) In the first embodiment described above, the server 200 performs the processing from acquiring vehicle position information to generating a driving control signal. In contrast, the vehicle 100 may perform at least a part of the processing from acquiring vehicle position information to generating a driving control signal. For example, the following forms (1) to (3) may be used.

[0057] (1) The server 200 may acquire vehicle location information, determine the next target location that vehicle 100 should head to, and generate a route from the vehicle 100's current location, as shown in the acquired vehicle location information, to the target location. The server 200 may generate a route to the target location between the current location and the destination, or it may generate a route to the destination. The server 200 may transmit the generated route to vehicle 100. Vehicle 100 may generate a driving control signal so that vehicle 100 travels along the route received from the server 200, and may use the generated driving control signal to control the actuator group 120.

[0058] (2) The server 200 may acquire vehicle location information and transmit the acquired vehicle location information to the vehicle 100. The vehicle 100 may determine the next target location to which the vehicle 100 should go, generate a route from the vehicle 100's current location shown in the received vehicle location information to the target location, generate a driving control signal so that the vehicle 100 travels along the generated route, and control the actuator group 120 using the generated driving control signal.

[0059] (3) In the embodiments of (1) and (2) above, the vehicle 100 is equipped with internal sensors, and the detection results output from the internal sensors may be used in at least one of the generation of a route and the generation of a driving control signal. The internal sensors are sensors mounted on the vehicle 100. The internal sensors may include, for example, sensors that detect the motion state of the vehicle 100, sensors that detect the operating state of each part of the vehicle 100, and sensors that detect the environment around the vehicle 100. Specifically, the internal sensors may include, for example, cameras, LiDAR, millimeter-wave radar, ultrasonic sensors, GPS sensors, acceleration sensors, gyroscopes, etc. For example, in the embodiment of (1) above, the server 200 may acquire the detection results of the internal sensors and reflect the detection results of the internal sensors in the route when generating a route. In the embodiment of (1) above, the vehicle 100 may acquire the detection results of the internal sensors and reflect the detection results of the internal sensors in the driving control signal when generating a driving control signal. In the embodiment of (2) above, the vehicle 100 may acquire the detection results of the internal sensors and reflect the detection results of the internal sensors in the route when generating a route. In the embodiment described in (2) above, the vehicle 100 may acquire the detection results of the internal sensors and reflect the detection results of the internal sensors in the driving control signal when generating the driving control signal.

[0060] (D3) In the second embodiment described above, the vehicle 100v is equipped with an internal sensor, and the detection result output from the internal sensor may be used in at least one of the generation of the route and the generation of the driving control signal. For example, the vehicle 100v may acquire the detection result from the internal sensor and reflect the detection result from the internal sensor in the route when generating the route. The vehicle 100v may acquire the detection result from the internal sensor and reflect the detection result from the internal sensor in the driving control signal when generating the driving control signal.

[0061] (D4) In the second embodiment described above, the vehicle 100v acquires vehicle position information using the detection results of the external sensor 300. In contrast, the vehicle 100v may be equipped with an internal sensor, and the vehicle 100v may acquire vehicle position information using the detection results of the internal sensor, determine the target position to which the vehicle 100v should go next, generate a route from the vehicle 100v's current location to the target position as shown in the acquired vehicle position information, generate a driving control signal for driving along the generated route, and control the actuator group 120 using the generated driving control signal. In this case, the vehicle 100v can drive without using the detection results of the external sensor 300 at all. The vehicle 100v may also acquire the target arrival time and congestion information from outside the vehicle 100v and reflect the target arrival time and congestion information in at least one of the route and the driving control signal. Furthermore, all the functional configurations of the system 50v may be provided in the vehicle 100v. That is, the processing realized by the system 50v in this disclosure may be realized by the vehicle 100v alone.

[0062] (D5) In the first embodiment described above, the server 200 automatically generates a driving control signal to be transmitted to the vehicle 100. Alternatively, the server 200 may generate a driving control signal to be transmitted to the vehicle 100 in accordance with the operation of an external operator located outside the vehicle 100. For example, the external operator may operate a control device that includes a display for displaying captured images output from the external sensor 300, a steering wheel for remotely controlling the vehicle 100, an accelerator pedal, a brake pedal, and a communication device for communicating with the server 200 via wired or wireless communication, and the server 200 may generate a driving control signal in accordance with the operation applied to the control device.

[0063] (D6) In each of the above embodiments, the vehicle 100 only needs to have a configuration that allows it to move by unmanned operation, and may be in the form of a platform having the configuration described below. Specifically, in order for the vehicle 100 to perform the three functions of "driving," "turning," and "stopping" by unmanned operation, it is sufficient to have at least a vehicle control device 110 and an actuator group 120. When the vehicle 100 acquires information from the outside for unmanned operation, the vehicle 100 may further have a communication device 130. That is, the vehicle 100 that can move by unmanned operation does not need to have at least some of the interior parts such as the driver's seat and dashboard attached, it does not need to have at least some of the exterior parts such as the bumper and fender attached, and it does not need to have a body shell attached. In this case, the remaining parts such as the body shell may be attached to the vehicle 100 before the vehicle 100 is shipped from the factory FC, or the remaining parts such as the body shell may be attached to the vehicle 100 after the vehicle 100 has been shipped from the factory FC without the remaining parts such as the body shell attached to the vehicle 100. Each component may be attached to the vehicle 100 from any direction, such as the top, bottom, front, rear, right, or left side, and may be attached from the same direction or from different directions. The positioning of the platform can also be determined in the same way as for the vehicle 100 in the first embodiment.

[0064] (D7) Vehicle 100 may be manufactured by combining multiple modules. A module means a unit composed of one or more parts grouped together according to the configuration and function of vehicle 100. For example, the platform of vehicle 100 may be manufactured by combining a front module that constitutes the front part of the platform, a central module that constitutes the central part of the platform, and a rear module that constitutes the rear part of the platform. The number of modules that constitute the platform is not limited to three, but may be two or fewer, or four or more. In addition to the platform, or in place of the platform, parts of vehicle 100 other than the platform may be modularized. Various modules may also include any exterior parts such as bumpers and grilles, or any interior parts such as seats and consoles. Furthermore, not limited to vehicle 100, any type of mobile body may be manufactured by combining multiple modules. Such modules may be manufactured, for example, by joining multiple parts by welding or fasteners, or by integrally molding at least a part of the module as a single part by casting. The molding method of integrally molding at least a part of a module as a single part is also called gigacast or megacast. By using Gigacast, parts of a mobile body that were conventionally formed by joining multiple components can be formed as single components. For example, the front module, central module, and rear module mentioned above may be manufactured using Gigacast.

[0065] (D8) Transporting vehicle 100 using the unmanned operation of vehicle 100 is also called "autonomous transport." The configuration for realizing autonomous transport is also called a "vehicle remote control autonomous driving transport system." Furthermore, a production method that uses autonomous transport to produce vehicle 100 is also called "autonomous production." In autonomous production, for example, at a factory FC that manufactures vehicle 100, at least a portion of the transport of vehicle 100 is realized by autonomous transport.

[0066] (D9) In each of the above embodiments, some or all of the functions and processes implemented in software may be implemented in hardware. Also, some or all of the functions and processes implemented in hardware may be implemented in software. As hardware for implementing the various functions in each of the above embodiments, various circuits such as integrated circuits and discrete circuits may be used.

[0067] (D10) In the second embodiment described above, the system 50v does not include a server. In another embodiment, the system 50v may include a server equipped with a learning unit 213 similar to that of the first embodiment, and the retrained detection model DM may be provided to the vehicle control device 110v.

[0068] This disclosure is not limited to the embodiments described above, and can be implemented in various configurations without departing from its spirit. For example, the technical features of the embodiments corresponding to the technical features in each form described in the summary of the invention can be replaced or combined as appropriate in order to solve some or all of the above-described problems, or to achieve some or all of the above-described effects. Furthermore, if a technical feature is not described as essential in this specification, it can be deleted as appropriate. [Explanation of symbols]

[0069] 50, 50V... System, 100, 100V... Vehicle, 110, 110V... Vehicle control device, 111, 111V... Processor, 112, 112V... Memory, 113... Input / Output interface, 114... Internal bus, 115, 115V... Vehicle control unit, 116, 211... Acquisition unit, 117, 212... Position estimation unit, 118, 213... Learning unit, 120... Actuator group, 121, 221... Reference data, 13 0...Communication device, 200...Server, 201...Processor, 202...Memory, 203...Input / Output interface, 204...Internal bus, 205...Communication device, 210...Remote control unit, 300...External sensor, 300A...Camera, 300B...Distance measuring device, DM...Detection model, FC...Factory, IA...Imaging area, MA...Measurement area, OBJ...Object, PL1...First location, PL2...Second location, RR...Reference path, TR...Track

Claims

1. It is a system, A camera that captures an imaging area including the path of a moving object, A distance measuring device that performs measurements on at least a portion of the imaging area, An acquisition unit that acquires image data captured by the camera and 3D point cloud data measured by the distance measuring device, A position estimation unit estimates the position of the moving object by inputting the image data of the moving object captured into a trained machine learning model and using the shape of the moving object obtained from that shape, A system comprising: a learning unit that retrains the machine learning model using the current image data when there is a difference between the past three-dimensional point cloud data and the current three-dimensional point cloud data.

2. The system according to claim 1, The learning unit sets past 3D point cloud data as reference data, performs a judgment process multiple times to determine whether there is a difference between the reference data and the current 3D point cloud data, and retrains the machine learning model if the judgment results in the judgment process consecutively determine that there is a difference.

3. A system according to claim 1 or 2, The aforementioned track is provided on the manufacturing line, The learning unit is a system that determines whether or not there is a difference between past three-dimensional point cloud data and current three-dimensional point cloud data before manufacturing begins on the manufacturing line.

4. It is a device, An acquisition unit that acquires image data of an imaging area including the path of a moving object, and 3D point cloud data measured by a distance measuring device that measures at least a part of the imaging area, A position estimation unit estimates the position of the moving object by inputting the image data of the moving object captured into a trained machine learning model and using the shape of the moving object obtained from that shape, A device comprising: a learning unit that retrains the machine learning model using the current image data when there is a difference between the past three-dimensional point cloud data and the current three-dimensional point cloud data.

5. It is a method, An acquisition step of acquiring 3D point cloud data measured by a distance measuring device that measures at least a portion of the imaging area including the path of a moving object, A method comprising: a learning step of retraining a previously trained machine learning model to determine the outline of a moving object using the current image data of the imaging region, which includes the moving object, as input, when there is a difference between past three-dimensional point cloud data and current three-dimensional point cloud data.

Images