System, apparatus and method

Abstract

The present invention addresses the problem of suppressing a decrease in estimation accuracy of a vehicle position due to a decrease in detection accuracy of a vehicle profile. This system is provided with: a camera that captures an image capture region including a travel path of a moving body; a distance measuring device that measures at least a portion of the imaging region; an acquisition unit that acquires image data captured by the camera and three-dimensional point cloud data measured by the distance measuring device; a position estimation unit that estimates the position of the moving body using an outer shape of the moving body obtained by inputting image data of the moving body into the learned machine learning model; and a learning unit that re-learns the machine learning model using the current image data when there is a difference between the past three-dimensional point cloud data and the current three-dimensional point cloud data.

Description

Technical Field

[0001] This invention relates to a system, apparatus, and method. Background Technology

[0002] Patent document 1 discloses a control device for controlling a vehicle capable of autonomous driving. This control device uses a method of detecting the vehicle's shape by inputting images obtained from photographs of the vehicle into a trained machine learning model to obtain the vehicle's position.

[0003] Patent Document 1: Japanese Patent Application Publication No. 2024-49382 Summary of the Invention

[0004] When objects in images not reflected in the learning process of a machine learning model are positioned around the vehicle's path, the accuracy of vehicle shape detection using the machine learning model may decrease. Furthermore, decreased accuracy in vehicle shape detection leads to decreased accuracy in the estimated vehicle position. This problem applies not only to vehicles but also to moving bodies.

[0005] The present invention can be implemented in the following ways.

[0006] (1) According to a first aspect of the present invention, a system is provided. The system includes: a camera that captures an image area including the travel path of a moving object; a ranging device that measures at least a portion of the image area; an acquisition unit that acquires image data captured by the camera and three-dimensional point cloud data measured by the ranging device; a position estimation unit that estimates the position of the moving object by using the shape of the moving object obtained by inputting the image data containing the moving object into a learned machine learning model; and a learning unit that, in the event of a difference between past and current three-dimensional point cloud data, relearns the machine learning model using the current image data. According to this method, the learning unit can detect the presence of an object within the image area. Furthermore, in the event of an object within the image area, the learning unit uses the current image data to relearn the machine learning model. Therefore, the machine learning model can identify newly placed objects within the image area and distinguish between moving objects and objects, thus suppressing a decrease in the detection accuracy of the moving object's shape. Therefore, a decrease in the accuracy of the estimated position of the vehicle can be suppressed.

[0007] (2) In the system described above, the learning unit can set the past 3D point cloud data as reference data and perform multiple judgment processes to determine whether there is a difference between the reference data and the current 3D point cloud data. If the judgment results indicating a difference occur consecutively during the judgment process, the machine learning model is relearned. According to this method, relearning can be performed when the probability of the object being permanently configured is high. Therefore, unnecessary relearning can be avoided.

[0008] (3) In the system described above, the travel path can be set on a manufacturing line, and the learning unit determines whether there is a difference between the past 3D point cloud data and the current 3D point cloud data before manufacturing begins on the manufacturing line. Reconfiguration of objects on the manufacturing line is usually performed during manufacturing downtime. Therefore, according to this method, the probability of relearning after determining the object's reconfiguration is higher, thus enabling effective relearning processing.

[0009] (4) According to a second aspect of the present invention, an apparatus is provided. The apparatus comprises: an acquisition unit that acquires image data obtained by capturing a camera area including the travel path of a moving body, and three-dimensional point cloud data measured by a ranging device that measures at least a portion of the camera area; a position estimation unit that estimates the position of the moving body by inputting the image data of the moving body into a learned machine learning model and obtaining the shape of the moving body; and a learning unit that, in the event of a difference between past and current three-dimensional point cloud data, relearns the machine learning model using the current image data. According to this method, the decrease in the detection accuracy of the moving body's shape can be suppressed, thereby suppressing the decrease in the accuracy of the estimated position of the vehicle.

[0010] (5) According to a third aspect of the present invention, a method is provided. The method includes: an acquisition step, which acquires three-dimensional point cloud data determined by a ranging device that measures at least a portion of a camera area including the travel path of a moving body; and a learning step, which, in the event of a difference between past and current three-dimensional point cloud data, relearns a machine learning model that has been trained to determine the shape of the moving body using current image data obtained by photographing the camera area, wherein the image data is obtained by photographing the camera area including the moving body. According to this method, the decrease in the detection accuracy of the moving body's shape can be suppressed.

[0011] In addition to the system described above, the present invention can also be implemented by means of a non-temporary tangible recording medium in which a control device, control method, or control program can be readablely recorded by a computer. Attached Figure Description

[0012] Figure 1 This is a conceptual diagram showing the structure of the system in the first embodiment.

[0013] Figure 2 It is a block diagram representing the structure of the system.

[0014] Figure 3 This is a flowchart illustrating the processing steps for vehicle driving control.

[0015] Figure 4 It is a diagram illustrating the relationship between the imaging area and the measurement area.

[0016] Figure 5 This is a flowchart representing the processing steps of the relearning process.

[0017] Figure 6 This is a block diagram illustrating the structure of the vehicle in the second embodiment.

[0018] Figure 7 This is a flowchart illustrating the processing steps for vehicle driving control in the second embodiment. Detailed Implementation

[0019] A. Implementation Method 1: Figure 1 This is a conceptual diagram showing the structure of system 50 in the first embodiment. System 50 includes one or more vehicles 100 as mobile bodies, a server 200, and one or more external sensors 300.

[0020] In this invention, "mobile body" refers to an object capable of movement, such as a vehicle or an electric vertical takeoff and landing aircraft (so-called flying car). A vehicle can be a wheeled vehicle or a vehicle that travels on an unlimited track, such as a sedan, truck, bus, two-wheeled vehicle, four-wheeled vehicle, or engineering vehicle. Vehicles include battery electric vehicles (BEVs), gasoline vehicles, hybrid electric vehicles, and fuel cell vehicles. When the mobile body is not a vehicle, the terms "vehicle" or "car" in this invention can be appropriately replaced with "mobile body," and the term "travel" can be appropriately replaced with "move."

[0021] Vehicle 100 is configured to operate autonomously. "Autonomous driving" refers to driving without relying on occupant-based driving operations. Driving operations refer to operations related to at least one of "driving," "turning," or "stopping" of vehicle 100. Autonomous driving is achieved through automatic or manual remote control using devices located outside vehicle 100, or through autonomous control of vehicle 100. Vehicle 100 operating autonomously may carry occupants who do not perform driving operations. Occupants who do not perform driving operations include, for example, people who simply sit in the seats of vehicle 100, or people who perform tasks different from driving operations such as assembly, inspection, or switching operations while seated in vehicle 100. Furthermore, driving based on occupant-based driving operations is sometimes referred to as "manned driving."

[0022] In this specification, "remote control" includes all "fully remote control" where the actions of vehicle 100 are completely determined from outside vehicle 100, and "partially remote control" where a portion of the actions of vehicle 100 are determined from outside vehicle 100. Furthermore, "autonomous control" includes "fully autonomous control" where vehicle 100 autonomously controls its own actions without receiving any information from external devices, and "partially autonomous control" where vehicle 100 autonomously controls its own actions using information received from external devices.

[0023] In this embodiment, system 50 is used in a factory FC that manufactures vehicle 100. The reference coordinate system of the factory FC is the global coordinate system GC, and any position within the factory FC can be represented by the X, Y, and Z coordinates in the global coordinate system GC. The factory FC has a first location PL1 and a second location PL2. The first location PL1 and the second location PL2 are connected by a travel path TR that the vehicle 100 can travel on. In the factory FC, multiple external sensors 300 are arranged along the travel path TR. The positions of each external sensor 300 in the factory FC are pre-adjusted. The vehicle 100 moves from the first location PL1 to the second location PL2 via the travel path TR in an autonomous manner.

[0024] Figure 2 This is a block diagram showing the structure of system 50. Vehicle 100 includes: a vehicle control unit 110 for controlling various parts of vehicle 100; an actuator assembly 120 including one or more actuators driven under the control of the vehicle control unit 110; and a communication unit 130 for communicating wirelessly with external devices such as server 200. Actuator assembly 120 includes actuators for a drive mechanism to accelerate vehicle 100, actuators for a steering mechanism to change the direction of travel of vehicle 100, and actuators for a braking mechanism to decelerate vehicle 100.

[0025] The vehicle control unit 110 comprises a computer having a processor 111, a memory 112, an input / output interface 113, and an internal bus 114. The processor 111, memory 112, and input / output interface 113 are connected bidirectionally via the internal bus 114. An actuator assembly 120 and a communication device 130 are connected to the input / output interface 113. The processor 111 executes a program PG1 stored in the memory 112 to perform various functions, including those of the vehicle control unit 115.

[0026] The vehicle control unit 115 drives the vehicle 100 by controlling the actuator assembly 120. The vehicle control unit 115 controls the actuator assembly 120 using a driving control signal received from the server 200, thereby enabling the vehicle 100 to drive. The driving control signal is a control signal used to drive the vehicle 100. In this embodiment, the driving control signal includes the acceleration and steering angle of the vehicle 100 as parameters. In other embodiments, the driving control signal may replace the acceleration of the vehicle 100, or may include the speed of the vehicle 100 as a parameter in addition to the acceleration of the vehicle 100.

[0027] Server 200 comprises a computer having a processor 201, a memory 202, an input / output interface 203, and an internal bus 204. The processor 201, memory 202, and input / output interface 203 are connected bidirectionally via the internal bus 204. A communication device 205, used for communicating with various external devices of server 200, is connected to the input / output interface 203. The communication device 205 can communicate wirelessly with vehicle 100 and can communicate with various external sensors 300 via wired or wireless communication. The processor 201 executes a program PG2 stored in memory 202 to perform various functions, including those of a remote control unit 210.

[0028] The remote control unit 210 acquires sensor-based detection results, uses these results to generate a driving control signal for controlling the actuator assembly 120 of the vehicle 100, and sends the driving control signal to the vehicle 100, thereby enabling the vehicle 100 to move remotely. The remote control unit 210 can not only generate and output driving control signals, but also, for example, generate and output control signals for controlling actuators that operate various auxiliary devices or devices such as windshield wipers, power windows, or lights on the vehicle 100. In other words, the remote control unit 210 can remotely control these various devices or auxiliary devices to operate.

[0029] In addition to the above-described structure, processor 201 also includes an acquisition unit 211, a position estimation unit 212, and a learning unit 213. The acquisition unit 211, position estimation unit 212, and learning unit 213 are functional units implemented by executing program PG2. The acquisition unit 211 acquires image data captured by camera 300A (described later) and three-dimensional point cloud data measured by ranging device 300B (described later). The position estimation unit 212 estimates the position of vehicle 100 by using the shape of vehicle 100 obtained by inputting image data into a detection model DM, which is a machine learning model that has completed its training. If there is a difference between past and current three-dimensional point cloud data, the learning unit 213 relearns the detection model DM using the current image data.

[0030] The external sensor 300 is a sensor located outside the vehicle 100. In this embodiment, the external sensor 300 is a sensor that detects the vehicle 100 from outside the vehicle 100. The external sensor 300 has a communication device (not shown) and can communicate with other devices such as the server 200 via wired or wireless communication.

[0031] Specifically, the external sensor 300 consists of a camera 300A or a ranging device 300B. The camera 300A, acting as the external sensor 300, captures images of the vehicle 100 and outputs image data as a detection result. The ranging device 300B, also acting as the external sensor 300, captures images of the vehicle 100 and outputs three-dimensional point cloud data as a detection result. The ranging device may be, for example, a Light Detection and Ranging (LiDAR) system.

[0032] Figure 3 This is a flowchart illustrating the processing steps for driving control of the vehicle 100 in the first embodiment. Figure 3 In the processing steps, the processor 201 of the server 200 functions as a remote control unit 210 by executing program PG2. Furthermore, the processor 111 of the vehicle 100 functions as a vehicle control unit 115 by executing program PG1.

[0033] As in other embodiments, as described later, the data used to obtain the vehicle position information described next may be three-dimensional point cloud data instead of image data, but in this embodiment, image data is used to obtain the vehicle position information. In the following description, image data may sometimes be described as camera images. Furthermore, step S1 described next is performed by the position estimation unit 212.

[0034] In step S1, the processor 201 of the server 200 uses the detection results output from the external sensor 300 to acquire vehicle position information. This vehicle position information is the basis for generating driving control signals. In this embodiment, the vehicle position information includes the position and orientation of the vehicle 100 in the global coordinate system GC of the factory FC. Specifically, in step S1, the processor 201 uses camera images acquired from the camera 300A, which is the external sensor 300, to acquire the vehicle position information.

[0035] Specifically, in step S1, the processor 201 obtains the position of the vehicle 100, for example, by detecting the shape of the vehicle 100 from the camera image, calculating the coordinates of the vehicle 100's location points in the local coordinate system of the camera image, and converting the calculated coordinates into coordinates in the global coordinate system GC. The shape of the vehicle 100 contained in the camera image can be detected, for example, by inputting the camera image into a detection model DM utilizing artificial intelligence. The detection model DM is prepared, for example, within or outside the system 50 and pre-stored in the memory 202 of the server 200. As the detection model DM, for example, a learned machine learning model capable of either semantic segmentation or instance segmentation can be used. As this machine learning model, for example, a convolutional neural network (hereinafter referred to as CNN) learned through supervised learning using a learning dataset can be used. The learning dataset, for example, has multiple training images including the vehicle 100 and labels indicating whether each region in the training images represents a region of the vehicle 100 or a region other than the vehicle 100. During CNN learning, it is preferable to update the CNN parameters by using backpropagation (error backpropagation method) to reduce the error between the output of the detection model DM and the label. Furthermore, the processor 201 can, for example, obtain the orientation of the vehicle 100 by using optical flow and estimating the orientation of the vehicle 100 based on the positional changes of the feature points of the vehicle 100 between frames of the camera image.

[0036] In this embodiment, the learning dataset used to learn the detection model DM assigns labels to each region in the training images. For example, labels are assigned to the regions of vehicle 100, driving path TR, and manufacturing equipment.

[0037] In step S2, the processor 201 of the server 200 determines the next target location that the vehicle 100 should travel to. In this embodiment, the target location is represented by the X, Y, and Z coordinates in the global coordinate system GC. The memory 202 of the server 200 pre-stores a reference path RR, which serves as the path that the vehicle 100 should travel. The path is represented by nodes indicating the origin, nodes indicating transit points, nodes indicating the destination, and links connecting the nodes. The processor 201 uses the vehicle position information and the reference path RR to determine the next target location that the vehicle 100 should travel to. The processor 201 determines the target location on the reference path RR that is ahead of the current position of the vehicle 100.

[0038] In step S3, the processor 201 of the server 200 generates a driving control signal to cause the vehicle 100 to move towards the determined target position. The processor 201 calculates the vehicle 100's speed based on the vehicle 100's position shift and compares the calculated speed with the target speed. Generally, when the speed is lower than the target speed, the processor 201 determines acceleration to make the vehicle 100 accelerate; when the speed is higher than the target speed, the processor determines acceleration to make the vehicle 100 decelerate. Furthermore, when the vehicle 100 is on the reference path RR, the processor 201 determines the steering angle and acceleration to prevent the vehicle 100 from deviating from the reference path RR; when the vehicle 100 is not on the reference path RR, in other words, when the vehicle 100 has deviated from the reference path RR, the processor determines the steering angle and acceleration to return the vehicle 100 to the reference path RR.

[0039] In step S4, the processor 201 of the server 200 sends the generated driving control signal to the vehicle 100. The processor 201 repeatedly performs tasks such as acquiring vehicle position information, determining target position, generating driving control signals, and sending driving control signals at a predetermined cycle.

[0040] In step S5, the processor 111 of vehicle 100 receives a driving control signal sent from server 200. In step S6, the processor 111 of vehicle 100 uses the received driving control signal to control the actuator assembly 120, causing vehicle 100 to travel at the acceleration and steering angle represented by the driving control signal. The processor 111 repeatedly receives the driving control signal and controls the actuator assembly 120 at a predetermined cycle. According to system 50 in this embodiment, vehicle 100 can be driven remotely, and vehicle 100 can be moved without using conveying equipment such as cranes or conveyor belts.

[0041] Figure 4This is a graph illustrating the relationship between the image area IA captured by camera 300A and the measurement area MA measured by rangefinder 300B. (Example) Figure 4 As shown, camera 300A is configured such that vehicle 100 traveling on driving path TR is included in camera area IA.

[0042] For example, sometimes workers place objects (OBJ) around the travel path TR. For instance, on a manufacturing line within a factory FC where operations such as assembling or inspecting parts of a moving vehicle 100 are performed, workers sometimes place objects (OBJ) such as shelves or marker cones used in the operation. When an object (OBJ) is placed between the vehicle 100 and the camera 300A, or behind the vehicle 100, the accuracy of the vehicle 100's position estimated in step S1 may decrease. This is because data on object (OBJ) not included in the image data used in the learning dataset for learning the detection model DM is included in the image data, potentially leading to a decrease in segmentation accuracy. Specifically, the detection model DM may identify the region in the image where the object (OBJ) is merged with the vehicle 100 as the vehicle 100. Furthermore, not only when object (OBJ) is placed, but also when object (OBJ) included in the image data used in the learning dataset is removed for the same reason, a decrease in segmentation accuracy also occurs. The placement of new object (OBJ) by workers or the removal of existing object (OBJ) is also referred to as reconfiguration.

[0043] Therefore, in this embodiment, when a reconfiguration of object OBJ is detected, the detection model DM is relearned. This suppresses the decrease in the positional accuracy of vehicle 100 estimated in step S1. Furthermore, this embodiment is applicable to areas within a factory FC where cameras 300A and ranging devices 300B are redundantly arranged along the travel path TR. In this embodiment, the travel path TR, where cameras 300A and ranging devices 300B are arranged, is located on a manufacturing line.

[0044] like Figure 4 As shown, the measurement area MA includes at least a portion of the camera area IA. That is, the ranging device 300B is configured to measure at least a portion of the camera area IA. Furthermore, the relationship between the measurement areas MA is not limited to... Figure 4 The example shown is as follows. For instance, the measurement area MA can be approximately the same as the camera area IA. Furthermore, the measurement area MA can also be set to be measured by the ranging device 300B from an angle different from the angle at which the vehicle 100 is photographed by the camera 300A. The ranging device 300B is configured to measure at least a portion of the camera area IA, thereby enabling the detection of object OBJ reconfiguration occurring within at least a portion of the camera area IA.

[0045] Figure 5 This is a flowchart illustrating the relearning process. Server 200 performs relearning processing before manufacturing begins on the manufacturing line. Specifically, this occurs before multiple vehicles 100 begin continuous operation for manufacturing on a date when the factory FC is running. Typically, relearning processing is performed in the morning of a date when the factory FC is running. There is a high probability that operators will reconfigure object OBJs during manufacturing downtime on the manufacturing line. Therefore, performing relearning processing before manufacturing begins allows for efficient relearning processing, as the object OBJ configuration can be determined afterward.

[0046] The acquisition unit 211 stores one 3D point cloud data from one day in the memory 202. Furthermore, in the relearning process, the learning unit 213 sets the previous day's 3D point cloud data as reference data 221. In this embodiment, the reference data 221 is data representing the state of the vehicle 100 after manufacturing has ended and before it has been driven on the date the factory FC is running. In this embodiment, the server 200 performs the relearning process before the vehicle 100 begins driving in the factory FC. That is, the image data and 3D point cloud data used in the relearning process do not include data about the vehicle 100.

[0047] In step S11 of the judgment process, the learning unit 213 determines whether there is a difference between the reference data 221 stored the previous day (which is past 3D point cloud data) and the current 3D point cloud data. Specifically, the learning unit 213 determines that a difference exists if the difference is outside a preset reference range. The reference range is greater than the measurement error. That is, the learning unit 213 determines that a difference exists if the difference is greater than the measurement error. Alternatively, as another embodiment, the learning unit 213 may be configured to process in the following way: when multiple points with differences are clustered together and detected, that is, when a region of points with differences is formed, a difference is determined to exist. In this case, false detections can be suppressed and reconfiguration can be detected with good accuracy.

[0048] In step S11, if no difference is determined, the learning unit 213 terminates the processing routine since no relearning is required. If a difference is determined to exist in step S11, and reconfiguration has been performed, the learning unit 213 determines in step S12 whether a preset check time has elapsed since the start of the processing routine. In this embodiment, the check time is approximately 10 minutes. If no check time has elapsed in step S12, in step S13, the learning unit 213 returns to step S11 after a preset waiting time following step S12. In this embodiment, the waiting time is approximately 3 minutes.

[0049] In step S12, if it is determined that the inspection time has elapsed, the learning unit 213 proceeds to step S14. That is, in this embodiment, step S14 is performed if the judgment result indicating a difference occurs three consecutively in step S11. Sometimes, operators temporarily configure the object OBJ. Therefore, by performing step S14 when the judgment result indicating a difference occurs consecutively in step S11, step S14 can be performed even if the object OBJ is permanently configured, thereby avoiding unnecessary relearning. Furthermore, the waiting time is a long period of time corresponding to the degree of movement of the object OBJ, preferably around a few minutes.

[0050] In step S14, the learning unit 213 relearns the detection model DM using the current image data sent from the camera 300A. This allows relearning using reconfigured image data. Specifically, the learning unit 213 first defines regions in the image data corresponding to the regions in the 3D point cloud data that differ from the detected regions by assigning unused labels, such as "unknown object A". Next, the learning unit 213 relearns the detection model DM using a learning dataset that includes the labeled image data. As a result, the detection model DM is able to identify object OBJ in the image data.

[0051] After performing step S14, the learning unit 213 terminates this processing routine. If manufacturing begins, the position estimation unit 212 uses the relearned detection model DM to acquire vehicle position information. Therefore, the remote control unit 210 can reduce the deviation between the actual position and the estimated position of the vehicle 100, and control the vehicle 100 with good accuracy.

[0052] The processing step in which the acquisition unit 211 acquires image data and 3D point cloud data is also called the acquisition step. The processing step in which the position estimation unit 212 estimates the position of the vehicle 100 using the shape of the vehicle 100 obtained by inputting image data to the detection model DM is also called the position estimation step. Step S14 is also called the learning step.

[0053] According to the first embodiment described above, the system 50 includes a camera 300A, a ranging device 300B, an acquisition unit 211, a position estimation unit 212, a remote control unit 210 serving as a control unit, and a learning unit 213. The acquisition unit 211 acquires image data captured by the camera 300A and three-dimensional point cloud data measured by the ranging device 300B. In step S11, if there is a difference between past and current three-dimensional point cloud data, the learning unit 213 relearns the detection model DM using the current image data. This improves the detection accuracy of the vehicle 100's shape using the detection model DM. Therefore, it is possible to suppress the decrease in the accuracy of the estimated position of the vehicle 100 based on the position estimation unit 212.

[0054] Furthermore, the learning unit 213 sets the previous day's 3D point cloud data as the baseline data 221. Then, in the relearning process, the learning unit 213 performs step S11 multiple times to determine whether there is a difference between the baseline data 221 and the current 3D point cloud data. If the determination result that there is a difference occurs consecutively, the detection model DM is relearned in step S14. Therefore, by performing step S14 when the object OBJ is permanently configured, unnecessary relearning can be avoided.

[0055] Furthermore, before manufacturing begins on the manufacturing line, the learning unit 213 determines in step S11 whether there are differences between past 3D point cloud data and current 3D point cloud data. This allows for relearning processing after the configuration of the object OBJ is determined, thus enabling effective relearning.

[0056] B. Second Implementation Method: Figure 6 This is an explanatory diagram showing the schematic structure of the system 50v in the second embodiment. In this embodiment, the system 50v differs from the first embodiment in that it does not include the server 200. Furthermore, the vehicle 100v in this embodiment is capable of autonomous operation through its own control. Other structural details are the same as in the first embodiment unless otherwise specified.

[0057] In this embodiment, the processor 111v of the vehicle control device 110v functions as the vehicle control unit 115v by executing the program PG1 stored in the memory 112v. The vehicle control unit 115v can acquire sensor-based output results, generate driving control signals using the output results, and output the generated driving control signals to activate the actuator assembly 120, thereby enabling autonomous control of the vehicle 100v. In this embodiment, in addition to the program PG1, the memory 112v also stores a detection model DM and a reference path RR in advance.

[0058] In this embodiment, the processor 111v, in addition to the structure described above, also includes an acquisition unit 116, a position estimation unit 117, and a learning unit 118. The memory 112v, in addition to the structure described above, also stores reference data 121. The acquisition unit 116, the position estimation unit 117, and the learning unit 118 are functional units implemented by the processor 111v executing the program PG1. The acquisition unit 116, the position estimation unit 117, and the learning unit 118 function in the same manner as the acquisition unit 211, the position estimation unit 212, and the learning unit 213 in the first embodiment. Therefore, in the event of a reconfiguration, the learning unit 118 can relearn the detection model DM using the current image data.

[0059] Figure 7 This is a flowchart illustrating the processing steps for the driving control of the vehicle 100V in the second embodiment. Figure 7 In the processing steps, the processor 111v of the vehicle 100v functions as the vehicle control unit 115v by executing program PG1.

[0060] In step S901, the processor 111v of the vehicle control device 110v acquires vehicle position information using detection results output from the camera, which is an external sensor 300. In step S902, the processor 111v determines the next target location that the vehicle 100v should travel to. In step S903, the processor 111v generates a driving control signal to cause the vehicle 100v to move toward the determined target location. In step S904, the processor 111v uses the generated driving control signal to control the actuator assembly 120, thereby causing the vehicle 100v to travel according to the parameters represented by the driving control signal. The processor 111v repeatedly acquires vehicle position information, determines the target location, generates the driving control signal, and controls the actuators at a predetermined cycle. According to the system 50v in this embodiment, even without remote control of the vehicle 100v via the server 200, the vehicle 100v can be driven autonomously.

[0061] Similar to the first embodiment, the position estimation unit 117 uses a relearned detection model DM to estimate the position of the vehicle 100v. Therefore, it is possible to suppress the decrease in accuracy of the estimated position of the vehicle 100v based on the position estimation unit 117.

[0062] C. Other implementation methods (other implementation methods related to relearning): (C1) In the first embodiment described above, in step S11, it is determined whether there is a difference between the reference data 221 and the current 3D point cloud data. As another embodiment, in subsequent determination processes, it can be determined whether there is a difference between the reference data 221 and the current 3D point cloud data if there is no difference between the 3D point cloud data used in the previous determination process and the newly acquired current 3D point cloud data. Furthermore, in the first embodiment described above, the determination process of step S11 is performed three times, and if a difference is determined to exist in any one of the three steps of S11, relearning in step S14 is performed. As another embodiment, the number of times step S11 is performed can be multiple times other than three times. If the number of times step S11 is performed is three or more, it can be set to perform relearning in step S14 if a difference is determined to exist at least twice consecutively in the most recent determination process. Even in this case, it can be determined that the object OBJ is permanently configured.

[0063] (C2) In the first embodiment described above, the three-dimensional point cloud data does not include data about vehicle 100. In other embodiments, the three-dimensional point cloud data may also include data about vehicle 100. In this case, the learning unit 213 focuses on the region in the three-dimensional point cloud data other than the region corresponding to vehicle 100, and performs relearning when there are differences between the past and current three-dimensional point cloud data. For example, the learning unit 213 estimates the region of vehicle 100 in the three-dimensional point cloud data by performing template matching on the past and current three-dimensional point cloud data using pre-prepared reference point cloud data. The reference point cloud data is data representing the shape of vehicle 100. Next, the learning unit 213 generates two sets of data from the past and current three-dimensional point cloud data, respectively, by removing the region formed by merging the region of vehicle 100 in the past and current three-dimensional point cloud data. Then, the learning unit 213 uses the difference between these two sets of data to determine whether object OBJ exists. According to this method, even when the 3D point cloud data includes data about vehicle 100, the learning unit 213 can determine whether there are differences in the areas of the camera region IA other than vehicle 100. Furthermore, the image data used for relearning may or may not include data about vehicle 100. When the image data used for relearning includes data about vehicle 100, the regions of vehicle 100 are labeled with vehicle identifiers.

[0064] (C3) In the first embodiment described above, in step S14, the learning unit 213 defines regions in image data corresponding to regions where differences are detected in the 3D point cloud data by assigning unused temporary labels. Alternatively, a person can observe the image data displayed on the screen to define regions corresponding to the reconfigured object OBJ and assign specific labels. This allows for accurate definition of regions and labels. Furthermore, the processing steps can be configured as follows: a person observes the image data displayed on the screen, determines whether regions where differences are detected in the 3D point cloud data need relearning, and performs relearning if it is determined that relearning is necessary. By using human judgment, unnecessary relearning due to false detections can be avoided, for example. Moreover, after the learning unit 213 assigns temporary labels (e.g., "unknown object A"), the person can change the labels defined by the learning unit 213 to specific labels (e.g., "shelf"). This improves segmentation accuracy. In this case, the timing of the person changing to specific labels can be different from the timing of the learning unit 213 assigning temporary labels such as "unknown object". Specifically, after multiple objects OBJ that have been temporarily tagged by the learning department 213 have been accumulated, a person can assign specific tags to multiple objects OBJ at the same time.

[0065] (C4) In the first embodiment described above, step S11 is repeatedly performed during the relearning process. Alternatively, the relearning process may be configured to perform step S11 only once. This reduces the processing load of repeatedly performing step S11. Furthermore, in the first embodiment described above, the learning unit 213 performs the relearning process before manufacturing begins on the manufacturing line. The timing of the relearning process is not limited to before manufacturing begins. For example, it may be performed during manufacturing. Regardless of the timing, by performing the relearning process, the decrease in the detection accuracy of the vehicle 100's shape can be suppressed during reconfiguration.

[0066] D. Other implementation methods (other implementation methods related to self-convection): (D1) In the above embodiments, the external sensor 300 is not limited to the camera 300A, but may also be a ranging device 300B. The ranging device 300B may be, for example, a Light Detection and Ranging (LiDAR) device. In this case, the detection result output by the external sensor 300 may also be three-dimensional point cloud data representing the vehicle 100. In this case, the server 200 or the vehicle 100 can obtain vehicle position information by using the three-dimensional point cloud data as the detection result and pre-prepared reference point cloud data for template matching.

[0067] (D2) In the first embodiment described above, the server 200 performs the process from the acquisition of vehicle location information to the generation of the driving control signal. Alternatively, the vehicle 100 may perform at least a portion of the process from the acquisition of vehicle location information to the generation of the driving control signal. For example, it may be performed in the manner described in (1) to (3) below.

[0068] (1) Server 200 can obtain vehicle location information, determine the next target location that vehicle 100 should go to, and generate a path from the current location of vehicle 100 as indicated in the obtained vehicle location information to the target location. Server 200 can generate a path from the current location to the target location, or a path to the destination. Server 200 can send the generated path to vehicle 100. Vehicle 100 can generate a driving control signal to make vehicle 100 drive on the path received from server 200, and use the generated driving control signal to control actuator group 120.

[0069] (2) Server 200 can acquire vehicle location information and send the acquired vehicle location information to vehicle 100. Vehicle 100 can determine the next target location that vehicle 100 should go to, generate a path from the current location of vehicle 100 as indicated in the received vehicle location information to the target location, generate a driving control signal to make vehicle 100 drive on the generated path, and use the generated driving control signal to control actuator group 120.

[0070] (3) In the methods described in (1) and (2) above, an internal sensor may be mounted on the vehicle 100, and the detection results output from the internal sensor may be used in at least one of the path generation and the generation of the driving control signal. The internal sensor is a sensor mounted on the vehicle 100. The internal sensor may include, for example, a sensor that detects the motion state of the vehicle 100, a sensor that detects the motion state of various parts of the vehicle 100, or a sensor that detects the surrounding environment of the vehicle 100. Specifically, the internal sensor may include, for example, a camera, LiDAR, millimeter-wave radar, ultrasonic sensor, GPS sensor, accelerometer, gyroscope sensor, etc. For example, in the method described in (1) above, the server 200 may acquire the detection results of the internal sensor and reflect the detection results of the internal sensor in the path when generating the path. In the method described in (1) above, the vehicle 100 may acquire the detection results of the internal sensor and reflect the detection results of the internal sensor in the driving control signal when generating the driving control signal. In the manner described in (2) above, the vehicle 100 can acquire the detection results of the internal sensors and reflect the detection results of the internal sensors in the path when generating the path. In the manner described in (2) above, the vehicle 100 can acquire the detection results of the internal sensors and reflect the detection results of the internal sensors in the driving control signal when generating the driving control signal.

[0071] (D3) In the second embodiment described above, an internal sensor may be mounted on the vehicle 100v, and the detection results output from the internal sensor may be used in at least one of the path generation and driving control signal generation. For example, the vehicle 100v may acquire the detection results of the internal sensor and reflect the detection results of the internal sensor in the path when generating the path. The vehicle 100v may acquire the detection results of the internal sensor and reflect the detection results of the internal sensor in the driving control signal when generating the driving control signal.

[0072] (D4) In the second embodiment described above, vehicle 100v uses the detection results of external sensor 300 to obtain vehicle position information. In contrast, an internal sensor can be mounted on vehicle 100v. Vehicle 100v can use the detection results of the internal sensor to obtain vehicle position information, determine the next target location to which vehicle 100v should go, generate a path from the current position of vehicle 100v as indicated in the acquired vehicle position information to the target location, generate a driving control signal for traveling on the generated path, and use the generated driving control signal to control actuator assembly 120. In this case, vehicle 100v can travel without using the detection results of external sensor 300. Furthermore, vehicle 100v can obtain target arrival time or congestion information from outside vehicle 100v and reflect the target arrival time or congestion information in at least one of the path and driving control signal. Moreover, the functional structure of system 50v can be entirely set within vehicle 100v. That is, the processing implemented by system 50v in this invention can be implemented by vehicle 100v alone.

[0073] (D5) In the first embodiment described above, the server 200 automatically generates a driving control signal to be sent to the vehicle 100. Alternatively, the server 200 can generate a driving control signal to be sent to the vehicle 100 based on the operation of an external operator located outside the vehicle 100. For example, the server 200 can generate a driving control signal corresponding to the operation applied to the driving control device by an external operator operating a control device equipped with a display showing camera images output from an external sensor 300, a steering wheel for remotely operating the vehicle 100, an accelerator pedal, a brake pedal, and a communication device for communicating with the server 200 via wired or wireless communication.

[0074] (D6) In the above embodiments, the vehicle 100 only needs to have a structure that enables it to move autonomously, for example, it can be in the form of a platform with the structure described below. Specifically, in order to perform the three functions of "driving", "turning", and "stopping" through autonomous driving, the vehicle 100 only needs to have a vehicle control device 110 and an actuator assembly 120. When the vehicle 100 obtains information from the outside for autonomous driving, the vehicle 100 only needs to further have a communication device 130. That is, the vehicle 100 that can move autonomously may not have at least a part of the interior components such as the driver's seat or dashboard, may not have at least a part of the exterior components such as bumpers or fenders, and may not have a body shell. In this case, during the period from when the vehicle 100 is shipped from the factory FC, the remaining components such as the body shell may be installed on the vehicle 100, or the remaining components such as the body shell may not be installed on the vehicle 100, and the remaining components such as the body shell may be installed on the vehicle 100 after the vehicle 100 is shipped from the factory FC. Each component can be installed from any direction, such as the top, bottom, front, rear, right, or left side of the vehicle 100, or from the same direction, or from different directions. Furthermore, the platform's shape and position can be determined in the same way as the vehicle 100 in the first embodiment.

[0075] (D7) Vehicle 100 can be manufactured by combining multiple modules. A module refers to a unit consisting of one or more parts combined according to the structure or function of vehicle 100. For example, the platform of vehicle 100 can be manufactured by combining a front module constituting the front part of the platform, a central module constituting the central part of the platform, and a rear module constituting the rear part of the platform. In addition, the number of modules constituting the platform is not limited to three, and may be two or less or four or more. Furthermore, in addition to or in place of the platform, parts of vehicle 100 that are different from the platform can also be modularized. Furthermore, various modules may include any exterior parts such as bumpers or grilles, or any interior parts such as seats or center consoles. Furthermore, not limited to vehicle 100, any kind of moving body can also be manufactured by combining multiple modules. Such modules can be manufactured, for example, by joining multiple parts using welding or fasteners, or by integrally molding at least a part of the module into a single part through casting. The molding method of integrally molding at least a part of the module into a single part is also called GigaCasting or Mega Casting. By using Giga Casting, the various parts of a mobile body can be formed into a single component, whereas previously they were formed by joining multiple parts together. For example, the aforementioned front module, central module, or rear module can be manufactured using Giga Casting.

[0076] (D8) The method of transporting vehicle 100 using the driving of driverless vehicle 100 is also referred to as "autonomous transport". Furthermore, the structure used to achieve autonomous transport is also referred to as a "vehicle remote-controlled autonomous driving transport system". Furthermore, the production method of producing vehicle 100 using autonomous transport is also referred to as "autonomous production". In autonomous production, for example in factory FC where vehicle 100 is manufactured, at least a portion of the transport of vehicle 100 is achieved through autonomous transport.

[0077] (D9) In the above embodiments, some or all of the functions and processes implemented in the software can also be implemented in the hardware. Furthermore, some or all of the functions and processes implemented in the hardware can also be implemented in the software. As the hardware for implementing the various functions in the above embodiments, various circuits such as integrated circuits or discrete circuits can be used.

[0078] (D10) In the second embodiment described above, the system 50v does not have a server. As another embodiment, the system 50v may also be configured to have a server having the same learning unit 213 as in the first embodiment, and to provide the relearned detection model DM to the vehicle control device 110v.

[0079] This invention is not limited to the embodiments described above, and can be implemented in various structures without departing from its spirit. For example, in order to solve some or all of the above-described problems, or to achieve some or all of the above-described effects, the technical features of the embodiments corresponding to the technical features in the various methods described in the Summary of the Invention section can be appropriately replaced or combined. Furthermore, if a technical feature is not described as an essential feature in this specification, it can be appropriately deleted.

[0080] Symbol Explanation 50, 50V - System; 100, 100V - Vehicle; 110, 110V - Vehicle control unit; 111, 111V - Processor; 112, 112V - Memory; 113 - Input / output interface; 114 - Internal bus; 115, 115V - Vehicle control unit; 116, 211 - Acquisition unit; 117, 212 - Position estimation unit; 118, 213 - Learning unit; 120 - Actuator group; 121, 221 - Reference data; 130 - Communication. Device, 200-Server, 201-Processor, 202-Memory, 203-Input / Output Interface, 204-Internal Bus, 205-Communication Device, 210-Remote Control Unit, 300-External Sensor, 300A-Camera, 300B-Range Measuring Device, DM-Detection Model, FC-Factory, IA-Camera Area, MA-Measurement Area, OBJ-Object, PL1-Location 1, PL2-Location 2, RR-Reference Path, TR-Travel Path.

Claims

1. A system, characterized by Possessing: a camera that photographs a photographing region including a travel path of a mobile body; a distance measuring device that measures at least a part of the photographing region; an acquisition section that acquires image data photographed by the camera and three-dimensional point cloud data measured by the distance measuring device; a position estimation section that estimates a position of the mobile body using an outline of the mobile body calculated by inputting the image data in which the mobile body is photographed to a machine learning model that has completed learning; and a learning section that relearns the machine learning model using current image data in a case where there is a difference between past three-dimensional point cloud data and current three-dimensional point cloud data.

2. The system according to claim 1, wherein the learning section sets the past three-dimensional point cloud data as reference data, and performs a plurality of determination processes that determine whether there is a difference between the reference data and the current three-dimensional point cloud data, and relearns the machine learning model in a case where a determination result in which it is determined that there is a difference continuously occurs in the determination processes.

3. The system according to claim 1 or 2, wherein the travel path is provided on a manufacturing line, the learning section determines whether there is a difference between the past three-dimensional point cloud data and the current three-dimensional point cloud data before manufacturing on the manufacturing line is started.

4. An apparatus, characterized by Possessing: an acquisition section that acquires image data obtained by photographing a photographing region including a travel path of a mobile body, and three-dimensional point cloud data measured by a distance measuring device that measures at least a part of the photographing region; a position estimation section that estimates a position of the mobile body using an outline of the mobile body calculated by inputting the image data in which the mobile body is photographed to a machine learning model that has completed learning; and a learning section that relearns the machine learning model using current image data in a case where there is a difference between past three-dimensional point cloud data and current three-dimensional point cloud data.

5. A method characterized by, Including: an acquisition step that acquires three-dimensional point cloud data measured by a distance measuring device that measures at least a part of a photographing region including a travel path of a mobile body; and a learning step that relearns a machine learning model that has completed learning that calculates an outline of the mobile body by inputting image data in which the photographing region is photographed using current image data obtained by photographing the photographing region in a case where there is a difference between past three-dimensional point cloud data and current three-dimensional point cloud data. ​

Images