Driving assistance system, driving assistance method, and program for escort vehicles

Abstract

The present invention provides a driver assistance device, driver assistance method, and program that effectively prevent a support vehicle traveling adjacent to a target vehicle from making contact with the target vehicle. [Solution] The driving support device 100 includes: an extraction unit 132 that extracts 3D point cloud data showing the shape of the accompanying vehicle from acquired 3D point cloud data; a position calculation unit 133 that calculates the center of gravity position of the accompanying vehicle relative to the target vehicle at the time the 3D point cloud data was acquired, and the center of gravity position of the accompanying vehicle relative to the target vehicle a certain time after the time the 3D point cloud data was acquired, based on the extracted 3D point cloud data; a determination unit 134 that determines whether at least one of the calculated center of gravity positions of the accompanying vehicle is located within a collision risk area set relative to the target vehicle; and an output unit 135 that outputs a warning signal when it is determined that at least one of the current position and estimated position of the accompanying vehicle is located within a collision risk area.

Description

[Technical Field]

[0001] This invention relates to a driving assistance device, driving assistance method, and program for an escort vehicle. [Background technology]

[0002] A widely used harvesting method involves using a moving harvester to cut and launch the crops, which are then caught by a support vehicle that follows the harvester. The driver of the support vehicle visually checks the distance between the vehicle and the harvester to avoid contact. Since harvesters are expensive agricultural machines and are often shared by multiple farmers, the impact of a support vehicle colliding with a harvester and requiring repairs can be immeasurable. Therefore, a system has been developed to prevent contact between the two by measuring the distance between the harvester and the support vehicle and providing the results to the driver, as disclosed in Patent Document 1. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 5-158537 [Overview of the Initiative] [Problems that the invention aims to solve]

[0004] In the system described in Patent Document 1, since the direction of travel of the harvester is unknown, even if a sufficient distance is maintained between the vehicles, there is a risk of rapid approach and collision between the two if the vehicle is traveling towards the harvester or if the harvester's turning speed is faster than that of the accompanying vehicle. In particular, if the accompanying vehicle has front-wheel steering and the harvester has rear-wheel steering, the likelihood of collision increases if both vehicles turn simultaneously while in close proximity to each other. This problem is not limited to cases where the accompanying vehicle is traveling adjacent to the harvester, but also exists when the accompanying vehicle is traveling adjacent to other target vehicles.

[0005] The present invention is based on this background and aims to provide a driving assistance device, a driving assistance method, and a program that effectively prevent a escort vehicle traveling adjacent to a target vehicle from coming into contact with the target vehicle. [Means for solving the problem]

[0006] To achieve the above objective, the driving support device according to the first aspect of the present invention is: An acquisition unit periodically acquires 3D point cloud data showing the shape of an object measured by a range sensor installed on the target vehicle, An extraction unit extracts 3D point cloud data showing the shape of the accompanying vehicle from the 3D point cloud data acquired by the acquisition unit, A position calculation unit calculates the current position of the escort vehicle relative to the target vehicle at the time the 3D point cloud data was acquired, and the estimated position of the escort vehicle relative to the target vehicle a certain period of time after the acquisition date and time, based on the 3D point cloud data extracted by the extraction unit. A determination unit determines whether at least one of the current position and estimated position of the escort vehicle calculated by the position calculation unit is located within a contact risk area set relative to the target vehicle. An output unit that outputs a warning signal when the determination unit determines that at least one of the current position and estimated position of the escort vehicle is within the contact risk area, It is equipped with.

[0007] The current position and estimated position of the accompanying vehicle may both be the center of gravity of the accompanying vehicle.

[0008] The position calculation unit may calculate the direction vector of travel of the escort vehicle based on the three-dimensional point cloud data extracted by the extraction unit, and calculate the estimated position of the escort vehicle based on the calculated direction vector of travel and the relative speed of the escort vehicle with respect to the target vehicle.

[0009] The aforementioned direction vector may be a first singular vector or first principal component vector obtained by performing singular value decomposition or principal component analysis on the three-dimensional point cloud data extracted by the extraction unit.

[0010] The extraction unit may generate a two-dimensional image obtained by projecting the three-dimensional point cloud data acquired by the acquisition unit in the vertical direction of the target vehicle, and if the escort vehicle is present in the generated two-dimensional image, it may estimate the vertex coordinates that indicate the bounding box surrounding the escort vehicle.

[0011] The extraction unit may generate a 3D bounding box by assigning a height to the 2D bounding box, and extract the point cloud data located within the 3D bounding box from the 3D point cloud data acquired by the acquisition unit as 3D point cloud data representing the accompanying vehicle.

[0012] To achieve the above objective, the driving assistance method according to the second aspect of the present invention is: A driver assistance method performed by a driver assistance system, The process involves periodically acquiring 3D point cloud data representing the shape of an object measured by a range sensor installed on the target vehicle, and The steps include: extracting 3D point cloud data representing the shape of the support vehicle from the acquired 3D point cloud data; The steps include calculating the current position of the escort vehicle relative to the target vehicle at the time the 3D point cloud data was acquired, and the estimated position of the escort vehicle relative to the target vehicle a certain time after the time the data was acquired, based on the extracted 3D point cloud data. The steps include determining whether at least one of the calculated current position and estimated position of the escort vehicle is located within a collision risk area set relative to the target vehicle, The step of outputting a warning signal when it is determined that at least one of the current position and estimated position of the accompanying vehicle is within the contact risk area, Includes.

[0013] In order to achieve the above object, a program according to a third aspect of the present invention causes a computer to periodically acquire acquisition means for three-dimensional point cloud data indicating the shape of an object measured by a measurement range sensor installed in a target vehicle; extraction means for extracting three-dimensional point cloud data indicating the shape of an accompanying vehicle from the three-dimensional point cloud data acquired by the acquisition means; position calculation means for calculating a current position of the accompanying vehicle based on the target vehicle at the acquisition date and time of the three-dimensional point cloud data and an estimated position of the accompanying vehicle based on the target vehicle after a certain period of time from the acquisition date and time, based on the three-dimensional point cloud data extracted by the extraction means; determination means for determining whether at least one of the current position and the estimated position of the accompanying vehicle calculated by the position calculation means exists within a contact risk area set based on the target vehicle; output means for outputting a warning signal when it is determined by the determination means that at least one of the current position and the estimated position of the accompanying vehicle exists within the contact risk area; function as.

Advantages of the Invention

[0014] According to the present invention, it is possible to provide a driving support device, a driving support method, and a program that effectively prevent an accompanying vehicle that accompanies a target vehicle adjacent to the target vehicle from contacting the target vehicle.

Brief Description of the Drawings

[0015] [Figure 1] It is a plan view showing a state where a truck accompanies a traveling harvester. [Figure 2] It is a schematic diagram showing the configuration of a driving support system according to an embodiment of the present invention. ​​​​​​​​​This is a conceptual diagram of a neural network according to an embodiment of the present invention. [Figure 6] (a) is a diagram showing an example of a data table for a boundary box storage unit according to an embodiment of the present invention, and (b) is a diagram showing an example of a data table for a contact hazard area storage unit according to an embodiment of the present invention. [Figure 7] This is a plan view showing the relationship between the truck's center of gravity and the contact hazard area, which is set relative to the harvester. [Figure 8] This flowchart shows the learning process flow according to an embodiment of the present invention. [Figure 9] This flowchart shows the flow of the driver assistance process according to an embodiment of the present invention. [Figure 10] This flowchart shows the flow of the extraction process according to an embodiment of the present invention. [Figure 11] This flowchart shows the flow of the position calculation process according to an embodiment of the present invention. [Modes for carrying out the invention]

[0016] Hereinafter, the driving support device, driving support method, and program according to embodiments of the present invention will be described in detail with reference to the drawings. In each drawing, the same or equivalent parts are denoted by the same reference numerals. In the embodiment, a reference point is set on a range sensor provided on the harvester, and a moving coordinate system (Cartesian coordinate system) is used in which the front-to-back direction of the harvester is the X-axis direction, the width direction of the harvester is the Y-axis direction, and the direction perpendicular to the X and Y axes (up and down direction) is the Z-axis direction.

[0017] The driving assistance system according to this embodiment is a system that assists the driving operations of the driver operating the accompanying vehicle so that the accompanying vehicle, which is driving adjacent to the target vehicle, does not come into contact with the target vehicle. Specifically, the driving assistance system according to this embodiment calculates the current relative position of the target vehicle and the accompanying vehicle and the relative position after a certain period of time, and issues a warning to the driver if it is determined that there is a high probability that the accompanying vehicle will come into contact with the target vehicle based on the current relative position and the relative position after a certain period of time.

[0018] As shown in Figure 1, harvester A is an agricultural machine that cuts crops while driving in the field and discharges them to the outside. Truck B is a vehicle that travels alongside harvester A and receives the crops discharged from harvester A, for example, a truck with an open top. Harvester A is an example of a target vehicle that is accompanied, and truck B is an example of an accompanying vehicle that travels alongside the target vehicle.

[0019] As shown in Figure 2, the driver assistance system 1 includes a range sensor 2 installed on the harvester A that acquires 3D point cloud data of objects on the side of the harvester A, a driver assistance device 100 installed on the harvester A that transmits a warning signal when it determines, based on the measurement results from the range sensor 2, that there is a high probability that truck B will come into contact with the harvester A, and a monitoring device 200 installed on truck B that warns the driver of truck B when it receives a warning signal from the driver assistance device 100. The range sensor 2 and the driver assistance device 100 are connected to each other via a wired or wireless communication line, and the driver assistance device 100 and the monitoring device 200 are connected to each other via a wireless communication line.

[0020] The range sensor 2 is installed on the side of the harvester A, for example, on a support column extending vertically. The range sensor 2 periodically measures the distance between an object within the measurement area and a reference point, as well as the shape of the object, and generates 3D point cloud data of the object within the measurement area based on a moving coordinate system. The range sensor 2 transmits the 3D point cloud data to the driver assistance device 100 in real time each time it is acquired.

[0021] The range sensor 2 is, for example, a 3D-LiDAR (Light Detection and Ranging) system. A 3D-LiDAR system includes a light source that radially irradiates a pulsed laser onto the object to be measured, and a detector that detects the light reflected from the object. The 3D-LiDAR system determines the distance to the object from the time required from the emission of the pulsed laser to the detection of the reflected light, or from the phase difference between the emitted light and the reflected light. This distance is then used to determine the coordinate value P of a point representing the three-dimensional surface shape of the object in a moving coordinate system. j (x j ,y j ,z j This is converted to a 3D point cloud, which is a collection of many points.

[0022] The driver assistance device 100 is, for example, a general-purpose computer. The driver assistance device 100 extracts 3D point cloud data representing the shape of track B by applying image processing to the 3D point cloud data acquired from the range sensor 2, and calculates the current relative position of harvester A and track B and the relative position after a certain period of time based on the extracted 3D point cloud data. The range sensor 2 has difficulty acquiring 3D point cloud data representing the side of track B that is farther from harvester A, but by using a trained model generated by the learning process, it is possible to estimate the vertex coordinates that represent the bounding box surrounding track B, and based on this, 3D point cloud data representing the shape of track B can be extracted. Then, if it is determined that there is a high probability that track B will come into contact with harvester A based on the current relative position and the relative position after a certain period of time, a warning signal is sent to the monitoring device 200. The trained model is acquired from the learning device 3 and stored in memory in advance.

[0023] The monitoring device 200 is, for example, a smartphone or tablet device, and is supported inside the truck B so that it can be seen by the driver. The monitoring device 200 receives instructions from the user to request driving assistance from the driving assistance device 100 and transmits an operation signal to the driving assistance device 100 requesting driving assistance. The monitoring device 200 also warns the driver that truck B may come into contact with harvester A, based on a warning signal transmitted from the driving assistance device 100.

[0024] Next, the configuration of the learning device 3 according to the embodiment will be described. The learning device 3 is, for example, a general-purpose computer. The learning device 3 is a device that generates a trained model based on training data and transmits that data to the driver assistance device 100, and it is preferable that the learning device 3 has higher computing power than the driver assistance device 100. The learning device 3 comprises an operating device, a display, a communication interface, memory, and a processor. The processor comprises a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit), and executes the learning process shown in Figure 8 by executing a program stored in memory. The memory stores programs and various types of data, functions as a work memory for the processor to execute processing, and also includes a learning data storage unit as a database.

[0025] The training data storage unit stores training data used as training data for machine learning. The training data includes multiple datasets. Each dataset contains one 2D image (projected image) obtained by projecting 3D point cloud data obtained by the range sensor 2 in the Z-axis direction, as shown in Figure 4, and one set of vertex coordinates of a 2D bounding box that manually encloses track B on the projected image. The projected image is a 2D image on the XY plane, and the bounding box is a rectangular box manually set to enclose track B on the projected image. The bounding box enclosing track B is identified, for example, by the vertex coordinates (u1,v1) and (u2,v2) on the diagonal corners of the rectangle.

[0026] Furthermore, data augmentation may be performed as a preprocessing step for machine learning to increase the number of data points in the training dataset. Data augmentation is a process that increases the amount of data in the training dataset by treating the original data in the training dataset as images and transforming or combining them. For example, data augmentation can be performed by randomly flipping the sample images horizontally, changing the brightness, rotating them, or adding noise. Adding noise is most useful for improving the detection accuracy of track B. If noise is added during data augmentation, the feed ejected from harvester A can be ignored as noise during the detection process.

[0027] The learning device 3 reads the training data stored in the training data storage unit, generates a trained model using machine learning with the acquired training data, and stores it in memory. The learning device 3 generates the trained model by performing supervised learning using all or part of the multiple datasets included in the training data as training data.

[0028] More specifically, the learning device 3 uses multiple datasets included in the training data as training data to adjust the weight coefficients that represent the connection state of each layer in the neural network shown in Figure 5. The procedure is as follows: First, the pixel value I(i,j) of each pixel in the projection image in the training data is input to each neuron in the input layer, and the vertex coordinates O(i,j) of the bounding box surrounding track B output from each neuron in the output layer are compared with the vertex coordinates O'(i,j) of the bounding box surrounding track B in the training data. Then, the weight coefficients are optimized so that the difference between the vertex coordinates O(i,j) output from each neuron in the output layer and the vertex coordinates O'(i,j) of the training data is as small as possible. For example, backpropagation can be used to optimize the weight coefficients. The above describes the configuration of learning device 3.

[0029] Next, the hardware configuration of the driver assistance device 100 will be described with reference to Figure 3. The driver assistance device 100 is, for example, a general-purpose computer. The driver assistance device 100 includes a communication unit 110, a storage unit 120, and a control unit 130. The various parts of the driver assistance device 100 are interconnected via an internal bus (not shown).

[0030] The communication unit 110 is a communication interface for the driver assistance device 100 to communicate with external devices. The communication unit 110 is equipped with an antenna and input / output terminals and communicates with external devices via a communication circuit.

[0031] The memory unit 120 includes, for example, RAM (Random Access Memory), ROM (Read Only Memory), flash memory, and SSD (Solid State Drive). The SSD has excellent vibration resistance and is suitable for mounting on the harvester A. The memory unit 120 stores programs executed by the control unit 130 and various data. The memory unit 120 also temporarily stores various data and functions as a work memory for the control unit 130 to execute processing. Furthermore, the memory unit 120 includes a learned model memory unit 121, a boundary box memory unit 122, a contact risk area memory unit 123, and an operation log memory unit 124. The operation log memory unit 124 stores data related to the operation of the driver assistance system 1, such as point cloud data acquired by the range sensor 2.

[0032] The pre-trained model storage unit 121 stores pre-trained models generated by machine learning based on training data stored in the training data storage unit. The pre-trained model is a model that estimates the vertex coordinates that represent the bounding box surrounding track B in response to the input of the pixel values ​​of each pixel in the projected image, and is pre-generated based on the training dataset. As the pre-trained model, a multi-layered neural network consisting of multiple interconnected nodes is used. For this reason, the pre-trained model storage unit 121 stores optimized weight coefficients that indicate the strength of the connections between the nodes constituting the neural network. As an example of a pre-trained model, YOLO (You Only Look Once) is used.

[0033] As shown in FIG. 5, the neural network includes an input layer into which pixel values of each pixel of the projection image are input as input data, an output layer that outputs vertex coordinates of a bounding box surrounding Track B as output data, and at least one intermediate layer disposed between the input layer and the output layer. The arrows between the neurons of each layer represent the connection of parameters between the input layer and the output layer.

[0034] The input layer includes a plurality of input neurons. If the projection image is composed of m×n pixels, and the pixel values of each pixel of the projection image are I(1,1), I(1,2), …, I(m,n), then the pixel value I(i,j) is input to the neuron in the input layer corresponding to the pixel position (i,j). The pixel values input to the input neurons in the input layer are respectively output to each intermediate neuron in the intermediate layer.

[0035] Each intermediate layer includes a plurality of intermediate neurons. When each neuron in the intermediate layer receives an input value I i it calculates the product I i ×W i of the input value I i and the weight coefficient W i and calculates the total value of the products I i ×W i calculated for each neuron in the previous stage, and outputs an output value by substituting the total value of the products I i ×W i into an activation function. The activation function is a function that represents the non-linear relationship between input and output in a certain neuron, and for example, it is a sigmoid function, a max function, or a Gaussian function.

[0036] The output layer includes a plurality of output neurons that estimate vertex coordinates indicating a bounding box surrounding Track B. When each neuron in the output layer receives an input value I i from each neuron in the last intermediate layer, similar to each neuron in the intermediate layer, it calculates the product I i ×W i of the input value I i and the weight coefficient W iThe product I calculated for each neuron in the preceding stage is then calculated. i ×W i Calculate the sum of the products I i ×W i By substituting the sum of the values ​​into the activation function, the vertex coordinates O(1,1), O(1,2) of the bounding box surrounding track B are output. The output layer may also output the vertex coordinates O(1,1), O(1,2), ..., O(m',n') of multiple bounding boxes.

[0037] As shown in Figure 6(a), the bounding box storage unit 122 stores the vertex coordinates (u1,v1) and (u2,v2) of the bounding box surrounding track B, generated using the trained model, in association with the acquisition date and time of the original 3D point cloud data. The acquisition date and time of the 3D point cloud data is, for example, the date and time when the range sensor 2 acquired the 3D point cloud data. The vertex coordinates of the bounding box are, for example, the coordinates of the top-left and bottom-right vertices of the bounding box.

[0038] As shown in Figure 6(b), the contact hazard area storage unit 123 stores the pixel positions indicating the contact hazard area, which is set based on the range sensor 2. As shown in Figure 7, the contact hazard area is set on the side of the harvester A and is a rectangular area including the front and rear ends of the harvester A. Therefore, when the harvester A moves, the contact hazard area moves together with the harvester A and the moving coordinate system. The position and size of the contact hazard area are determined, for example, by the vertex coordinates (u3,v3) and (u4,v4) on the diagonal corners of the rectangle. The vertex coordinates of the contact hazard area are, for example, the coordinates of the top-left and bottom-right vertices of the contact hazard area.

[0039] Returning to Figure 3, the control unit 130 includes a processor and controls each part of the driving assistance device 100. The processor includes a CPU and a GPU. The control unit 130 executes the driving assistance processing in Figure 9, the extraction processing in Figure 10, and the position calculation processing in Figure 11 by executing the program stored in the memory unit 120. Functionally, the control unit 130 includes an acquisition unit 131, an extraction unit 132, a position calculation unit 133, a determination unit 134, and an output unit 135.

[0040] The acquisition unit 131 periodically acquires 3D point cloud data using the range sensor 2 and stores it in the storage unit 120, associating it with the acquisition date and time. The acquisition of 3D point cloud data by the acquisition unit 131 also includes reading the 3D point cloud data from the main memory of the storage unit 120 to the work memory.

[0041] The extraction unit 132 extracts 3D point cloud data representing the shape of track B from the 3D point cloud data acquired by the acquisition unit 131. Specifically, the processing procedure involves first projecting the 3D point cloud data acquired by the acquisition unit 131 in the Z-axis direction to generate a projected image on the XY plane.

[0042] Next, using the trained model stored in the trained model memory unit 121, the pixel values ​​of each pixel in the projected image are input as input data to each neuron in the input layer of the trained model. Then, the vertex coordinates of the bounding box surrounding track B are output as output data from each neuron in the output layer. If track B does not exist in the projected image, the vertex coordinates of the bounding box surrounding track B are not output.

[0043] Next, a 3D bounding box is generated by assigning a height in the Z-axis direction to a bounding box set on the XY plane, and the 3D point cloud data existing within the 3D bounding box is extracted as 3D point cloud data representing the shape of track B. The 3D bounding box is a cube with the 2D bounding box as its cross-section, and is identified, for example, by the coordinate values ​​of four vertices obtained by assigning two coordinate values ​​z1 and z2 that are separated in the Z-axis direction from each other to the coordinate values ​​(u1,v1,z1), (u1,v1,z2), (u2,v2,z1), and (u2,v2,z2) of opposite vertices (u1,v1) and (u2,v2) of the 2D bounding box.

[0044] The position calculation unit 133 calculates the centroid position G of track B relative to harvester A at the time point cloud data acquisition shown in Figure 7, and the centroid position G' of track B relative to harvester A after a certain period of time has elapsed from the time point cloud data acquisition, based on the 3D point cloud data showing the shape of track B. The centroid position G is an example of the current position of track B relative to harvester A, and the centroid position G' is an example of the estimated position of track B relative to harvester A. Both the centroid positions G and G' of track B are determined by coordinate values ​​(x,y) relative to the moving coordinate system set in the range sensor 2, making them easy to calculate even within the position of track B and suitable for real-time processing. The certain period of time can be set to, for example, a few seconds.

[0045] The centroid position G of track B at the time of point cloud data acquisition can be calculated by averaging the coordinate values ​​(x,y) of each point cloud data representing the shape of track B. Furthermore, the centroid position G' of track B after a certain period of time has elapsed since point cloud data acquisition can be calculated using the following procedure.

[0046] First, based on the 3D point cloud data representing the shape of track B, the angle α of the direction vector V, which indicates the direction of travel of track B relative to the X-axis of the moving coordinate system set on harvester A, is calculated. To calculate the angle α, for example, singular value decomposition or principal component analysis can be performed on the 3D point cloud data representing the shape of track B, and the first singular vector or first principal component vector can be calculated based on the moving coordinate system. The first singular vector and the first principal component vector are vectors that indicate the direction in which the variance of the projection of the 3D point cloud data in the Z-axis direction is maximized, and both are examples of the direction vector V of track B.

[0047] Next, the center of gravity position G' of track B is calculated based on the calculated angle α and the pre-set estimated relative speed of track B. The estimated relative speed of track B is an example of the relative speed of track B relative to harvester A. For the estimated relative speed of track B, it is preferable to set a constant value considering the relative speeds of harvester A and track B in past harvesting operations and store it in the memory unit 120 beforehand.

[0048] Returning to Figure 3, the determination unit 134 determines whether at least one of the center of gravity positions G and G' of truck B is within the contact hazard area. The determination unit 134 generates a warning signal if at least one of the center of gravity positions G and G' of truck B is within the contact hazard area. In the example in Figure 7, the center of gravity position G is outside the contact hazard area, and the center of gravity position G' is within the contact hazard area.

[0049] The output unit 135 outputs output data to the outside, which includes data relating to the projected image generated by the extraction unit 132, data relating to the centroid positions G and G' calculated by the position calculation unit 133, and a warning signal generated by the determination unit 134. The output unit 135 controls, for example, the communication unit 110 to transmit the output data to the monitoring device 200 in real time. The above describes the configuration of the driver assistance device 100.

[0050] Next, we will return to Figure 2 and explain the hardware configuration of the monitoring device 200. The monitoring device 200 is, for example, a general-purpose computer. The monitoring device 200 includes an operating device, a display, a communication interface, memory, and a processor. The operating device and the display are, for example, a touch panel, and the input / output interface includes, for example, an antenna and input / output terminals.

[0051] The monitoring device 200 receives a request from the user for driving assistance from the driving assistance device 100 by executing a program stored in memory, and sends an operation signal to the driving assistance device 100 requesting data transmission. Furthermore, when the monitoring device 200 receives data from the driving assistance device 100 regarding the projected image obtained by projecting the 3D point cloud data obtained by the range sensor 2 in the Z-axis direction, and data regarding the center of gravity positions G and G' of truck B, it displays this data on the display. The display should ideally overlay the center of gravity positions G and G' of truck B onto the projected image obtained by projecting in the Z-axis direction. In addition, when the monitoring device 200 receives a warning signal from the driving assistance device 100, it displays a warning screen on the display to warn the driver of truck B. The above describes the configuration of the monitoring device 200.

[0052] (Learning process) Next, with reference to the flowchart in Figure 8, the flow of the learning process performed by the learning device 3 according to the embodiment will be explained. The learning process is the process of generating a trained model based on the training data. The learning process starts when the user has stored the training data in the training data storage unit and has given the instruction to start the learning process.

[0053] First, the learning device 3 selects unselected learning data stored in the learning data storage unit (step S11).

[0054] Next, the learning device 3 adjusts the weight coefficients between each neuron in the neural network shown in Figure 5 based on the training data selected in step S11 (step S12). Specifically, by inputting the pixel values ​​of each pixel in the projected image in the selected training data to each neuron in the input layer, it outputs estimated values ​​of the vertex coordinates of the bounding box surrounding track B, which are output from each neuron in the output layer. Next, the Mean Squared Error (MSE) is calculated based on these estimated vertex coordinates and the set values ​​of the vertex coordinates of the manually drawn bounding box surrounding track B in the training data, and the weight coefficients between each neuron are adjusted so that the MSE is below a threshold.

[0055] Next, the learning device 3 determines whether learning using the training data has been completed (step S13). Specifically, it determines whether there is any untrained training data remaining. If it is determined that learning using the training data has been completed (step S13; Yes), the process moves to step S14. On the other hand, if it is determined that learning using the training data has not been completed (step S13; No), the process returns to step S11.

[0056] If the answer to step S13 is Yes, the learning device 3 stores the weight coefficients between each neuron, which were finally obtained in step S12, into memory (step S14), and terminates the process. The weight coefficients between each neuron stored in the memory on the learning device 3 in step S14 can be transmitted from the learning device 3 to the driving support device 100 during the preparation stage for the driving support process shown in Figure 9. When the driving support device 100 receives the weight coefficients between each neuron from the learning device 3, it is preferable to store them in the learned model storage unit 121. The above is the flow of the learning process.

[0057] (Driving support processing) Next, referring to the flowchart in Figure 9, the flow of the driving support process performed by the control unit 130 of the driving support device 100 according to the embodiment will be explained. The driving support process is the process of generating a warning signal when it is determined that there is a high probability that truck B will come into contact with harvester A based on the measurement results from the range sensor 2. The driving support process starts when the driver of truck B requests driving support from the driving support device 100 by operating the control device.

[0058] First, the acquisition unit 131 starts the periodic acquisition of 3D point cloud data from the measurement sensor 2 (step S21). Each time the acquisition unit 131 acquires 3D point cloud data, it stores it in the operation log storage unit 124, associating it with the date and time the data was acquired.

[0059] Next, the extraction unit 132 performs an extraction process to extract 3D point cloud data showing the shape of track B from the latest 3D point cloud data acquired by the acquisition unit 131 (step S22). The flow of the extraction process will be explained below with reference to the flowchart in Figure 10.

[0060] (Extraction process) First, the extraction unit 132 reads the latest 3D point cloud data acquired by the acquisition unit 131, and projects the read 3D point cloud data in the Z-axis direction to generate a 2D projected image on the XY plane (step S31).

[0061] Next, the extraction unit 132 uses the trained model stored in the trained model storage unit 121 to estimate the vertex coordinates of the bounding box surrounding track B based on the projected image obtained by the processing in step S31, and stores them in the bounding box storage unit 122 in association with the point cloud data acquisition date and time (step S32). Specifically, when the pixel values ​​of each pixel of the projected image obtained by the processing in step S31 are input to each neuron in the input layer of the neural network shown in Figure 5, calculation processing is performed sequentially to the hidden layer and output layer, and the vertex coordinates of the bounding box surrounding track B are output from each neuron in the output layer. If track B based on the projected image does not exist, the vertex coordinates of the bounding box surrounding track B cannot be obtained.

[0062] Next, the extraction unit 132 determines whether track B was detected in the process of step S32 (step S33). Whether track B was detected can be determined by whether the vertex coordinates of the bounding box surrounding track B were obtained. If it is determined that track B was detected (step S33; Yes), the process proceeds to step S34. On the other hand, if it is determined that track B was not detected (step S33; No), the extraction of 3D point cloud data is skipped and the process returns.

[0063] If the answer to step S33 is Yes, the extraction unit 132 generates a 3D bounding box by assigning a height in the Z-axis direction to the 2D bounding box on the XY plane, extracts the 3D point cloud data present within the 3D bounding box (step S34), and returns the process. This provides 3D point cloud data representing the shape of track B. The above is the flow of the extraction process.

[0064] Returning to Figure 9, the position calculation unit 133 performs a position calculation process to calculate the centroid positions G and G' of track B relative to harvester A, based on the 3D point cloud data showing the shape of track B obtained in step S22 (step S23). The flow of the position calculation process will be explained below with reference to the flowchart in Figure 11.

[0065] (Position calculation process) First, the position calculation unit 133 calculates the center of gravity G of track B relative to harvester A at the time of point cloud data acquisition (step S41). To calculate the center of gravity G, it is sufficient to calculate the average of the coordinate values ​​(x,y) of each point cloud data representing the shape of track B.

[0066] Next, the position calculation unit 133 calculates the angle α of the vector V, which indicates the direction of travel of track B with respect to the X-axis of the moving coordinate system set on the harvester A, based on the three-dimensional point cloud data showing the shape of track B (step S42). To calculate the angle α, for example, singular value decomposition or principal component analysis can be performed on the three-dimensional point cloud data showing the shape of track B to calculate the first singular vector or first principal component vector.

[0067] Next, based on the angle α calculated in step S42 and the pre-set estimated relative velocity of track B, the center of gravity position G' of track B relative to harvester A after a certain period of time has elapsed since the point cloud data acquisition is calculated (step S43), and the process is returned. In this process, the distance traveled in a certain period of time is calculated based on the pre-set estimated relative velocity, and the coordinate values ​​to which the center of gravity position G moves after a certain period of time are calculated based on the calculated distance and angle α. The above is the flow of the position calculation process.

[0068] Returning to Figure 9, the determination unit 134 determines whether at least one of the center of gravity positions G and G' of track B is within the contact hazard area (step S24). If it is determined that at least one of the center of gravity positions G and G' of track B is within the contact hazard area (step S24; Yes), the process moves to step S25. On the other hand, if at least one of the center of gravity positions G and G' of track B is not within the contact hazard area (step S24; No), the process moves to step S27.

[0069] If the result of step S24 is Yes, the determination unit 134 generates a warning signal to be sent to the monitoring device 200 (step S25). Next, the output unit 135 causes the monitoring device 200 to send in real time the data related to the projected image generated in step S22, the data related to the centroid position G and G' estimated in step S23, and the warning signal generated in step S25 (step S26), and then returns the process to step S22.

[0070] On the other hand, if the result in step S24 is No, the output unit 135 causes the monitoring device 200 to transmit in real time the data relating to the projected image generated in step S22 and the data relating to the centroid positions G and G' estimated in step S23 (step S27), and returns the process to step S22. The above is the flow of the driver assistance process.

[0071] When the monitoring device 200 receives data related to the projected image and data related to the center of gravity positions G and G' from the driving support device 100, it overlays the center of gravity positions G and G' onto the projected image and displays them on the display. Furthermore, when the monitoring device 200 receives a warning signal from the driving support device 100, it displays a warning screen on the display informing the driver of the high risk of contact with the harvester A. The driver refers to the various images displayed on the display to assist in driving operations.

[0072] As described above, the driving support device 100 according to the embodiment includes: an extraction unit 132 that extracts 3D point cloud data showing the shape of truck B from 3D point cloud data acquired by the acquisition unit 131; a position calculation unit 133 that calculates the center of gravity position G of truck B relative to harvester A at the time the 3D point cloud data was acquired, and the center of gravity position G' of truck B relative to harvester A at a certain time after the time the 3D point cloud data was acquired, based on the 3D point cloud data extracted by the extraction unit 132; a determination unit 134 that determines whether at least one of the center of gravity positions G and G' of truck B calculated by the position calculation unit 133 is located within a contact risk area set relative to harvester A; and an output unit 135 that outputs a warning signal when the determination unit 134 determines that at least one of the center of gravity positions G and G' of truck B is located within a contact risk area. Therefore, it is possible to effectively prevent truck B, which is running adjacent to and accompanying harvester A, from coming into contact with harvester A.

[0073] The present invention is not limited to the embodiments described above, and the following modifications are also possible.

[0074] (modified version) In the above embodiment, a 3D-LiDAR was used as the range sensor 2, but the present invention is not limited to this. For example, a stereo camera, a TOF (Time of Flight) camera, or an area camera may be used as the range sensor 2. Alternatively, a 2D-LiDAR may be used as the range sensor 2, and 3D point cloud data may be generated by scanning the 2D-LiDAR.

[0075] In the above embodiment, the learning process was performed using learning data stored in the learning data storage unit in advance, but the present invention is not limited thereto. Point cloud data acquired by the range sensor 2 and stored as an operation log in the operation log storage unit 124 may also be added to the learning process.

[0076] In the above embodiment, gradient descent was used as the update algorithm to reduce the evaluation function, but the present invention is not limited to this. For example, stochastic gradient descent or momentum methods may be used.

[0077] In the above embodiment, the weight coefficients were optimized by determining whether the evaluation function, MSE, was below a threshold; however, the present invention is not limited to this. Functions other than MSE may be used as the evaluation function. Furthermore, the weight coefficients may be determined to be optimized when the number of training iterations reaches an upper limit.

[0078] In the above embodiment, the optimized weight coefficients obtained by the learning process using the training data were stored in the trained model storage unit 121, but the present invention is not limited thereto. For example, after performing the learning process using a portion of the training data, the validity of the optimized weight coefficients may be evaluated using the remaining training data, and if the weight coefficients are evaluated as valid weight coefficients, they may be stored in the trained model storage unit 121.

[0079] In the above embodiment, the learning device 3 performed the learning process shown in Figure 8, but the present invention is not limited thereto. For example, the storage unit 120 of the driving support device 100 may be equipped with a learning data storage unit, and the control unit 130 may be configured to functionally include a learning unit. The learning unit of the driving support device 100 can read the learning data stored in the learning data storage unit and store the learned model obtained by performing the learning process shown in Figure 8 in the learned model storage unit 121.

[0080] In the above embodiment, the trained model was configured to estimate the vertex coordinates indicating the bounding box surrounding track B, for example, using YOLO (You Only Look Once). However, the present invention is not limited to this. For example, the trained model may be configured to use Semantic Segmentation and output a label image in which track B is filled in or a bounding box surrounding it is drawn.

[0081] In the above embodiment, a trained model was generated by performing machine learning using a neural network, but the present invention is not limited to this. The trained model defines the relationship between the projected image and the vertex coordinates of the bounding box surrounding track B, and any model that outputs the vertex coordinates of the bounding box surrounding track B from the output layer when the projected image is input to the input layer may be a model constructed using a method other than machine learning.

[0082] In the above embodiment, the centroid positions G and G' of track B were calculated based on 3D point cloud data showing the shape of track B, but the present invention is not limited to this. Other positions of track B other than the centroid positions G and G' may also be calculated. For example, for each vertex of the 2D bounding box, the position at the time of acquisition of the 3D point cloud data and the position a certain time after said acquisition may be calculated. When calculating other positions of track B, it is advisable to set contact risk areas corresponding to those positions.

[0083] In the above embodiment, the centroid positions G and G' were superimposed and drawn on the projected image, but the present invention is not limited to this. For example, a figure representing the shape of track B, such as a rectangle, may be superimposed on the projected image. Alternatively, instead of the projected image, a two-dimensional image obtained by photographing track B from above with a camera installed on harvester A may be used.

[0084] In the above embodiment, a preset estimated relative speed was used as the relative speed of track B relative to harvester A, but the present invention is not limited thereto. The relative speed of track B relative to harvester A may be calculated based on 3D point cloud data. Specifically, the travel distance may be calculated based on the latest center of gravity position G of track B and the center of gravity position G one cycle ago, and the relative speed of track B relative to harvester A may be calculated based on the calculated travel distance and period (time difference).

[0085] In the above embodiment, the boundary box and the contact hazard area were identified by the coordinates of two opposing vertices of a rectangle, but the present invention is not limited to this. For example, the boundary box and the contact hazard area may be identified by the coordinates of the center of the rectangle and the width and height of the rectangle.

[0086] In the above embodiment, the contact hazard area was rectangular, but the present invention is not limited to this. For example, the contact hazard area may be a polygon other than a rectangle, or a shape that combines a circle and a rectangle.

[0087] In the above embodiment, data relating to the projected image and data relating to the centroid positions G and G' were transmitted from the driving support device 100 to the monitoring device 200, but the present invention is not limited thereto. In order to reduce data communication charges, for example, the transmission of data relating to the projected image and data relating to the centroid positions G and G' may be omitted.

[0088] In the above embodiment, the monitoring device 200 displayed a warning screen on its display when it received a warning signal from the driver assistance device 100, but the present invention is not limited to this. For example, the monitoring device 200 may emit a warning sound or message from its speaker. Alternatively, the monitoring device 200 may be a wearable device that warns the driver by generating at least one of sound and vibration.

[0089] In the above embodiment, the driver assistance device 100 did not have an operation unit and a display unit, but the present invention is not limited thereto. For example, the driver assistance device 100 may have an operation unit and a display unit. Various screens may be displayed on the display unit, and the program stored in the storage unit 120 of the driver assistance device 100 may be debugged by modifying the program code using the operation unit.

[0090] In the above embodiment, various data were stored in the storage unit 120 or memory of the learning device 3 and the driving support device 100, but the present invention is not limited thereto. For example, all or part of the various data may be stored in an external computer or data logger via a communication network.

[0091] In the above embodiment, the learning device 3, the driving support device 100, and the monitoring device 200 each operated based on programs stored in a storage unit or memory, but the present invention is not limited to this. For example, a functional configuration realized by a program may be realized by hardware.

[0092] In the above embodiment, the learning device 3 and the driving support device 100 were general-purpose computers, but the present invention is not limited thereto. For example, the learning device 3 and the driving support device 100 may be implemented using computers located on the cloud.

[0093] In the above embodiment, the processing performed by the learning device 3, the driving support device 100, and the monitoring device 200 was realized by the device having the above-described physical configuration executing a program stored in a storage unit or memory. However, the present invention may be realized as a program, or as a storage medium on which that program is recorded.

[0094] Alternatively, a device that performs the above-mentioned processing operations may be configured by distributing a program for executing the above-mentioned processing operations on a computer-readable non-temporary recording medium such as a flexible disk, CD-ROM (Compact Disk Read-Only Memory), DVD (Digital Versatile Disk), or MO (Magneto-Optical Disk), and then installing that program on a computer.

[0095] In the above embodiment, the target vehicle was a harvester A and the accompanying vehicle was a truck B, but the present invention is not limited to this. For example, the target vehicle may be a rotary snowplow and the accompanying vehicle may be a truck.

[0096] The embodiments described above are illustrative, and the present invention is not limited thereto. Various embodiments are possible without departing from the spirit of the invention as described in the claims. The components described in the embodiments and modifications can be freely combined. Furthermore, inventions equivalent to the invention described in the claims are also included in the present invention. [Explanation of symbols]

[0097] 1. Driver assistance system 2. Range Sensor 3. Learning device 100 Driving support devices 110 Communications Department 120 Storage section 121 Pre-trained model memory 122 Boundary Box Memory Unit 123 Contact Hazard Area Memory Unit 124 Operation Log Storage Unit 130 Control Unit 131 Acquisition Department 132 Extraction part 133 Position calculation section 134 Judgment section 135 Output section 200 Monitoring equipment A Harvester Track B

Claims

1. An acquisition unit periodically acquires 3D point cloud data showing the shape of an object measured by a range sensor installed on the target vehicle, An extraction unit extracts three-dimensional point cloud data showing the shape of the accompanying vehicle from the three-dimensional point cloud data acquired by the acquisition unit, A position calculation unit calculates the current position of the escort vehicle relative to the target vehicle at the time the 3D point cloud data was acquired, and the estimated position of the escort vehicle relative to the target vehicle a certain period of time after the acquisition date and time, based on the 3D point cloud data extracted by the extraction unit. A determination unit determines whether at least one of the current position and estimated position of the escort vehicle calculated by the position calculation unit is located within a contact risk area set relative to the target vehicle. An output unit that outputs a warning signal when the determination unit determines that at least one of the current position and estimated position of the escort vehicle is within the contact risk area, A driver assistance system equipped with the following features.

2. The current position and estimated position of the aforementioned escort vehicle are both the center of gravity of the aforementioned escort vehicle. The driving support device according to claim 1.

3. The position calculation unit calculates the direction vector of travel of the escort vehicle based on the three-dimensional point cloud data extracted by the extraction unit, and calculates the estimated position of the escort vehicle based on the calculated direction vector of travel and the relative speed of the escort vehicle with respect to the target vehicle. The driving support device according to claim 1 or 2.

4. The aforementioned direction vector is a first singular vector or first principal component vector obtained by performing singular value decomposition or principal component analysis on the three-dimensional point cloud data extracted by the extraction unit. The driving support device according to claim 3.

5. The extraction unit generates a two-dimensional image obtained by projecting the three-dimensional point cloud data acquired by the acquisition unit in the vertical direction of the target vehicle, and estimates the vertex coordinates indicating the bounding box surrounding the escort vehicle if the escort vehicle is present in the generated two-dimensional image. The driving support device according to claim 1 or 2.

6. The extraction unit generates a three-dimensional bounding box by assigning a height to the two-dimensional bounding box, and extracts the point cloud data located within the three-dimensional bounding box from the three-dimensional point cloud data acquired by the acquisition unit as three-dimensional point cloud data representing the accompanying vehicle. The driving support device according to claim 5.

7. A driver assistance method performed by a driver assistance system, The process involves periodically acquiring 3D point cloud data representing the shape of an object measured by a range sensor installed on the target vehicle, and The steps include: extracting 3D point cloud data representing the shape of the support vehicle from the acquired 3D point cloud data; A step of calculating the current position of the escort vehicle relative to the target vehicle at the time the 3D point cloud data was acquired, and the estimated position of the escort vehicle relative to the target vehicle a certain period of time after the acquisition date and time, based on the extracted 3D point cloud data. The steps include determining whether at least one of the calculated current position and estimated position of the escort vehicle is located within a collision risk area set relative to the target vehicle, The step of outputting a warning signal when it is determined that at least one of the current position and estimated position of the accompanying vehicle is within the contact risk area, Driving assistance methods including

8. Computers, Acquisition means for periodically acquiring 3D point cloud data showing the shape of an object measured by a range sensor installed on the target vehicle, Extraction means for extracting three-dimensional point cloud data showing the shape of the accompanying vehicle from the three-dimensional point cloud data acquired by the acquisition means, A position calculation means that calculates the current position of the escort vehicle relative to the target vehicle at the time the 3D point cloud data was acquired, and the estimated position of the escort vehicle relative to the target vehicle a certain period of time after the time the 3D point cloud data was acquired, based on the 3D point cloud data extracted by the extraction means. A determination means for determining whether at least one of the current position and estimated position of the escort vehicle calculated by the position calculation means is located within a collision risk area set relative to the target vehicle. An output means that outputs a warning signal when the determination means determines that at least one of the current position and estimated position of the accompanying vehicle is within the contact risk area. A program designed to function as such.

Images