Self-driving vehicle navigation device and method

Abstract

The present disclosure provides a self-driving vehicle navigation device and method thereof, the device comprising a color depth camera, at least one sensor and a processor. The color depth camera captures depth image data and color image data of each path when the self-driving vehicle is operated to move along a target route including at least one path. The sensors (including a six-axis sensor and a rotary encoder) record the acceleration, moving speed, direction, steering angle and moving distance of the self-driving vehicle on each path. The processor generates training data based on the depth image data, the color image data, the accelerations, the moving speeds, the directions, the steering angles and the moving distances of the paths, and inputs the training data into a machine learning model for deep learning to generate training results. Therefore, automatic navigation of the self-driving vehicle can be achieved without using positioning technology, which can reduce the cost of automatic navigation technology.

Description

Technical Field

[0001] The present disclosure relates to a navigation device, and in particular to a navigation device and method for a self-propelled vehicle. Background Art

[0002] Due to the advancement of artificial intelligence, the application of automatic navigation technology is becoming more and more extensive. Existing automatic navigation technology needs to be combined with different positioning technologies according to different application environments or accuracy requirements; however, these positioning technologies have certain deployment difficulties and complexities, which will greatly increase the cost of automatic navigation technology.

[0003] Existing machine learning technology can imitate human operation modes through image recognition and several sensors. However, if the existing machine learning technology is applied to the automatic navigation technology of self-driving vehicles (AGVs), the learning results may not converge due to factors such as too much image information, too much noise background, and too high image similarity. Therefore, the existing machine learning technology cannot be applied to the automatic navigation technology of self-driving vehicles. Summary of the invention

[0004] The purpose of the present disclosure is to provide a self-driving vehicle navigation device and method thereof, which can realize automatic navigation of the self-driving vehicle without using positioning technology.

[0005] Another purpose of the present invention is to utilize machine learning, and in combination with deep image learning environment contour logic, apply dimensionality reduction methods to process depth image data, so as to significantly reduce the amount of information in the depth image data, produce an environment contour upper view, and perform deep learning based on the reduced dimensionality depth image data through a machine learning model.

[0006] According to one embodiment of the present disclosure, the present disclosure proposes a self-driving vehicle navigation device, which includes a color depth camera, at least one sensor and a processor. The color depth camera captures depth image data and color image data of each path when the self-driving vehicle is operated to move along a target route including at least one path. The sensors include a six-axis sensor and a rotary encoder; the sensors record the acceleration, moving speed, direction, steering angle and moving distance of the self-driving vehicle on each path. The processor generates training data based on the depth image data, the color image data, the accelerations, the moving speeds, the directions, the steering angles and the moving distances of the paths, and inputs the training data into a machine learning model for deep learning to generate training results.

[0007] According to another embodiment of the present disclosure, the present disclosure proposes a self-driving vehicle navigation method, which includes operating the self-driving vehicle to move along a target route including at least one path; capturing depth image data and color image data of each path by a color depth camera; recording the acceleration, moving speed, direction, steering angle and moving distance of the self-driving vehicle on each path by at least one sensor, including a six-axis sensor and a rotary encoder; generating training data according to the depth image data, the color image data, the accelerations, the moving speeds, the directions, the steering angles and the moving distances of the paths by a processor; and inputting the training data into a machine learning model by the processor for deep learning to generate training results.

[0008] As described above, the self-driving vehicle navigation device and method disclosed herein may have one or more of the following advantages:

[0009] (1) The self-driving vehicle navigation device disclosed in the present invention can generate training results through a machine learning model to imitate human operation modes without using positioning technology to achieve automatic navigation of the self-driving vehicle, thereby significantly reducing the cost of automatic navigation technology.

[0010] (2) The self-driving vehicle navigation device disclosed in the present invention uses a dimensionality reduction method to process depth image data, thereby significantly reducing the amount of information in the depth image data, and performs deep learning based on the depth image data after dimensionality reduction through a machine learning model, thereby being able to effectively perform deep learning and improve the efficiency of deep learning.

[0011] (3) The self-driving vehicle navigation device disclosed in the present invention performs deep learning based on depth image data and color image data simultaneously through a machine learning model. The environmental profile and spatial logic concept of the self-driving vehicle navigation device can be enhanced through the depth image data after dimensionality reduction. Therefore, the self-driving vehicle navigation device no longer uses image similarity learning alone, but compares the environmental profile, spatial logic concept and image similarity to determine the acceleration, moving speed, direction, steering angle, moving distance and stop of the self-driving vehicle. The above mechanism can effectively solve the problem of over-learning of image data or the problem that the amount of image data is too complex to converge, and can effectively improve the navigation recognition effect.

[0012] (4) The self-driving car navigation device disclosed in the present invention integrates a variety of sensors, including a six-axis sensor, a rotary encoder, a compass, and an obstacle avoidance sensor, so that it can more accurately simulate human senses and make the automatic navigation of the self-driving car closer to the human operation mode. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a schematic diagram of a self-propelled vehicle navigation device provided on a self-propelled vehicle according to a first embodiment of the present disclosure.

[0014] Figure 2FIG. 4 is a block diagram of a self-driving vehicle navigation device according to a first embodiment of the present disclosure.

[0015] Figure 3 FIG. 4 is a schematic diagram of a training path of a self-driving vehicle navigation device according to the first embodiment of the present disclosure.

[0016] Figure 4 FIG. 1 is a schematic diagram of a machine learning model of a self-driving vehicle navigation device according to a first embodiment of the present disclosure.

[0017] Figures 5A to 5E Schematic diagram of a dimensionality reduction method for depth image data of a self-driving vehicle navigation device according to the first embodiment of the present disclosure.

[0018] Figures 6A to 6C Schematic diagram of an automatic navigation mode using existing color vision recognition learning technology.

[0019] Figures 7A to 7D FIG. 1 is a schematic diagram of an automatic navigation mode of a self-driving vehicle navigation device according to a first embodiment of the present disclosure.

[0020] Figure 8 The figure is a flowchart of the self-driving vehicle navigation method according to the first embodiment of the present disclosure.

[0021] Fig. 9 FIG. 4 is a block diagram of a self-driving vehicle navigation device according to a second embodiment of the present disclosure.

[0022] Description of Reference Numerals

[0023] 1, 2 Self-driving car navigation device

[0024] 11,21 Color Depth Camera

[0025] 12, 22 Controller

[0026] 13,23 Processor

[0027] 14, 24 six-axis sensor

[0028] 15, 25 Rotary encoder

[0029] 26 Obstacle Avoidance Sensor

[0030] V Self-propelled car

[0031] R Remote Control

[0032] M Machine Learning Model

[0033] m1 convolutional layer

[0034] m2 Dropout layer

[0035] m3 flat layer

[0036] m4 fully connected layer

[0037] d1 Depth image data

[0038] d2 color image data

[0039] d3 acceleration

[0040] d4 Movement speed

[0041] d5 Direction

[0042] d6 Movement distance

[0043] d7 Steering angle

[0044] C Screen

[0045] A The center line of the screen

[0046] L The length of the screen

[0047] W The width of the screen

[0048] O origin

[0049] f Virtual distance

[0050] N Virtual position of the lens

[0051] K The actual position of the object

[0052] θ is the viewing angle of the color depth camera

[0053] θx Angle

[0054] The depth value of the line segment is dp pixels

[0055] (x, y) pixel coordinates

[0056] WX The real coordinate of the pixel on the X axis

[0057] TR Target Route

[0058] P1~P7 Path

[0059] Stops S1~S7

[0060] Lw Left wall

[0061] Rw Right wall

[0062] Gs color block

[0063] S81~S87 Steps DETAILED DESCRIPTION

[0064] The following will refer to the relevant drawings to illustrate the embodiments of the self-driving vehicle navigation device and method thereof according to the present disclosure. For the sake of clarity and convenience of the illustration, the components in the drawings may be exaggerated or reduced in size and proportion. In the following description and / or patent application, when it is mentioned that an element is "connected" or "coupled" to another element, it may be directly connected or coupled to the other element or there may be an intervening element; and when it is mentioned that an element is "directly connected" or "directly coupled" to another element, there is no intervening element, and other words used to describe the relationship between elements or layers should be interpreted in the same way. For ease of understanding, the same elements in the following embodiments are illustrated with the same symbols.

[0065] See also Figure 1 , Figure 2 and Figure 3 , which is a schematic diagram of a self-driving vehicle navigation device provided on a self-driving vehicle, a block diagram of the self-driving vehicle navigation device, and a schematic diagram of a training path of the self-driving vehicle navigation device according to the first embodiment of the present disclosure. Figure 1 and Figure 2 As shown, a color depth (RGB-D) camera 11 may be disposed on a self-driving vehicle V; the color depth camera 11 may be one of the components of a self-driving vehicle navigation device 1, and the self-driving vehicle navigation device 1 may perform automatic navigation for the self-driving vehicle V after a training procedure.

[0066] like Figure 2 As shown, the self-driving vehicle navigation device 1 includes a color depth camera 11, a controller 12, a processor 13 and at least one sensor, wherein the sensors include a six-axis sensor 14 and a rotary encoder 15. In one embodiment, the processor 13 may be a central processing unit, an application specific integrated circuit chip or other similar components.

[0067] The color depth camera 11 is connected to the processor 13 and is used to capture depth image data and color image data and provide them to the processor 13 .

[0068] The six-axis sensor 14 is connected to the processor 13. The six-axis sensor 14 is used to sense the acceleration, moving speed, direction and steering angle of the self-propelled vehicle V when it moves and provide them to the processor 13.

[0069] The rotary encoder 15 is connected to the processor 13. The rotary encoder 15 is used to sense the number of steps of the self-driving vehicle V, and convert the number of steps of the self-driving vehicle V into a moving distance and provide it to the processor 13. In addition, the self-driving vehicle navigation device 1 may also include an obstacle avoidance sensor (such as Figure 8 Obstacle avoidance sensor 26 shown).

[0070] The processor 13 receives the depth image data and color image data captured by the color depth camera 11, the sensing data of the six-axis sensor 14 and the sensing data of the rotary encoder 15, and inputs the above data into the machine learning model for deep learning to generate a training result, and inputs the training result into the processor 13. The processor 13 then generates a signal and transmits the signal to the controller 12, so that the controller 12 generates a control signal to perform navigation work.

[0071] When the self-driving vehicle V enters the automatic navigation mode, the controller 12 performs automatic navigation according to the real-time depth image data and real-time color image data captured by the color depth camera 11 and the training result.

[0072] like Figure 3 As shown, the user can use a remote control device R, such as a mobile device, a mobile phone, a tablet, etc., to operate the self-driving car V to move on a target route TR for training. The target route TR includes at least one path P1-P7 and at least one stop S1-S7; wherein the paths P1-P7 are not limited to straight paths, but can also be curved paths or paths with more than one turning area. The user can use the remote control R to operate the self-driving car V to move from a starting point along the path P1 to the path P7, and then return to the starting point. During the movement of the self-driving car V, the color depth camera 11 captures the depth image data and color image data of each path P1-P7; the six-axis sensor 14 senses the acceleration, moving speed, direction and steering angle of the self-driving car V when it moves on each path P1-P7; the rotary encoder 15 senses the number of steps of the self-driving car V on each path P1-P7 and converts it into a moving distance. Therefore, the processor 13 can obtain the depth image data of each path P1 - P7 , the color image data of each path P1 - P7 , the acceleration, moving speed, direction, steering angle, and moving distance of the self-driving vehicle V when moving in each path P1 - P7 .

[0073] Next, the processor 13 generates training data according to the depth image data, the color image data, the accelerations, the moving speeds, the directions, the moving distances and the turning angles of the paths P1 to P7, and inputs the training data into the machine learning model for deep learning to generate training results, and inputs the training results into the processor 13, and the processor 13 generates a signal and transmits the signal to the controller 12, so that the controller 12 generates a control signal to perform navigation of the self-driving car V. In this embodiment, the machine learning model can be a convolutional neural network model (CNN); in another embodiment, the machine learning model can also be other similar models. The user operates the self-driving car V to move at least once on the target route TR for training.

[0074] Among them, the machine learning model performs deep learning based on the depth image data of the paths P1~P7 to imitate the user's operation of the self-driving car V in each path P1~P7 (including acceleration, moving speed, direction, steering angle and moving distance), and determines the positions of the stops S1~S7 based on the color image data of the paths P1~P7.

[0075] Through the above-mentioned training mechanism, when the self-driving car V enters the automatic navigation mode, the controller 12 obtains the accelerations, moving speeds, directions, steering angles and moving distances corresponding to the real-time depth image data according to the training results and the real-time depth image data and real-time color image data captured by the color depth camera 11, and controls the self-driving car V to move with the accelerations, moving speeds, directions, steering angles and moving distances in real time, and determines the positions of the stops S1 to S7, so that the self-driving car V can stop moving when it reaches the stops S1 to S7. The details of machine learning will be described in the subsequent embodiments.

[0076] See also Figure 4 , which is a schematic diagram of a machine learning model of a self-driving vehicle navigation device according to the first embodiment of the present disclosure. As shown in the figure, the machine learning model M (a convolutional neural network model in this embodiment) may include a convolution layer m1, a dropout layer m2, a flatten layer m3, and a fully connected layer m4.

[0077] As mentioned above, the processor 13 generates training data according to the depth image data d1, the color image data d2, the accelerations d3, the moving speeds d4, the directions d5, the moving distances d6 and the turning angles d7, and inputs the training data into the machine learning model M. Among them, the convolution layer m1 can find the important features of the image data; the drop layer m2 can perform regularization to avoid over-learning; the flat layer m3 can convert the image data into one-dimensional information; and the fully connected layer m4 can integrate the features obtained from the previous layers. The machine learning model M can perform deep learning through the above data to generate a training result FR.

[0078] See also Figures 5A to 5E, which is a schematic diagram of a dimensionality reduction method for the depth image data of the self-driving vehicle navigation device of the first embodiment of the present disclosure. The processor 13 can execute the dimensionality reduction method to process the depth image data d1. The processor 13 can execute the dimensionality reduction method to convert the depth image data d1 of each path P1 to P7 into two-dimensional contour data, and then convert the two-dimensional contour data into one-dimensional array data, and then perform deep learning on the two-dimensional contour data and the one-dimensional array data. Therefore, as mentioned above, the processor 13 can reduce the dimensionality of the depth image data d1 when performing deep learning, so as to serve as part of the training data, so as to more effectively perform deep learning and improve the efficiency of deep learning.

[0079] Similarly, the processor 13 also performs deep learning based on the color image data d2 through the machine learning model M, so that the processor 13 can identify the color images corresponding to each stop S1-S7 based on the image features of the color image. In this way, after the training program is completed, the controller 11 can determine the stops S1-S7 in the target route TR based on the real-time color image data captured by the color depth camera 11 and the training results FR; the details of this part will be Fig.7D The details are described in the embodiments.

[0080] Figure 5A is one of the depth images of the depth image data d1 of any path P1-P7, and the depth image includes the right wall Rw and the left wall Lw. The length and width of the screen C of the color depth camera 11 are L and W respectively.

[0081] like Figure 5B As shown, the pixel coordinates of each pixel are (x, y); x can be 1 to L, and y can be 1 to W, and the depth value of this pixel coordinate is z x，y .

[0082] like Figure 5C As shown, N represents the virtual position of the lens; f represents the virtual distance; dp represents the depth value z of this pixel x，y θ represents the angle of the field of view (FOV) of the color depth camera 11; A represents the center line of the screen C; O represents the origin, that is, the bottom point of the field of view of the color depth camera 11; θ x represents the angle between the center line A and the origin O and the pixel at the same time; K represents the real position of the object; WX represents the real coordinate of the pixel on the X-axis (the length L, width W and the angle θ of the visible range of the color depth camera 11 are known).

[0083] Therefore, the virtual distance f can be calculated by the following formula (1):

[0084]

[0085] Since f and the coordinates (x, y) of this pixel are known, θ x It can be calculated by the following formula (2):

[0086]

[0087] Since θ x and line segment dp are known, so the actual coordinate WX of this pixel on the X-axis can be calculated by the following formula (3):

[0088] sinθ x ×dp=WX........................(3)

[0089] The actual coordinate WY of the pixel on the Y axis can also be calculated in the same way.

[0090] like Figure 5D As shown, the real coordinates (WX, WY, WZ) of each pixel are converted to their corresponding angle θ wx，wy，wz Then a three-dimensional real coordinate stereogram can be obtained.

[0091] like Figure 5E As shown, a two-dimensional image can be obtained according to the minimum z value of each WY of the aforementioned three-dimensional real coordinate stereogram, and then a one-dimensional image (composed of a series of angles and z values) can be obtained by recording the z value of the WY axis of the two-dimensional image and its corresponding angle.

[0092] From the above, it can be seen that the processor 13 can process the depth image data d1 through a dimensionality reduction method to convert the depth image data d1 into two-dimensional contour data and one-dimensional array data, thereby greatly reducing the amount of information, and perform deep learning based on the two-dimensional contour data and one-dimensional array data through a machine learning model M, so that deep learning can be effectively performed and the efficiency of deep learning can be improved.

[0093] See also Figures 6A to 6C , which is a schematic diagram of an automatic navigation mode using existing color vision recognition learning technology. When the automatic navigation mode is trained using existing color vision recognition learning technology, the navigation device of the self-driving vehicle V needs to learn how to move based on the color features in the color image data. Fig. 6A As shown, the navigation device of the self-driving car V needs to learn that there are left walls Lw and right walls Rw on the left and right sides of the self-driving car V based on the color features in the color image data, i.e., the color block Gs below the left wall Lw and the right wall Rw, and that when the image has this color feature, the self-driving car V should go straight.

[0094] like Figure 6BAs shown, the navigation device of the self-driving car V needs to learn based on the color feature in the image, that is, the color block Gs below the left wall Lw, that when the image has this color feature, there is a left wall Lw in front of the self-driving car V, and the self-driving car V should turn right and then go straight.

[0095] like Figure 6C As shown, the navigation device of the self-driving car V needs to learn that there are left walls Lw and right walls Rw on the left and right sides of the self-driving car V based on the color features in the image, i.e., the color blocks Gs below the left wall Lw and the right wall Rw, and that the self-driving car V should turn right and then go straight when the image has this color feature.

[0096] As can be seen from the above, the automatic navigation mode using the existing color vision recognition learning technology needs to learn based on the color features in the color image data, so a large amount of image similarity analysis needs to be performed, resulting in a significant reduction in the stability of this automatic navigation mode. Therefore, the present disclosure can logically analyze the environment response behavior based on the above-mentioned depth image data to supplement and improve the low stability when using color image data and enhance stability.

[0097] See also Figures 7A to 7D , which is a schematic diagram of the automatic navigation mode of the self-driving vehicle navigation device of the first embodiment of the present disclosure. When the self-driving vehicle V enters the automatic navigation mode, the controller 12 performs automatic navigation according to the real-time depth image data captured by the color depth camera 11, the real-time color image data captured by the color depth camera 11 and the training result FR. Among them, the controller 12 compares each real-time depth image of the real-time depth image data with the training result FR to determine the acceleration d3, moving speed d4, direction d5, moving distance d6 and steering angle d7 corresponding to the real-time depth image, and controls the self-driving vehicle V to move with the acceleration d3, moving speed d4, direction d5, moving distance d6 and steering angle d7. As mentioned above, the processor 13 mainly determines the corresponding acceleration d3, moving speed d4, direction d5, moving distance d6 and steering angle d7 through the real-time depth image data, and reduces the dimension of the depth image data d1 when performing deep learning to serve as part of the training data, thereby greatly reducing the amount of information so that the learning result can converge. Therefore, the controller 12 can effectively imitate human operation patterns through the results of deep learning.

[0098] like Fig. 7A As shown, the controller 12 can determine based on the comparison results that there are left walls Lw and right walls Rw on both sides of the current position, and there is a higher depth in front of the current position; therefore, the controller 12 can control the self-propelled vehicle V to move straight with the corresponding acceleration d3, moving speed d4, direction d5 and moving distance d6.

[0099] like Figure 7BAs shown, the controller 12 can determine that the left side and the front of the current position are both the left wall Lw according to the comparison result; therefore, the controller 12 can control the self-propelled vehicle V to turn at the corresponding steering angle d7, and then control the self-propelled vehicle V to go straight.

[0100] like Figure 7C As shown, the controller 12 can determine that there are left walls Lw and right walls Rw on both sides of the current position based on the comparison results, and the right front of the current position has a higher depth; therefore, the controller 12 can control the self-driving vehicle V to turn at the corresponding steering angle d7, and then control the self-driving vehicle V to go straight.

[0101] As mentioned above, the processor 13 performs deep learning based on the color image data d2 through the machine learning model M, so that the processor 13 can identify the color image corresponding to each stop S1-S7 based on the image features of the color image. In this way, the controller 11 can determine one or more stops S1-S7 in the target route TR based on the real-time color image data captured by the color depth camera 11 and the training result FR. Fig.7D As shown, the controller 12 determines the stops S1 to S7 in the target route TR according to the real-time color image data captured by the color depth camera 11 and the training result FR. The controller 12 compares each real-time color image of the real-time color image data with the training result FR to determine whether the real-time color image corresponds to the color image of each stop S1 to S7 in the training result FR. Then, the controller 12 compares the similarity between the real-time color image data captured by the color depth camera 11 and the color images of the stops S1 to S7 in the training result FR with a preset threshold value to determine whether the current position is one of the stops S1 to S7; the preset threshold value can be set according to the requirements; in the present embodiment, the preset threshold value can be 60%. If the controller 12 determines that the similarity between the real-time color image and the color image of any stop S1-S7 in the training result FR is greater than the preset threshold value, the controller 12 determines that the current position is one of the stops S1-S7, and controls the self-driving vehicle V to stop at the current position, so that the user can place the goods on the self-driving vehicle V, or remove the goods from the self-driving vehicle V. The aforementioned preset threshold value can be set according to actual needs. Of course, the stops S1-S7 can also be trained and judged through the depth image data d1, but the color image can provide more image features, so the stops S1-S7 can be more effectively identified through the color image.

[0102] As can be seen from the above, the self-driving car navigation device 1 of the present embodiment generates training data based on the depth image data d1, the color image data d2, the accelerations d3, the moving speeds d4, the directions d5, the moving distances d6 and the turning angles d7, and generates training results FR through deep learning through the machine learning model M to imitate the human operation mode. Therefore, the self-driving car navigation device 1 can realize the automatic navigation of the self-driving car without using positioning technology, so the cost of automatic navigation technology can be greatly reduced.

[0103] In addition, the self-driving car navigation device 1 does not completely use the color image data d2 for training through image similarity, but uses the depth image data d1 to determine the acceleration d3, moving speed d4, direction d5, moving distance d6 and turning angle d7 of each path P1~P7, and the color image data d2 is used to determine the stops S1~S7. Therefore, the self-driving car navigation device 1 can compare according to the environmental contour, spatial logic concept and image similarity, so it can greatly reduce the amount of data and effectively solve the problem of over-learning of image data or the image data volume is too complex to converge. Therefore, the self-driving car navigation device 1 performs deep learning more effectively and can improve the efficiency of deep learning.

[0104] Of course, the above is for illustration only, and the components of the self-driving vehicle navigation device 1 and their coordination relationships can be changed according to actual needs, and the present disclosure is not limited thereto.

[0105] As mentioned above, existing automatic navigation technology needs to be matched with different positioning technologies according to different application environments or accuracy requirements; however, these positioning technologies all have certain deployment difficulties and complexities, which will greatly increase the cost of automatic navigation technology. However, according to the embodiments of the present disclosure, the self-driving car navigation device can generate training results through a machine learning model to imitate human operation modes without using positioning technology to achieve automatic navigation of the self-driving car, thereby greatly reducing the cost of automatic navigation technology.

[0106] In addition, when the existing machine learning technology is applied to the automatic navigation technology of self-driving cars, the learning results may not converge due to factors such as too much image information, too much noise background, and too high image similarity. Therefore, the existing machine learning technology cannot be applied to the automatic navigation technology of self-driving cars. However, according to the embodiments of the present disclosure, the self-driving car navigation device uses a dimensionality reduction method to process the depth image data, so that the amount of information of the depth image data is greatly reduced, and deep learning is performed based on the depth image data after dimensionality reduction through a machine learning model, so that deep learning can be effectively performed and the efficiency of deep learning can be improved.

[0107] In addition, according to the embodiments of the present disclosure, the self-driving vehicle navigation device performs deep learning based on the depth image data and the color image data simultaneously through the machine learning model, determines the acceleration, moving speed, direction, steering angle and moving distance of the self-driving vehicle based on the depth image data, and identifies the stop based on the color image data, thereby effectively improving the recognition effect.

[0108] Furthermore, according to the embodiments of the present disclosure, the self-driving vehicle navigation device integrates multiple sensors, including a six-axis sensor, a rotary encoder, a compass, and an obstacle avoidance sensor, etc., so it can more accurately simulate human senses, making the automatic navigation of the self-driving vehicle closer to the human operation mode.

[0109] See also Figure 8 , which is a flow chart of the self-driving vehicle navigation method of the first embodiment of the present disclosure. The self-driving vehicle navigation method of this embodiment includes the following steps:

[0110] Step S81: operate the self-propelled vehicle to move along a target route including at least one path and at least one stop.

[0111] Step S82: Move the self-driving vehicle along the target route at least once.

[0112] Step S83: Capture depth image data and color image data of each path using a color depth camera.

[0113] Step S84: Record the acceleration, moving speed, direction, steering angle and moving distance of the self-propelled vehicle on each path through at least one sensor, including a six-axis sensor and a rotary encoder.

[0114] Step S85: Generate training data via a processor according to the depth image data, the color image data, the accelerations, the moving speeds, the directions, the turning angles, and the moving distances of the paths.

[0115] Step S86: The processor inputs the training data into the machine learning model for deep learning to generate training results.

[0116] Step S87: Automatically navigate the self-propelled vehicle according to the real-time depth image data captured by the color depth camera, the real-time color image data captured by the color depth camera, and the training results through the controller.

[0117] See also Fig. 9 , which is a block diagram of a self-driving vehicle navigation device according to a second embodiment of the present disclosure. As shown in the figure, the self-driving vehicle navigation device 2 can be applied to a self-driving vehicle V (such as Figure 1The self-driving vehicle navigation device 2 can perform automatic navigation for the self-driving vehicle V after a training procedure. The self-driving vehicle navigation device 2 includes a color depth camera 21, a controller 22, a processor 23, a six-axis sensor 24 and a rotary encoder 25.

[0118] The color depth camera 21 is connected to the processor 23 and is used to capture depth image data and color image data.

[0119] The six-axis sensor 24 is connected to the processor 23. The six-axis sensor 24 is used to sense the acceleration, moving speed, direction and steering angle of the self-propelled vehicle V when it moves.

[0120] The rotary encoder 25 is connected to the processor 23. The rotary encoder 25 is used to sense the number of steps of the self-propelled vehicle V and convert the number of steps of the self-propelled vehicle V into a moving distance.

[0121] The above components are similar to the above embodiments, so they are not described in detail here. Different from the above embodiments, the self-driving vehicle navigation device 2 may further include an obstacle avoidance sensor 26. The obstacle avoidance sensor 26 may detect whether there is an obstacle in front of the self-driving vehicle V; in one embodiment, the obstacle avoidance sensor may be an ultrasonic sensor, an infrared sensor or other similar components.

[0122] When the self-driving car V enters the automatic navigation mode, the controller 22 performs automatic navigation according to the real-time depth image data captured by the color depth camera 21, the real-time color image data captured by the color depth camera 21, and the training results. If the obstacle avoidance sensor 26 detects an obstacle (person or object) in front of the self-driving car V, the controller 22 can immediately control the self-driving car V to stop to prevent the self-driving car V from colliding with the obstacle.

[0123] Of course, the above is for illustration only, and the components of the self-driving vehicle navigation device 2 and their coordination relationships can be changed according to actual needs, and the present disclosure is not limited thereto.

[0124] In summary, according to the embodiments of the present disclosure, the self-driving vehicle navigation device can generate training results through a machine learning model to imitate human operation modes without using positioning technology to achieve automatic navigation of the self-driving vehicle, thereby significantly reducing the cost of automatic navigation technology.

[0125] In addition, according to the embodiments of the present disclosure, the self-driving vehicle navigation device applies a dimensionality reduction method to process the depth image data, thereby significantly reducing the amount of information in the depth image data, and performs deep learning based on the depth image data after dimensionality reduction through a machine learning model, thereby effectively performing deep learning and improving the efficiency of deep learning.

[0126] In addition, according to the embodiments of the present disclosure, the self-driving car navigation device performs deep learning based on the depth image data and the color image data simultaneously through the machine learning model. The environmental profile and spatial logic concept of the self-driving car navigation device can be enhanced through the depth image data after dimensionality reduction; therefore, the self-driving car navigation device no longer uses image similarity learning alone, but compares the environmental profile, spatial logic concept and image similarity to determine the acceleration, moving speed, direction, steering angle, moving distance and stop of the self-driving car. The above mechanism can effectively solve the problem of over-learning of image data or the problem that the amount of image data is too complex to converge, so it can effectively improve the navigation recognition effect.

[0127] Furthermore, according to the embodiments of the present disclosure, the self-driving vehicle navigation device integrates multiple sensors, including a six-axis sensor, a rotary encoder, a compass, and an obstacle avoidance sensor, etc., so it can more accurately simulate human senses, making the automatic navigation of the self-driving vehicle closer to the human operation mode.

[0128] It can be seen that the present disclosure has indeed achieved the desired improved effect by breaking through the previous technology, and it is not easy for technical personnel in this field to think of. Its progressiveness and practicality obviously meet the application requirements of the patent, and a patent application is filed in accordance with the law.

[0129] The above description is for illustrative purposes only and is not intended to be limiting. Any other equivalent modifications or changes made to the present invention without departing from the spirit and scope of the present invention should be included in the scope of the appended patent applications.

Claims

1. A self-propelled vehicle navigation device, arranged on a self-propelled vehicle, characterized in that: Include: a color depth camera for capturing a depth image data and a color image data of each path when the self-propelled vehicle is operated to move in a moving mode along a target route including at least one path and at least one stop; at least one sensor for recording an acceleration, a moving speed, a direction, a steering angle, and a moving distance of the self-propelled vehicle on each of the paths; and a processor, generating training data according to the depth image data, the color image data, the accelerations, the moving speeds, the directions, the turning angles, and the moving distances of the paths, and inputting the training data into a machine learning model for deep learning to generate a training result; wherein the machine learning model performs deep learning according to the depth image data of the paths to imitate the moving mode, and determines the position of one or more of the stops according to the color image data or the depth image data of the paths, The processor reduces the dimension of the depth image data of each path to convert the real coordinates of each pixel of the depth image data of each path and the angle of the visible range of the corresponding color depth camera to obtain a three-dimensional real coordinate stereogram, wherein the real coordinates of each pixel include the real coordinates of the pixel on the X-axis and the real coordinates of the pixel on the Y-axis, and then converts the three-dimensional real coordinate stereogram into two-dimensional contour data according to the minimum z value of each real coordinate of the Y-axis of the three-dimensional real coordinate stereogram, and converts the two-dimensional contour data into one-dimensional array data, and performs deep learning based on the two-dimensional contour data and the one-dimensional array data as part of the training data.

2. The self-driving vehicle navigation device according to claim 1, characterized in that: The invention also comprises a controller, which automatically navigates the self-propelled vehicle according to a real-time depth image data and a real-time color image data captured by the color depth camera and the training result.

3. The self-driving vehicle navigation device according to claim 2, characterized in that: The controller obtains the accelerations, moving speeds, directions, steering angles, and moving distances corresponding to the real-time depth image data according to the training results and the real-time depth image data captured by the color depth camera, and controls the self-driving vehicle to move with the accelerations, moving speeds, directions, steering angles, and moving distances in real time.

4. The self-driving vehicle navigation device according to claim 3, characterized in that: The controller determines one or more stops in the target route according to the real-time color image data captured by the color depth camera and the training result.

5. The self-driving vehicle navigation device according to claim 4, characterized in that: The controller compares the similarity between the real-time color image data captured by the color depth camera and the color image of any one of the stop stations in the training result with a preset threshold value to determine whether the current position is the stop station.

6. The self-driving vehicle navigation device according to claim 1, characterized in that: The self-propelled vehicle is operated to move along the target route at least once.

7. The self-driving vehicle navigation device according to claim 1, characterized in that: The sensors include a six-axis sensor, and the six-axis sensor detects the acceleration, the moving speed, the direction and the steering angle of the self-propelled vehicle along each of the paths.

8. The self-driving vehicle navigation device according to claim 1, characterized in that: The sensors further include a rotary encoder, which detects the moving distance of the self-propelled vehicle along each of the paths.

9. The self-driving vehicle navigation device according to claim 1, characterized in that: The sensors further include an obstacle avoidance sensor, which detects whether there is an obstacle in front of the self-propelled vehicle.

10. The self-driving vehicle navigation device according to claim 9, characterized in that: The obstacle avoidance sensor is an ultrasonic sensor or an infrared sensor.

11. The self-driving vehicle navigation device according to claim 1, characterized in that: The machine learning model is a convolutional neural network model.

12. A self-driving vehicle navigation method, characterized in that: Include: Operating a self-propelled vehicle to move in a moving mode along a target route including at least one path and at least one stop; Capturing a depth image data and a color image data of each path by a color depth camera; Recording an acceleration, a moving speed, a direction, a steering angle, and a moving distance of the self-propelled vehicle on each of the paths by at least one sensor; generating training data through a processor according to the depth image data, the color image data, the accelerations, the moving speeds, the directions, the turning angles, and the moving distances of the paths; and The processor inputs the training data into a machine learning model for deep learning to generate a training result; wherein the machine learning model performs deep learning based on the depth image data of the paths to imitate the movement mode, and determines the location of one or more of the stops based on the color image data or the depth image data of the paths; The step of generating training data according to the depth image data, the color image data, the accelerations, the moving speeds, the directions, the turning angles and the moving distances of the paths by the processor further includes: The processor reduces the dimension of the depth image data of each path to convert the real coordinates of each pixel of the depth image data of each path and the angle of the visual range of the corresponding color depth camera to obtain a three-dimensional real coordinate stereogram, wherein the real coordinates of each pixel include the real coordinates of the pixel on the X-axis and the real coordinates of the pixel on the Y-axis, and then converts the three-dimensional real coordinate stereogram into two-dimensional contour data according to the minimum z value of each real coordinate of the Y-axis of the three-dimensional real coordinate stereogram, and converts the two-dimensional contour data into one-dimensional array data, and performs deep learning based on the two-dimensional contour data and the one-dimensional array data as part of the training data.

13. The self-driving vehicle navigation method according to claim 12, characterized in that: Also includes: The self-driving vehicle is automatically navigated by a controller according to a real-time depth image data captured by the color depth camera, a real-time color image data and the training result.

14. The self-driving vehicle navigation method according to claim 13, characterized in that: The step of automatically navigating the self-driving vehicle according to the real-time depth image data captured by the color depth camera, the real-time color image data and the training result by the controller further includes: The controller obtains the accelerations, moving speeds, directions, steering angles, and moving distances corresponding to the real-time depth image data according to the training results and the real-time depth image data captured by the color depth camera, and controls the self-driving vehicle to move with the accelerations, moving speeds, directions, steering angles, and moving distances in real time.

15. The self-driving vehicle navigation method according to claim 14, characterized in that: Also includes: The controller determines one or more stops in the target route according to real-time color image data captured by the color depth camera and the training result.

16. The self-driving vehicle navigation method according to claim 15, characterized in that: The step of determining one or more stops in the target route by the controller according to the real-time color image data captured by the color depth camera and the training result further includes: The controller compares the similarity between the real-time color image data captured by the color depth camera and the color image of any one of the stop stations in the training result with a preset threshold value to determine whether the current position is the stop station.

17. The self-driving vehicle navigation method according to claim 12, characterized in that: The step of operating the self-propelled vehicle to move along the target route including the paths further comprises: The self-propelled vehicle is moved along the target route at least once.

18. The self-driving vehicle navigation method according to claim 12, characterized in that: The step of recording the acceleration, the moving speed, the direction, the steering angle, and the moving distance of the self-propelled vehicle on each of the paths by the sensors comprises: Using a six-axis sensor to detect the acceleration, the moving speed, the direction, and the steering angle of the self-propelled vehicle along each of the paths; and The moving distance of the self-propelled vehicle along each path is detected by a rotary encoder.

19. The self-driving vehicle navigation method according to claim 12, characterized in that: Also includes: An obstacle avoidance sensor is used to detect whether there is an obstacle in front of the self-propelled vehicle.

20. The self-driving vehicle navigation method according to claim 19, characterized in that: The obstacle avoidance sensor is an ultrasonic sensor or an infrared sensor.

21. The self-driving vehicle navigation method according to claim 12, characterized in that: The machine learning model is a convolutional neural network model.

Images