Road recognition device and method, electronic device

Abstract

To provide a road recognition device, a method, and an electronic apparatus that quickly and accurately recognize a plurality of types of roadways.SOLUTION: A road recognition device includes: a first detection unit that detects a traffic lane in an image by using a first neural network, and acquires a first detection result of the traffic lane; a second detection unit that detects a vehicle and / or a road direction sign in the image by using a second neural network, and acquires a second detection result of the vehicle and / or a third detection result of the road direction sign; a classification unit that performs classification for a display direction of the vehicle based on the second detection result to acquire a first classification result of the display direction, and performs classification for the direction of the road direction sign based on the third detection result of the road direction sign to acquire a second classification result of the direction of the road direction sign; and a first recognition unit 104 that recognizes a forward direction roadway and / or a reverse direction roadway in the image based on at least one of the first classification result of the display direction of the vehicle and the second classification result of the direction of the road direction sign and the first detection result of the vehicle.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to the field of information technology.

Background Art

[0002] Recognizing various types of roads is a useful technology, which can enable a driver to drive on the correct lane. When there is a sidewalk, road recognition can prompt the driver to slow down the vehicle speed. When an emergency stop is necessary, road recognition can prompt the driver to select a temporary parking space when there is a roadside strip. Also, road recognition can help to obtain traffic accident information based on an in-vehicle camera head or a surveillance camera head. Furthermore, road recognition is also very useful for autonomous driving.

[0003] Currently, people mainly Road line (means a line drawn on the road surface along the longitudinal direction of the road) focus on the detection of Road line in the forward direction, that is, in the same direction as the vehicle they are driving. Road line They pay attention to it.

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, the inventor has discovered the following. That is, for some complex road scenes, especially city scenes, it is not sufficient to only consider the forward lane, and generally multiple lanes exist simultaneously in these scenes. Currently, there is still no effective method that can recognize multiple types of lanes.

[0005] In order to solve at least one of the above problems, embodiments of the present invention provide a road recognition device and method, and an electronic device, whereby multiple types of lanes can be recognized quickly and accurately.

Means for Solving the Problems

[0006] According to a first aspect of an embodiment of the present invention, a road recognition device is provided, and the device includes: For an image acquired by an in-vehicle imaging device, using a first neural network to detect Road line in the image, and obtaining a first detection result of the Road line by a first detection unit; For the image acquired by the in-vehicle imaging device, using a second neural network to detect vehicles and / or Travel direction of road signs in the image, and obtaining a second detection result of the vehicle and / or a third detection result of the Travel direction of road signs by a second detection unit; Based on the second detection result of the vehicle, classifying the Shooting direction of the vehicle , obtaining a first classification result of the Shooting direction of the vehicle , and / or, based on the third detection result of the Travel direction of road signs , classifying the Travel direction of road signs , obtaining a second classification result of the Travel direction of road signs by a classification unit; and Based on at least one of the first classification result of the Shooting direction of the vehicle and the second classification result of the Travel direction of road signs , and the first detection result of the Road line , recognizing a forward lane and / or a reverse lane in the image by a first recognition unit.

[0007] According to a second aspect of an embodiment of the present invention, an electronic device is provided, and the electronic device includes the device described in the first aspect of the embodiment of the present invention.

[0008] According to a third aspect of an embodiment of the present invention, a road recognition method is provided, and the method includes: For an image acquired by an in-vehicle imaging device, using a first neural network to detect Road line in the image, and obtaining a first detection result of the Road line ; For the image acquired by the in-vehicle imaging device, using a second neural network to detect vehicles and / or Travel direction of road signs in the image, and obtaining a second detection result of the vehicle and / or a third detection result of the Travel direction of road signs ; Based on the second detection result of the vehicle, classify the Shooting direction of the vehicle , and obtain the first classification result of the Shooting direction of the vehicle . And / or, classify the Travel direction of road signs based on the third detection result of the Travel direction of road signs , and obtain the second classification result of the Travel direction of road signs ; and Based on at least one of the first classification result of the Shooting direction of the vehicle and the second classification result of the Travel direction of road signs , and the first detection result of the Road line , recognize the forward lane and / or the reverse lane in the image.

[0009] The advantageous effects of the embodiments of the present invention are as follows, that is, Shooting direction of the vehicle Based on at least one of the first classification result of the Travel direction of road signs and the second classification result of the Road line , and the first detection result of the By recognizing the forward lane and / or the reverse lane in the image, multiple types of lanes can be recognized quickly and accurately.

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Modes for Carrying Out the Invention

[0011] Hereinafter, with reference to the accompanying drawings, preferred embodiments for carrying out the present invention will be described in detail.

Embodiment

[0012] An embodiment of the present invention provides a road recognition device. FIG. 1 is a diagram showing the road recognition device in Embodiment 1 of the present invention.

[0013] As shown in FIG. 1, the road recognition device 100 includes the following.

[0014] First detection unit 101: For the image acquired by the in-vehicle imaging device, using the first neural network to detect the Road line in the image, and obtaining the first detection result of the Road line ; Second detection unit 102: For the image acquired by the in-vehicle imaging device, using the second neural network to detect the vehicle and / or Travel direction of road signs in the image, and obtaining the second detection result of the vehicle and / or the third detection result of the Travel direction of road signs ; Classification unit 103: Performing classification on the Shooting direction of the vehicle based on the second detection result of the vehicle, and obtaining the first classification result of the Shooting direction of the vehicle , and / or performing classification on the Travel direction of road signs based on the third detection result of the Travel direction of road signs , and obtaining the second classification result of the Travel direction of road signs ; and First recognition unit 104: Based on at least one of the first classification result of the Shooting direction of the vehicle and the second classification result of the Travel direction of road signs , and the first detection result of the Road line , recognizing the forward lane and / or the reverse lane in the image.

[0015] In one implementation manner of the embodiment of the present invention, the object (target) detected by the road recognition device 100 is the image acquired by the in-vehicle imaging device.

[0016] For example, the in-vehicle imaging device is an in-vehicle camera head, and the in-vehicle camera head can record each frame of the video captured from the driving perspective. In this case, the road recognition device 100 can perform processing for each frame on each frame, and can also update the previous recognition result based on the recognition result of the current frame.

[0017] For the image acquired by the in-vehicle imaging device, the first detection unit 101 uses a first neural network to detect Road line in the image, and obtains the first detection result of the Road line . The first neural network may be various neural networks that can detect Road line .

[0018] For example, the first neural network is trained by a deep learning method.

[0019] The first neural network is, for example, LaneNet -based. For example, Of TuSimple Holdings,Inc. The model is used as a pre-training model, and training is performed by correcting the model using a pre-collected training data set. After the training is completed, the first neural network can be obtained. By using the first neural network, Road line in the image can be accurately detected.

[0020] FIG. 2 is a diagram showing an image acquired by the in-vehicle imaging device in Embodiment 1 of the present invention. As shown in FIG. 2, by the detection of the first neural network, Road line 201 and Road line 202 are acquired.

[0021] In one implementation manner of the embodiment of the present invention, for the image acquired by the in-vehicle imaging device, the second detection unit 102 uses a second neural network to detect vehicles and / or Travel direction of road signs in the image, and obtains the second detection result of the vehicle and / or the third detection result of the Travel direction of road signs .

[0022] The second detection unit 102 performs detection on the same image processed by the first detection unit 101 using a second neural network. The targets to be detected are vehicles and Travel direction of road signs at least one of them.

[0023] The second neural network may be various neural networks capable of detecting vehicles and Travel direction of road signs . For example, the second neural network is trained by a deep learning method.

[0024] The second neural network is, for example, FPN (Feature Pyramid Networks). FPN is a neural network suitable for multi-scene multi-target detection with good performance, and relatively high detection accuracy can be obtained by using this network.

[0025] In one implementation manner of an embodiment of the present invention, the classification unit 103 classifies the Shooting direction of the vehicle based on the second detection result of the vehicle, and obtains the first classification result of the Shooting direction of the vehicle , and / or classifies the Travel direction of road signs based on the third detection result of the Travel direction of road signs and obtains the second classification result of the Travel direction of road signs .

[0026] For example, the classification unit 103 uses a first classifier to classify the detected Shooting direction of the vehicle and obtains the first classification result, and / or uses a second classifier to classify the detected Travel direction of road signs and obtains the second classification result. The first classifier and the second classifier may be the same classifier or different classifiers, and the first classifier and the second classifier are classifiers trained by a deep learning method.

[0027] In one implementation manner of an embodiment of the present invention, Shooting direction of the vehicleIt refers to the direction in which the detected vehicle is displayed in the image, or it can also be said to refer to the part of the vehicle where the detected vehicle is displayed. For example, in the image, since the front part of the vehicle is displayed, the Shooting direction of the vehicle is "front".

[0028] In one implementation manner of an embodiment of the present invention, the quantity of the categories output by the first classifier may be determined according to actual needs. For example, there are 8 categories that the first classifier can output.

[0029] Figure 3 is a diagram showing various Shooting direction of the vehicle in Embodiment 1 of the present invention. As shown in Figure 3, in the upper row, from left to right in sequence are "front", "right front", "left front", and "right"; in the lower row, from left to right in sequence are "back", "right back", "left back", and "left".

[0030] In one implementation manner of an embodiment of the present invention, Travel direction of road signs refers to the orientation of the detected Travel direction of road signs in the image. For example, an Travel direction of road signs including an arrow pointing in the forward direction is the Travel direction of road signs in the forward direction, and an Travel direction of road signs including an arrow pointing in the reverse direction is the Travel direction of road signs in the reverse direction.

[0031] In one implementation manner of an embodiment of the present invention, the forward direction refers to the traveling direction of the vehicle where the in-vehicle imaging device is located, that is, the forward traveling direction; the reverse direction refers to the direction opposite to the traveling direction of the vehicle where the in-vehicle imaging device is located, that is, the reverse traveling direction.

[0032] In one implementation manner of an embodiment of the present invention, the quantity of the categories output by the second classifier may be determined according to actual needs.

[0033] For example, there are 2 categories that the second classifier can output, that is, the Travel direction of road signsand in the reverse direction Travel direction of road signs It is.

[0034] FIG. 4 is a diagram showing various Travel direction of road signs in Example 1 of the present invention. As shown in FIG. 4, all of the Travel direction of road signs in the upper row include arrows pointing in the forward direction Travel direction of road signs Therefore, they are classified as the forward Travel direction of road signs , and all of the Travel direction of road signs in the lower row include arrows pointing in the reverse direction Travel direction of road signs Therefore, they are classified as the reverse Travel direction of road signs .

[0035] In one implementation manner of the embodiment of the present invention, the first recognition unit 104 recognizes the forward lane and / or the reverse lane in the image based on at least one of the first classification result of the Shooting direction of the vehicle and the second classification result of the Travel direction of road signs , and the first detection result of the Road line .

[0036] In one implementation manner of the embodiment of the present invention, the forward lane is the lane in the same direction as the traveling direction of the vehicle where the in-vehicle imaging device that captures the image is located, and the reverse lane is the lane in the direction opposite to the traveling direction of the vehicle where the in-vehicle imaging device that captures the image is located.

[0037] In one implementation manner of the embodiment of the present invention, Road line the lane can be determined based on the first detection result of, for example, there is one lane between two Road line , or there is a lane on one side of one Road line .

[0038] For example, when there is a vehicle with Shooting direction in front in one lane or there is a reverse Travel direction of road signs , it is determined that the lane is a reverse lane; and / or when there is a vehicle with Shooting direction behind in one lane or there is a forward Travel direction of road signs , it is determined that the lane is a forward lane.

[0039] In this way, it is possible to recognize whether the lane in the image is a forward lane or a reverse lane, and it is also possible to obtain information such as road structure and road conditions. Therefore, useful information for various applications can be provided. For example, driving can be guided based on this information, or this information can be used for autonomous driving and traffic accident evaluation, etc.

[0040] In one implementation manner of an embodiment of the present invention, the apparatus 100 may further include the following.

[0041] Third detection unit 105: Detect the roadside reference object in the image using the second neural network, and obtain the fourth detection result of the roadside reference object; and Second recognition unit 106: Based on the first detection result of the Road line and the fourth detection result of the roadside reference object, recognize the sidewalk and / or roadside strip in the image.

[0042] In this way, by further recognizing the sidewalk and / or roadside strip in the image, more useful information can be provided for various applications.

[0043] In one implementation manner of an embodiment of the present invention, the third detection unit 105 detects the roadside reference object in the image using the second neural network. As described above, for example, the second neural network is FPN (Feature Pyramid Networks), and FPN is a neural network with good performance and suitable for multi-scene multi-target detection. By using this network, relatively high detection accuracy can be obtained.

[0044] In one implementation manner of an embodiment of the present invention, the roadside strip refers to a space for temporary parking or an area for other specific uses on the side of the road.

[0045] In one implementation of an embodiment of the present invention, the roadside reference object is a specific object for recognizing a sidewalk and a roadside strip. For example, it is a curb, a guardrail, a fence, and Green plants at least one of them.

[0046] For example, for a lane running closer to the left, when the second recognition unit 106 detects a roadside reference object on the left side of one lane, it determines that there is a sidewalk in the image; and for a lane running closer to the right, when it detects a roadside reference object on the right side of one lane, it determines that there is a sidewalk in the image.

[0047] For example, for a lane running closer to the left, when the second recognition unit 106 cannot detect a roadside reference object on the left side of one lane and there is at least one Road line on the left side of the vehicle where the in-vehicle imaging device is located, it determines that there is a roadside strip in the image; and for a road running closer to the right, when it cannot detect a roadside reference object on the right side of one lane and there is at least one Road line on the right side of the vehicle where the in-vehicle imaging device is located, it determines that there is a roadside strip in the image.

[0048] For example, for the image shown in FIG. 2, if it belongs to a road running closer to the left, the lane in which the vehicle where the in-vehicle imaging device that captures the image is traveling is the forward lane, Road line and in the lane on the right side of 202 Shooting direction where there is a vehicle 203 in front, in this case, this lane is the reverse lane. Also, since roadside reference objects such as a guardrail 204 and a fence 205 are detected on the left side of the lane in which the vehicle where the in-vehicle imaging device is located is traveling, that is, the forward lane, it is determined that there is a sidewalk in the image.

[0049] In one implementation of an embodiment of the present invention, the first detection result, the second detection result, the third detection result, and the fourth detection result are only for distinguishing each detection result in description and do not limit each detection result.

[0050] As can be seen from the above embodiments, Shooting direction of the vehicle at least one of the first classification results of Travel direction of road signs and the second classification results of Road line Based on the first detection result of, by detecting the forward lane and / or the reverse lane in the image, multiple types of lanes can be recognized quickly and accurately.

Embodiment

[0051] An embodiment of the present invention further provides an electronic device. FIG. 5 is a diagram showing the electronic device in Embodiment 2 of the present invention. As shown in FIG. 5, the electronic device 500 includes a road recognition device 501. Since the structure and function of the road recognition device 501 are the same as those described in Embodiment 1, the detailed description thereof is omitted here.

[0052] In one implementation manner of the embodiment of the present invention, the electronic device 500 may be various types of electronic devices, such as an in-vehicle terminal, a mobile terminal, or a computer.

[0053] FIG. 6 is a block diagram showing the system configuration of the electronic device in Embodiment 2 of the present invention. As shown in FIG. 6, the electronic device 600 may further include a processor 601 and a memory 602; the memory 602 is connected to the processor 601. It should be noted that this figure is only an example, and other types of structures may be used to supplement or replace this structure to realize the electrical communication function or other functions.

[0054] As shown in FIG. 6, the electronic device 600 may further include an input unit 603, a display 604, and a power supply 605.

[0055] In one implementation manner of the embodiment of the present invention, the function of the road recognition device described in Embodiment 1 may be integrated into the processor 601. Among them, the processor 601 may be configured as follows, that is, for the image acquired by the in-vehicle imaging device, using the first neural network to Road line in the image Road lineObtain the first detection result; for the image acquired by the in-vehicle imaging device, use a second neural network to detect vehicles and / or Travel direction of road signs in the image, and obtain the second detection result of the vehicle and / or the Travel direction of road signs third detection result; perform classification on the Shooting direction of the vehicle based on the second detection result of the vehicle, and obtain the first classification result of the Shooting direction of the vehicle , and / or perform classification on the Travel direction of road signs based on the third detection result of the Travel direction of road signs , and obtain the second classification result of the Travel direction of road signs ; and, based on at least one of the first classification result of the Shooting direction of the vehicle and the second classification result of the Travel direction of road signs , and the first detection result of the Road line , recognize the forward lane and / or the reverse lane in the image.

[0056] For example, the processor 601 may be further configured as follows, that is, use the second neural network to detect roadside reference objects in the image, and obtain the fourth detection result of the roadside reference objects; and, based on the first detection result of the Road line and the fourth detection result of the roadside reference objects, recognize the sidewalk and / or the roadside strip in the image.

[0057] For example, the first neural network and the second neural network are trained by a deep learning method.

[0058] For example, the first neural network is LaneNet , and the second neural network is FPN (Feature Pyramid Networks).

[0059] For example, performing classification on the Shooting direction of the vehicle based on the second detection result of the vehicle, and obtaining the first classification result of the Shooting direction of the vehicle , and / or performing classification on the Travel direction of road signs based on the third detection result of the Travel direction of road signs , and obtaining the second classification result of the Travel direction of road signs means using a first classifier to detect theShooting direction of the vehicle perform classification on the Travel direction of road signs and obtain the first classification result, and / or perform classification on the

[0060] For example, based on at least one of the first classification result of the Shooting direction of the vehicle and the second classification result of the Travel direction of road signs , and the first detection result of the Road line , recognizing the forward lane and / or the reverse lane in the image includes: when there is a vehicle with Shooting direction in front on one lane or there is a Travel direction of road signs in the reverse direction, determining that the lane is a reverse lane; and / or when there is a vehicle with Shooting direction behind on one lane or there is a Travel direction of road signs in the forward direction, determining that the lane is a forward lane.

[0061] For example, based on the first detection result of the Road line and the fourth detection result of the roadside reference object, recognizing the sidewalk in the image includes: for a lane running closer to the left, when a roadside reference object is detected on the left side of one lane, determining that there is a sidewalk in the image; and for a lane running closer to the right, when a roadside reference object is detected on the right side of one lane, determining that there is a sidewalk in the image.

[0062] For example, based on the first detection result of the Road line and the fourth detection result of the roadside reference object, recognizing the roadside strip in the image includes: for a lane running closer to the left, when no roadside reference object is detected on the left side of one lane and there is at least one Road line on the left side of the vehicle where the in-vehicle imaging device is located, determining that there is a roadside strip in the image; and for a road running closer to the right, when no roadside reference object is detected on the right side of one lane and there is at least one Road lineincluding determining that there is a roadside strip in the image when it exists.

[0063] In another implementation manner of the embodiment of the present invention, the road recognition device described in Embodiment 1 may be arranged independently of the processor 601. For example, the road recognition device may be configured as a chip connected to the processor 601, and the function of the road recognition device may be realized under the control of the processor 601.

[0064] In one implementation manner of the embodiment of the present invention, the electronic device 600 does not necessarily include all the components shown in FIG. 6.

[0065] As shown in FIG. 6, the processor 601 may sometimes be referred to as a controller or an operation controller, and may include a microprocessor or other processor device and / or logic device. The processor 601 can receive an input and control the operation of each component of the electronic device 600.

[0066] The memory 602 may be, for example, one or more of a buffer, a flash memory, an HDD, a removable medium, a volatile memory, a non-volatile memory, or other suitable devices. The processor 601 can execute the program stored in the memory 602 to realize information storage or processing, etc. Since the functions of other components are the same as those in the prior art, the detailed description thereof is omitted here. In addition, each component of the electronic device 600 may be realized by dedicated hardware, firmware, software, or a combination thereof, and all of them belong to the scope of the present invention.

[0067] As can be seen from the above embodiments, Shooting direction of the vehicle at least one of the first classification results of Travel direction of road signs and Road line the second classification results of

Embodiment

[0068] An embodiment of the present invention further provides a road recognition method, and this method corresponds to the road recognition device in Embodiment 1. FIG. 7 is a diagram showing the road recognition method in Embodiment 3 of the present invention. As shown in FIG. 7, this method includes the following steps.

[0069] Step 701: For the image acquired by the in-vehicle imaging device, use a first neural network to detect the Road line in the image, and obtain the first detection result of the Road line ; Step 702: For the image acquired by the in-vehicle imaging device, use a second neural network to detect the vehicle and / or Travel direction of road signs in the image, and obtain the second detection result of the vehicle and / or the third detection result of the Travel direction of road signs ; Step 703: Perform classification on the Shooting direction of the vehicle based on the second detection result of the vehicle, and obtain the first classification result of the Shooting direction of the vehicle , and / or perform classification on the Direction of road signs based on the third detection result of the Direction of road signs , and obtain the second classification result of the Direction of road signs ; and Step 704: Recognize the forward lane and / or reverse lane in the image based on at least one of the first classification result of the Shooting direction of the vehicle and the second classification result of the Direction of road signs , and the first detection result of the Road lines .

[0070] In one implementation manner of the embodiment of the present invention, since the specific implementation methods of the above steps are the same as those described in Embodiment 1, duplicate descriptions are omitted here.

[0071] In one implementation manner of the embodiment of the present invention, there is no limitation on the execution order of Step 701 and Step 702, and they may be executed in parallel or sequentially.

[0072] As can be seen from the above embodiments, Shooting direction of the vehicle of the first classification result and Direction of road signs of the second classification result, at least one of them, andRoad lines Based on the first detection result of Road lines , by recognizing the forward lane and / or the reverse lane in the image, multiple types of lanes can be recognized quickly and accurately.

[0073] An embodiment of the present invention further provides a computer-readable program. When the program is executed in a road recognition device or an electronic device, the program causes the computer to execute the road recognition method described in Embodiment 3 in the road recognition device or the electronic device.

[0074] An embodiment of the present invention further provides a storage medium storing a computer-readable program. The computer-readable program causes the computer to execute the road recognition method described in Embodiment 3 in a road recognition device or an electronic device.

[0075] Also, the device, method, etc. according to the embodiments of the present invention may be implemented by software, may be implemented by hardware, or may be implemented by a combination of hardware and software. The present invention also relates to such a computer-readable program, that is, when the program is executed by a logic component, the logic component can implement the above-described device or component, or the logic component can implement the above-described method or its steps. Further, the present invention also relates to a storage medium storing the above-described program, such as a hard disk, a magnetic disk, an optical disk, a DVD, a flash memory, etc.

[0076] Also, regarding the above-described embodiments, etc., the following supplementary notes are further disclosed.

[0077] (Supplementary Note 1) A road recognition device, For an image acquired by an in-vehicle imaging device, using a first neural network to detect in the image, Road lines and obtaining a first detection result of Road lines ; a first detection unit. Road lines For the image acquired by the in-vehicle imaging device, using a second neural network, vehicles and / or Direction of road signs in the image are detected, and a second detection unit that obtains a second detection result of the vehicle and / or a third detection result of the Direction of road signs ; Based on the second detection result of the vehicle, classification is performed on the Shooting direction of the vehicle , and a first classification result of the Shooting direction of the vehicle is obtained, and / or, based on the third detection result of the Direction of road signs , classification is performed on the Direction of road signs , and a classification unit that obtains a second classification result of the Direction of road signs ; and Based on at least one of the first classification result of the Shooting direction of the vehicle and the second classification result of the Direction of road signs , and the first detection result of the Road lines , a first recognition unit that recognizes a forward lane and / or a reverse lane in the image. An apparatus comprising.

[0078] (Appendix 2) The apparatus according to Appendix 1, further comprising a third detection unit that detects roadside reference objects in the image using the second neural network and obtains a fourth detection result of the roadside reference objects; and Based on the first detection result of the Road lines and the fourth detection result of the roadside reference objects, a second recognition unit that recognizes a sidewalk and / or a roadside strip in the image. An apparatus comprising.

[0079] (Appendix 3) The apparatus according to Appendix 1, wherein the first neural network and the second neural network are trained by a deep learning method. An apparatus.

[0080] (Appendix 4) The apparatus according to Appendix 3, wherein the first neural network is LaneNet and the second neural network is FPN (Feature Pyramid Networks). An apparatus.

[0081] (Appendix 5) The device according to Appendix 1, wherein the classification unit uses a first classifier to classify the detected Shooting direction of the vehicle and obtain the first classification result, and / or uses a second classifier to classify the detected Direction of road signs and obtain the second classification result, wherein the first classifier and the second classifier are the same or different classifiers, and the first classifier and the second classifier are classifiers trained by a deep learning method, a device.

[0082] (Appendix 6) The device according to Appendix 1, wherein the first recognition unit determines the lane as a reverse lane when there is a vehicle in front of it in one lane Shooting direction or there is a vehicle in the reverse direction, and / or determines the lane as a forward lane when there is a vehicle behind it in one lane Direction of road signs or there is a vehicle in the forward direction, a device. Shooting direction Direction of road signs

[0083] (Appendix 7) The device according to Appendix 2, wherein the second recognition unit determines that there is a sidewalk in the image when a roadside reference object is detected on the left side of one lane for a lane running closer to the left side, and determines that there is a sidewalk in the image when a roadside reference object is detected on the right side of one lane for a lane running closer to the right side, a device.

[0084] (Appendix 8) The device according to Appendix 2, wherein the second recognition unit, for a lane running closer to the left side, when no roadside reference object is detected on the left side of one lane and there is at least one Road lines ​​When it exists, it is determined that there is a roadside strip in the image; and for a road running closer to the right side, no roadside reference object is detected on the right side of one lane and at least one Road lines When it exists, it determines that there is a roadside strip in the image, device.

[0085] (Appendix 9) An electronic device, The electronic device includes the device described in Appendix 1, electronic device.

[0086] (Appendix 10) A road recognition method, For the image acquired by the in-vehicle imaging device, using a first neural network, the Road lines in the image is detected, and the first detection result of the Road lines is obtained; For the image acquired by the in-vehicle imaging device, using a second neural network, vehicles and / or Direction of road signs in the image are detected, and the second detection result of the vehicle and / or the third detection result of the Direction of road signs are obtained; Based on the second detection result of the vehicle, classification is performed on the Shooting direction of the vehicle , and the first classification result of the Shooting direction of the vehicle is obtained, and / or, based on the third detection result of the Direction of road signs , classification is performed on the Direction of road signs , and the second classification result of the Direction of road signs is obtained; and the Shooting direction of the vehicle first classification result and at least one of the second classification results of the Direction of road signs , and the first detection result of the Road lines are used to recognize the forward lane and / or the reverse lane in the image, method.

[0087] (Appendix 11) The method described in Appendix 10, further, using the second neural network to detect roadside reference objects in the image, and obtaining the fourth detection result of the roadside reference objects; and theRoad lines A method including recognizing a sidewalk and / or a roadside strip in the image based on the first detection result of the vehicle and the fourth detection result of the roadside reference object.

[0088] (Appendix 12) The method according to Appendix 10, wherein the first neural network and the second neural network are trained by a deep learning method.

[0089] (Appendix 13) The method according to Appendix 12, wherein the first neural network is LaneNet and the second neural network is FPN (Feature Pyramid Networks).

[0090] (Appendix 14) The method according to Appendix 10, performing classification on the Shooting direction of the vehicle based on the second detection result of the vehicle, obtaining the first classification result of the Shooting direction of the vehicle , and / or performing classification on the Direction of road signs based on the third detection result of the Direction of road signs , obtaining the second classification result of the Direction of road signs includes using a first classifier to perform classification on the detected to obtain the first classification result, and / or using a second classifier to perform classification on the detected Shooting direction of the vehicle to obtain the second classification result, Direction of road signs wherein the first classifier and the second classifier are the same or different classifiers, and the first classifier and the second classifier are classifiers trained by a deep learning method.

[0091] (Appendix 15) The method according to Appendix 10, the Shooting direction of the vehicle first classification result and at least one of the Direction of road signs second classification results of theRoad lines Based on the first detection result of in one lane Shooting direction when there is a vehicle in front or a vehicle Direction of road signs in the reverse direction exists, determining the lane as a reverse lane; and / or in one lane Shooting direction when there is a vehicle behind or a vehicle Direction of road signs in the forward direction exists, determining the lane as a forward lane, a method.

[0092] (Appendix 16) The method according to Appendix 11, said Road lines recognizing a sidewalk in the image based on the first detection result of For a lane running closer to the left, when a roadside reference object is detected on the left side of one lane, determining that there is a sidewalk in the image; and For a lane running closer to the right, when a roadside reference object is detected on the right side of one lane, determining that there is a sidewalk in the image, a method.

[0093] (Appendix 17) The method according to Appendix 11, said Road lines recognizing a roadside strip in the image based on the first detection result of For a lane running closer to the left, when a roadside reference object cannot be detected on the left side of one lane and at least one Road lines exists on the left side of the vehicle where the in-vehicle imaging device is located, determining that there is a roadside strip in the image; and For a road running closer to the right, when a roadside reference object cannot be detected on the right side of one lane and at least one Road lines exists on the right side of the vehicle where the in-vehicle imaging device is located, determining that there is a roadside strip in the image, a method.

[0094] The preferred embodiments of the present invention have been described above. However, the present invention is not limited to these embodiments, and any modifications to the present invention belong to the technical scope of the present invention as long as they do not depart from the gist of the present invention.

Claims

1. A road recognition device, a first detection unit that inputs an image acquired by an in-vehicle imaging device into a first neural network and the first neural network outputs road lines in the image, thereby detecting the road lines in the image and obtaining a first detection result of the road lines; a second detection unit that inputs the image acquired by the in-vehicle imaging device into a second neural network and the second neural network outputs the traveling directions of vehicles and / or road signs in the image, thereby detecting the traveling directions of vehicles and / or road signs in the image and obtaining a second detection result of the vehicles and / or a third detection result of the traveling directions of the road signs; a classification unit that classifies with respect to the shooting direction of the vehicle in the second detection result of the vehicle to obtain a first classification result of the shooting direction of the vehicle, and / or classifies with respect to the traveling direction of the road sign in the third detection result of the traveling direction of the road sign to obtain a second classification result of the traveling direction of the road sign; and a first recognition unit that recognizes a forward lane and / or a reverse lane in the image based on at least one of the first classification result of the shooting direction of the vehicle and the second classification result of the traveling direction of the road sign and the first detection result of the road lines, wherein the first recognition unit determines the lane as a reverse lane when there is a vehicle with a forward shooting direction or the traveling direction of a reverse road sign exists in one lane obtained from the first detection result of the road lines; and / or determines the lane as a forward lane when there is a vehicle with a rearward shooting direction or the traveling direction of a forward road sign exists in one lane obtained from the first detection result of the road lines. A road recognition device.

2. The road recognition device according to claim 1, further comprising a third detection unit that inputs the image acquired by the in-vehicle imaging device into the second neural network and the second neural network further outputs roadside reference objects in the image, thereby detecting the roadside reference objects in the image and obtaining a fourth detection result of the roadside reference objects; and a second recognition unit that recognizes a sidewalk and / or a roadside strip in the image based on the first detection result of the road lines and the fourth detection result of the roadside reference objects. When the second recognition unit detects a roadside reference object on the left side of one lane obtained from the first detection result of the road line for a lane running closer to the left side, it determines that there is a sidewalk in the image; and for a lane running closer to the right side, when it detects a roadside reference object on the right side of one lane obtained from the first detection result of the road line, it determines that there is a sidewalk in the image. When the second recognition unit cannot detect a roadside reference object on the left side of one lane obtained from the first detection result of the road line for a lane running closer to the left side and there is at least one road line on the left side of the vehicle where the in-vehicle imaging device is located, it determines that there is a roadside strip in the image; and for a road running closer to the right side, when it cannot detect a roadside reference object on the right side of one lane obtained from the first detection result of the road line and there is at least one road line on the right side of the vehicle where the in-vehicle imaging device is located, it determines that there is a roadside strip in the image. A road recognition device.

3. The road recognition device according to claim 1, The first neural network and the second neural network are trained by a deep learning method. A road recognition device.

4. The road recognition device according to claim 3, The first neural network is LaneNet, and the second neural network is FPN (Feature Pyramid Networks). A road recognition device.

5. The road recognition device according to claim 1, The classification unit inputs the second detection result of the vehicle into a first classifier, and obtains the first classification result by classifying the second detection result of the vehicle with respect to the shooting direction of the vehicle by the first classifier, and / or inputs the third detection result of the traveling direction of the road sign into a second classifier, and obtains the second classification result by classifying the third detection result of the traveling direction of the road sign with respect to the traveling direction of the road sign by the second classifier. The first classifier and the second classifier are the same, or the first classifier and the second classifier are different, and the first classifier and the second classifier are trained by a deep learning method. A road recognition device.

6. An electronic device including the road recognition device according to claim 1.

7. A road recognition method, Input the image acquired by the in-vehicle imaging device into a first neural network, and the first neural network outputs the road lines in the image, thereby detecting the road lines in the image and obtaining a first detection result of the road lines; Input the image acquired by the in-vehicle imaging device into a second neural network, and the second neural network outputs the traveling directions of the vehicles and / or road signs in the image, thereby detecting the traveling directions of the vehicles and / or road signs in the image and obtaining a second detection result of the vehicles and / or a third detection result of the traveling directions of the road signs; Perform classification on the shooting direction of the vehicle in the second detection result of the vehicle to obtain a first classification result of the shooting direction of the vehicle, and / or perform classification on the traveling direction of the road sign in the third detection result of the traveling direction of the road sign to obtain a second classification result of the traveling direction of the road sign; and Recognizing a forward lane and / or a reverse lane in the image based on at least one of the first classification result of the shooting direction of the vehicle and the second classification result of the traveling direction of the road sign and the first detection result of the road lines, Recognizing a forward lane and / or a reverse lane in the image based on at least one of the first classification result of the shooting direction of the vehicle and the second classification result of the traveling direction of the road sign and the first detection result of the road lines is When there is a vehicle with a forward shooting direction or a traveling direction of a reverse road sign in one lane obtained from the first detection result of the road lines, determining the lane as a reverse lane; and / or when there is a vehicle with a rear shooting direction or a traveling direction of a forward road sign in one lane obtained from the first detection result of the road lines, determining the lane as a forward lane, a road recognition method.

Images