Road centerline recognition device, recognition method, and electronic device

Abstract

To provide a road center line recognition device, a recognition method and electronic equipment.SOLUTION: A method includes the steps of: detecting objects from images acquired by on-vehicle imaging devices by using a first neural network; regarding a road on a left-side traffic, determining a position of each detected object in the case where the detected object includes a vehicle in an opposite direction and / or road signs located to a right side of the vehicle where the on-vehicle imaging device is located and a road line, or regarding a road on a right-side traffic, in the case where the detected object includes a vehicle in an opposite direction and / or a road sign in an opposite direction located on a left side of the vehicle, and the road line; and determining an opposite vehicle and / or an opposite road sign with a minimum horizontal distance to the vehicle as a reference vehicle and / or a reference road direction sign on the basis of, the position of each object; and determining a road center line, on the basis of a distance and a positional relation between the road line and the reference vehicle and / or the reference road direction sign in the detected object.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to the field of information technology. [Background technology]

[0002] Recognizing different types of roads is a useful technology that can help guide drivers to drive their vehicles in the correct lanes. The centerline of the road is a lane that divides the roadway into two parts: the forward driving area and the reverse driving area. Recognizing the centerline of the road is very important for guiding driver's actions, determining responsibility for traffic accidents, and autonomous driving technology, etc.

[0003] In the conventional technology, usually, after detecting road lines in an image, it is recognized whether or not the detected road lines are road center lines.

[0004] It should be noted that the above description of the background art is for the purpose of making the technical solutions of the present invention clear and complete, and is described for the purpose of making those skilled in the art understand. These technical solutions are described merely as the background art of the present invention, and are not well known to those skilled in the art. Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the conventional method, the detection accuracy and detection efficiency depend on the detection of road lines, and reliable road line detection results may not be obtained, so the recognition accuracy and recognition efficiency of the road center line are low. In addition, the model for road line detection is generally complicated, which makes the overall structure of the detection device complicated.

[0006] In order to solve at least one of the above problems, embodiments of the present invention provide a road centerline recognition device, a recognition method, and an electronic device that can quickly and accurately recognize a road centerline with a simple device structure, thereby providing useful information for various possible applications. [Means for solving the problem]

[0007] In a first aspect of an embodiment of the present invention, there is provided a road centerline recognition device including: a first detection unit that detects objects from an image acquired by an on-board photographing device using a first neural network; a first determination unit that determines a position of each detected object when the detected objects include an opposite-direction vehicle and / or an opposite-direction road direction sign located on the right side of the vehicle on which the on-board photographing device is located, and a road line, for a left-hand traffic road, or when the detected objects include an opposite-direction vehicle and / or an opposite-direction road direction sign located on the left side of the vehicle on which the on-board photographing device is located, and a road line, for a right-hand traffic road; a second determination unit that determines the opposite-direction vehicle and / or the opposite-direction road direction sign that has the smallest horizontal distance from the vehicle on which the on-board photographing device is located as a reference vehicle and / or a reference road direction sign based on the position of each object; and a third determination unit that determines a road centerline among the road lines at the detected object based on the distance and positional relationship between the road line at the detected object and the reference vehicle and / or the reference road direction sign.

[0008] In a second aspect of the present invention, there is provided an electronic device including an apparatus according to the first aspect of the present invention.

[0009] In a third aspect of the embodiment of the present invention, there is provided a method for recognizing a road centerline, comprising the steps of: detecting an object using a first neural network from an image acquired by an on-vehicle photographing device; if the detected object includes an opposite-direction vehicle and / or an opposite-direction road direction sign located on the right side of a vehicle on which the on-vehicle photographing device is located, and a road line, for a road with left-hand traffic, or if the detected object includes an opposite-direction vehicle and / or an opposite-direction road direction sign located on the left side of a vehicle on which the on-vehicle photographing device is located, and a road line, for a road with right-hand traffic, determining a position of each detected object; determining, based on the position of each object, the opposite-direction vehicle and / or the opposite-direction road direction sign that has the smallest horizontal distance from the vehicle on which the on-vehicle photographing device is located as a reference vehicle and / or a reference road direction sign; and determining a road centerline among the road lines at the detected object based on a distance and positional relationship between the road line at the detected object and the reference vehicle and / or the reference road direction sign.

[0010] The advantageous effects of the embodiment of the present invention are as follows: by using the first neural network to directly detect each object in the image, determining a reference vehicle and / or a reference road direction marking based on the road lines in the object and the opposite direction vehicle and / or the opposite direction road direction marking, and determining a road centerline based on the distance and positional relationship between the detected road lines and the reference vehicle and / or the reference road direction marking, the road centerline can be quickly and accurately recognized with a simple device structure, thereby providing useful information for various possible applications.

[0011] Certain embodiments of the invention are disclosed in detail, as set forth in the following description and in the drawings, which show how the principles of the invention may be employed, but are not intended to be limited in scope, and all such modifications, alterations, and equivalents are within the spirit and content of the appended claims.

[0012] Features described and / or shown in one embodiment may be used in the same or similar manner in one or more other embodiments, may be combined with features in the other embodiments, or may substitute for features in the other embodiments.

[0013] It should be noted that the term "comprising" when used in the present text means the presence of a feature, element, step or component and does not exclude the presence or addition of one or more other features, elements, steps or components. [Brief description of the drawings]

[0014] The drawings included herein are for understanding the embodiments of the present invention, constitute a part of this specification, and are for illustrating the embodiments of the present invention, and together with the description of the text, explain the principles of the present invention. Note that the drawings described herein are merely for illustrating the embodiments of the present invention, and those skilled in the art can easily obtain other drawings based on these drawings. [Figure 1] 1 is a schematic diagram of a road centerline recognition device according to a first embodiment of the present invention; [Diagram 2] FIG. 2 is a schematic diagram of a detection result of an object according to the first embodiment of the present invention. [Diagram 3] FIG. 2 is a schematic diagram of a second determination unit according to the first embodiment of the present invention; [Figure 4] FIG. 4 is a schematic diagram of a third determination unit according to the first embodiment of the present invention; [Diagram 5] FIG. 11 is a schematic diagram of a sixth decision unit according to the first embodiment of the present invention. [Figure 6] FIG. 3 is a schematic diagram of a further process for FIG. 2. [Figure 7] FIG. 4 is another schematic diagram of the detection result of an object according to the first embodiment of the present invention. [Figure 8] FIG. 11 is a schematic diagram of an electronic device according to a second embodiment of the present invention. [Figure 9] FIG. 11 is a schematic block diagram of a system configuration of an electronic device according to a second embodiment of the present invention. [Figure 10]FIG. 11 is a schematic diagram of a road centerline recognition method according to a third embodiment of the present invention; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] The above and other features of the present invention can be understood from the drawings and the following description. The specification and drawings disclose certain embodiments of the present invention, i.e., some embodiments according to the principles of the present invention. It is to be understood that the present invention is not limited to the described embodiments, and the present invention includes all modifications, variations, and equivalents within the scope of the appended claims.

[0016] <Example 1> An embodiment of the present invention provides a road centerline recognition device. Fig. 1 is a schematic diagram of a road centerline recognition device according to a first embodiment of the present invention.

[0017] As shown in FIG. 1, the road centerline recognition device 100 includes a first detection unit 101, a first determination unit 102, a second determination unit 103, and a third determination unit 104.

[0018] The first detection unit 101 detects an object from an image captured by an in-vehicle imaging device using a first neural network.

[0019] The first determination unit 102 determines the position of each detected object when, for a road with left-hand traffic, the detected object includes an opposite-direction vehicle and / or an opposite-direction road direction sign located on the right side of the vehicle on which the on-board camera is located, as well as a road line, or when, for a road with right-hand traffic, the detected object includes an opposite-direction vehicle and / or an opposite-direction road direction sign located on the left side of the vehicle on which the on-board camera is located, as well as a road line.

[0020] The second determination unit 103 determines, based on the position of each object, the opposite direction vehicle and / or the opposite direction road direction sign that has the smallest horizontal distance from the vehicle on which the vehicle-mounted imaging device is located, as the reference vehicle and / or the reference road direction sign.

[0021] The third determination unit 104 determines a road centerline among the road lines in the detected object, based on the distance and positional relationship between the road lines in the detected object and the reference vehicle and / or the reference road direction marking.

[0022] In an embodiment of the present invention, the road centerline is a lane that divides the vehicle driving road into a forward driving area and an opposite direction driving area. In other words, a vehicle driving in the forward driving area is a forward vehicle, and a vehicle driving in the opposite direction driving area is a opposite direction vehicle, which is a vehicle that drives in the opposite direction to the forward vehicle.

[0023] In an embodiment of the present invention, the vehicle in which the on-board camera is located is also referred to as the current vehicle, a vehicle whose traveling direction is the same as that of the vehicle in which the on-board camera is located is a forward direction vehicle, and a vehicle whose traveling direction is opposite to that of the vehicle in which the on-board camera is located is an opposite direction vehicle.

[0024] In the embodiment of the present invention, the detection target of the road centerline recognition device 100 is an image captured by an on-vehicle photographing device. After the on-vehicle photographing device captures an image, it transmits the image to the road centerline recognition device 100 for recognition.

[0025] In the embodiment of the present invention, the on-board camera may be various types of camera mounted on a vehicle. For example, the on-board camera is a car camera, which records each frame of the video taken from the driving perspective. The road centerline recognition device 100 may process each frame one by one, and update the previous recognition result according to the recognition result of the current frame.

[0026] The first detection unit 101 detects an object from an image captured by an in-vehicle imaging device using a first neural network.

[0027] For example, the first neural network is obtained by being trained by a deep learning method.

[0028] For example, the first neural network is a feature pyramid network (FPN). The feature pyramid network is a neural network that has high performance and is suitable for detecting multiple scenes and targets, and the use of the network can provide high object detection accuracy.

[0029] In an embodiment of the present invention, the objects detected by the first neural network include vehicles, road directional markings, and road lines. In other words, the vehicles, road directional markings, and road lines in the image are all detected as objects, and there is no need to detect the road lines separately.

[0030] Additionally, the objects detected by the first neural network may include non-motor vehicles and pedestrians.

[0031] In an embodiment of the present invention, the vehicles in the object may include a forward vehicle and an opposite direction vehicle. The road direction markings in the object include a forward road direction marking and an opposite direction road direction marking, where the forward road direction marking is a road direction marking installed in the forward direction driving area, and the opposite direction road direction marking is a road direction marking installed in the opposite direction driving area. The road lines in the object may include various types of road lines, such as road center lines, roadway outside lines, and sidewalk road lines.

[0032] In one aspect of the embodiment of the present invention, the detection result of the object output by the first detection unit 101 may be represented by a detection frame of each object.

[0033] Fig. 2 is a schematic diagram of an object detection result in Example 1 of the present invention. As shown in Fig. 2, for one frame image captured by the vehicle-mounted imaging device, the detected objects include road lines 201, 202, 204, and vehicles 203, 205, and further include a person 206 and a bicycle 207, and Fig. 2 shows the detection frames of these objects.

[0034] In an embodiment of the present invention, the first determination unit 102 determines the position of each detected object when, for a left-hand traffic road, the detected object includes an opposite-direction vehicle and / or an opposite-direction road direction sign located on the right side of the vehicle on which the on-board camera is located, as well as a road line, or when, for a right-hand traffic road, the detected object includes an opposite-direction vehicle and / or an opposite-direction road direction sign located on the left side of the vehicle on which the on-board camera is located, as well as a road line.

[0035] In an embodiment of the present invention, since the first determination unit 102 makes a determination based on whether the detected objects include an opposite direction vehicle and / or an opposite direction road direction marking, for example, the road centerline recognition device 100 may further include a second detection unit 105 and a third detection unit 106.

[0036] The second detection unit 105 uses a second neural network to determine the vehicle's traveling direction at the detected object.

[0037] The third detection unit 106 uses a third neural network to determine the direction of the road direction markings on the detected object.

[0038] In one aspect of the embodiment of the present invention, the second neural network determines the direction of the vehicle based on the displayed direction of the vehicle, and the output result of the second neural network may include two types of results, namely, "vehicle in the forward direction" and "vehicle in the opposite direction".

[0039] In the embodiment of the present invention, the display direction of the vehicle refers to the direction displayed by the detected vehicle in the image, or the part of the vehicle displayed by the detected vehicle. For example, if the front, right front or left front of the vehicle is displayed in the image, the display direction of the vehicle is "forward" and the running direction of the vehicle is the opposite direction. In other words, if the running direction of the vehicle in which the on-board camera is located is the forward direction, the vehicle whose display direction is "forward" in the image captured by the on-board camera is the vehicle in the opposite running direction of the vehicle in which the on-board camera is located, that is, the opposite direction vehicle.

[0040] In an embodiment of the present invention, the direction of the road direction marking refers to the orientation of the detected road direction marking in the image. For example, a road direction marking including an arrow pointing in the forward direction is referred to as a forward road direction marking, i.e., a road direction marking installed in the forward driving area. A road direction marking including an arrow pointing in the opposite direction is referred to as an opposite direction road direction marking.

[0041] In one embodiment of the present invention, the direction of travel refers to the driving direction of the vehicle in which the on-board camera is located, i.e., the forward driving direction, and the opposite direction refers to the direction opposite to the driving direction of the vehicle in which the on-board camera is located, i.e., the reverse driving direction.

[0042] In one aspect of the embodiment of the present invention, the results output by the third neural network may include two types of results, namely "headway road direction sign" and "opposite road direction sign".

[0043] For roads with left-hand traffic, if the detected objects include an opposite-direction vehicle and / or an opposite-direction road direction sign located on the right side of the vehicle on which the on-board camera is located, as well as a road line, or for roads with right-hand traffic, if the detected objects include an opposite-direction vehicle and / or an opposite-direction road direction sign located on the left side of the vehicle on which the on-board camera is located, as well as a road line, the first determination unit 102 determines the position of each detected object.

[0044] For example, the road in the image shown in FIG. 2 belongs to a left-hand traffic road. The detected objects include an opposite direction vehicle 203 located on the right side of the vehicle on which the on-board image capture device is located, and road lines 201 and 202 located on the right side of the vehicle on which the on-board image capture device is located, and the first determination unit 102 determines the positions of these objects based on the detection frame, and sets the position of the object to, for example, the center position of the bottom side of the detection frame. In addition, since the distance between the vehicle 205 in FIG. 2 and the vehicle on which the on-board image capture device is located is long and the detection frame is small, it is not detected as an "opposite direction vehicle" and there is no vehicle that is selected as a candidate for the reference vehicle. If the vehicle 205 is also detected as belonging to the "opposite direction vehicle", the vehicle 205 may also be selected as a candidate for the reference vehicle.

[0045] In an embodiment of the present invention, the origin of the coordinates of an image captured by an on-board imaging device is the vertex in the upper left corner of the image, and the coordinates of each point in the image are expressed in terms of the number of pixels, starting from the upper left corner of the image, with the horizontal coordinate gradually increasing toward the right and the vertical coordinate gradually increasing toward the bottom.

[0046] In the embodiment of the present invention, the second determination unit 103 determines the opposite direction vehicle and / or the opposite direction road direction sign having the smallest horizontal distance from the vehicle where the vehicle-mounted image capturing device is located as the reference vehicle and / or the reference road direction sign.

[0047] For example, when either an opposite direction vehicle or an opposite direction road direction sign having the smallest horizontal distance from the vehicle where the on-board camera is located is detected in the image, the second determination unit 103 determines the detected opposite direction vehicle or opposite direction road direction sign as a reference vehicle or reference road direction sign. When both an opposite direction vehicle and an opposite direction road direction sign having the smallest horizontal distance from the vehicle where the on-board camera is located are detected in the image, the second determination unit 103 determines the opposite direction vehicle and the opposite direction road direction sign having the smallest horizontal distance from the vehicle where the on-board camera is located as a reference vehicle and reference road direction sign.

[0048] 3 is a schematic diagram of the second determination unit 103 according to the first embodiment of the present invention. As shown in FIG. 3, the second determination unit 103 includes a fourth determination unit 301 and a fifth determination unit 302.

[0049] For a road where traffic is on the left side, the fourth determination unit 301 determines the leftmost opposite direction vehicle and / or opposite direction road direction sign as a reference vehicle and / or a reference road direction sign.

[0050] For a road with right-hand traffic, the fifth determination unit 302 determines the rightmost vehicle going in the opposite direction and / or the road direction sign going in the opposite direction as a reference vehicle and / or a reference road direction sign.

[0051] For example, as shown in FIG. 2, the leftmost vehicle 203 going in the opposite direction is taken as the reference vehicle.

[0052] In an embodiment of the present invention, the third determination unit 104 determines a road centerline among the road lines in the detected object based on the distance and positional relationship between the road lines in the detected object and a reference vehicle and / or a reference road direction marking.

[0053] The following will be described separately for two cases: a case where either a reference vehicle or a reference road direction marking is detected, and a case where both a reference vehicle and a reference road direction marking are detected.

[0054] 4 is a schematic diagram of the third determination unit 104 according to the first embodiment of the present invention. As shown in FIG.

[0055] When either a reference vehicle or a reference road direction marking is detected, the sixth determination unit 401 determines, as the road center line, the road line which has the smallest horizontal distance from the reference vehicle or the reference road direction marking and is located on the vehicle side on which the on-board imaging device is located with respect to the reference vehicle or the reference road direction marking.

[0056] 5 is a schematic diagram of the sixth determination unit 401 according to the first embodiment of the present invention. As shown in FIG. 5, the sixth determination unit 401 includes a seventh determination unit 501, a drawing unit 502, an eighth determination unit 503, a first calculation unit 504 and a ninth determination unit 505.

[0057] The seventh determination unit 501 determines the center of the bottom of the detection frame of the reference vehicle or the reference road direction marking as the vehicle reference point or the road marking reference point.

[0058] The drawing unit 502 draws each road line based on the diagonal lines of each detected road line detection frame.

[0059] An eighth determination unit 503 determines the position of the road line reference point based on the vertices of each road line detection frame.

[0060] The first calculation unit 504 calculates the difference in horizontal coordinate between the road line reference point and the vehicle reference point or road marking reference point.

[0061] The ninth determination unit 505 determines, for roads with left-hand traffic, the road line where the difference is smaller than zero and the absolute value of the difference is the smallest as the road centerline, and for roads with right-hand traffic, determines, for roads with right-hand traffic, the road line where the difference is larger than zero and the absolute value of the difference is the smallest as the road centerline.

[0062] In one aspect of the embodiment of the present invention, the drawing unit 502 determines the diagonal lines of the road line detection frame as the road lines. For example, for a road with left-hand traffic, the diagonal line connecting the vertex at the upper left corner and the vertex at the lower right corner of the road line detection frame is determined as the road line, and for a road with right-hand traffic, the diagonal line connecting the vertex at the upper right corner and the vertex at the lower left corner of the road line detection frame is determined as the road line.

[0063] In one aspect of the embodiment of the present invention, the eighth determination unit 503 determines the position of the road line reference point according to the following formula (1).

number

[0064] where x irepresents the abscissa of the road line reference point, y represents the ordinate of the road line reference point, and (x min ,y min ) represents the coordinates of the top left corner of the road line detection frame, and (x max ,y max ) represents the coordinate of the lower right corner vertex of the road line detection frame, and i represents the road line number, where i is a positive integer.

[0065] In an embodiment of the present invention, the ordinate y of the road line reference point is equal to the ordinate of the vehicle reference point or the road marking reference point.

[0066] As shown in FIG. 5, the sixth determination unit 401 may further include an extension unit 506 .

[0067] If the horizontal coordinate of the vehicle reference point or the road marking reference point is not included in the horizontal coordinate range of the drawn road line, the extension unit 506 extends the drawn road line so that the horizontal coordinate range of the extended road line includes the horizontal coordinate of the vehicle reference point or the road marking reference point.

[0068] The object shown in FIG. 2 will be described as an example. FIG. 6 is a schematic diagram of one of the further processes for FIG. 2. As shown in FIG. 6, the center of the bottom side of the detection frame of the reference vehicle 203 is set as the vehicle reference point 2031, the diagonal line of the detection frame connecting the road lines 201 and 202 is set as the drawn road line, and the road line 202 is extended so that the horizontal coordinate of the vehicle reference point 2031 is included in the range of the horizontal coordinate of the extended road line. Based on the vertices of the detection frames of each road line 201 and 202, the road line reference points 2011 and 2021 are determined, and the difference in horizontal coordinate between the road line reference points 2011 and 2021 and the vehicle reference point 2031 is calculated. If the road shown in FIG. 2 is a left-hand traffic road, the road line 201 where the road line reference point 2011, whose difference in horizontal coordinate with the vehicle reference point 2031 is smaller than 0 and whose absolute value of the difference is the smallest, is determined as the road center line.

[0069] In one aspect of the embodiment of the present invention, when both a reference vehicle and a reference road direction marking are detected, as shown in FIG. 4, the third determination unit 104 may further include a tenth determination unit 402 and an eleventh determination unit 403.

[0070] When both the reference vehicle and the reference road direction marking are detected, the tenth determination unit 402 determines the road line which has the smallest horizontal distance from the reference vehicle and is located on the vehicle side on which the vehicle-mounted photographing device is located relative to the reference vehicle as the first candidate road centerline, and determines the road line which has the smallest horizontal distance from the reference road direction marking and is located on the vehicle side on which the vehicle-mounted photographing device is located relative to the reference road direction marking as the second candidate road centerline.

[0071] An eleventh determination unit 403 determines, as the road centerline, one of the first candidate road centerline and the second candidate road centerline, which has a smaller horizontal distance from the vehicle on which the vehicle-mounted image capturing device is located.

[0072] For example, FIG. 7 is another schematic diagram of the object detection result of the embodiment 1 of the present invention. As shown in FIG. 7, the detected object includes road lines 701 and 702 on the right side of the vehicle where the on-vehicle photographing device is located, a reference vehicle 703, and a reference road direction marking 704. The road line 702, which has the smallest horizontal distance from the reference vehicle 703 and is located on the left side of the reference vehicle 703, is determined as the first candidate road center line, and the road line 701, which has the smallest horizontal distance from the reference road direction marking 704 and is located on the left side of the reference road direction marking 704, is determined as the second candidate road center line. Finally, of the first candidate road center line 702 and the second candidate road center line 701, the one having the smaller horizontal distance from the vehicle where the on-vehicle photographing device is located, that is, the second candidate road center line 701, is determined as the road center line.

[0073] In an embodiment of the present invention, the first candidate road centerline and the second candidate road centerline determined by the tenth determination unit 402 may be the same road line, so in this case, the road line is directly determined as the road centerline.

[0074] According to this embodiment, the first neural network is used to directly detect each object in the image, a reference vehicle and / or a reference road direction marking is determined based on the road lines in the object and the opposite direction vehicle and / or the opposite direction road direction marking, and the road centerline is determined based on the distance and positional relationship between the detected road lines and the reference vehicle and / or the reference road direction marking, thereby making it possible to quickly and accurately recognize the road centerline with a simple device structure, and to provide useful information for various possible applications.

[0075] <Example 2> The embodiment of the present invention further provides an electronic device, and Fig. 8 is a schematic diagram of an electronic device according to the second embodiment of the present invention. As shown in Fig. 8, the electronic device 800 includes a road centerline recognition device 801, and the configuration and function of the road centerline recognition device 801 are the same as those described in the first embodiment, and the description thereof will be omitted here.

[0076] In this embodiment, the electronic device 800 may be various electronic devices, such as an in-vehicle terminal, a mobile terminal, or a computer.

[0077] Fig. 9 is a schematic block diagram of a system configuration of an electronic device according to a second embodiment of the present invention. As shown in Fig. 9, an electronic device 900 may include a processor 901 and a memory 902, and the memory 902 is connected to the processor 901. The diagram is merely exemplary, and other types of configurations may be used to supplement or replace the configurations to realize electrical communication functions or other functions.

[0078] As shown in FIG. 9, the electronic device 900 may further include an input unit 903, a display 904, and a power source 905.

[0079] In one embodiment, the functionality of the road centerline recognition device of the first embodiment may be integrated into the processor 901. Here, the processor 901 may be configured to execute the steps of: detecting an object using a first neural network from an image acquired by the on-board photographing device; determining a position of each detected object when the detected object includes an opposite-direction vehicle and / or an opposite-direction road direction sign located on the right side of the vehicle on which the on-board photographing device is located, and a road line, for a road with left-hand traffic, or when the detected object includes an opposite-direction vehicle and / or an opposite-direction road direction sign located on the left side of the vehicle on which the on-board photographing device is located, and a road line, based on the position of each object; determining the opposite-direction vehicle and / or the opposite-direction road direction sign having the smallest horizontal distance from the vehicle on which the on-board photographing device is located as a reference vehicle and / or a reference road direction sign; and determining a road centerline among the road lines at the detected object based on the distance and positional relationship between the road line at the detected object and the reference vehicle and / or the reference road direction sign.

[0080] For example, the first neural network is a feature pyramid network.

[0081] For example, the processor 901 may be further configured to perform the steps of determining, using a second neural network, a vehicle driving direction at the detected object, and determining, using a third neural network, a direction of a road direction marking at the detected object.

[0082] For example, the step of determining, based on the position of each object, the opposite direction vehicle and / or the opposite direction road direction sign which is the smallest horizontal distance from the vehicle on which the on-board camera is located as the reference vehicle and / or the reference road direction sign includes a step of determining, for a road with left-hand traffic, the leftmost opposite direction vehicle and / or the opposite direction road direction sign as the reference vehicle and / or the reference road direction sign, and a step of determining, for a road with right-hand traffic, the rightmost opposite direction vehicle and / or the opposite direction road direction sign as the reference vehicle and / or the reference road direction sign.

[0083] For example, the step of determining a road centerline among the road lines in the detected object based on the distance and positional relationship between the road lines in the detected object and the reference vehicle and / or the reference road direction marking includes a step of determining, when either the reference vehicle or the reference road direction marking is detected, as the road centerline, the road line which has the smallest horizontal distance to the reference vehicle or the reference road direction marking and is located on the vehicle side on which the on-board imaging device is located with respect to the reference vehicle or the reference road direction marking.

[0084] For example, the step of determining as the road centerline the road line which has the smallest horizontal distance from the reference vehicle or the reference road direction marking and is located on the vehicle side of the reference vehicle or the reference road direction marking on which the on-board imaging device is located includes the steps of setting the center of the bottom of the detection frame of the reference vehicle or the reference road direction marking as the vehicle reference point or road marking reference point, drawing each road line based on the diagonal lines of each detected road line detection frame, determining the position of the road line reference point based on the vertices of each road line detection frame, calculating the difference in horizontal coordinates between the road line reference point and the vehicle reference point or road marking reference point, and determining as the road centerline the road line where the difference is smaller than zero and the absolute value of the difference is smallest for left-hand traffic roads, and determining as the road centerline the road line where the difference is greater than zero and the absolute value of the difference is smallest for right-hand traffic roads.

[0085] For example, the step of determining as the road center line the road line which has the smallest horizontal distance from the reference vehicle or the reference road direction marking and is located on the vehicle side of the reference vehicle or the reference road direction marking on which the on-board imaging device is located, further includes a step of extending the drawn road line so that the horizontal coordinate range of the extended road line includes the horizontal coordinate of the vehicle reference point or the road marking reference point if the horizontal coordinate of the vehicle reference point or the road marking reference point is not included in the horizontal coordinate range of the drawn road line.

[0086] For example, the step of determining a road centerline among the road lines in the detected object based on the distance and positional relationship between the road lines in the detected object and the reference vehicle and / or the reference road direction marking includes, when both the reference vehicle and the reference road direction marking are detected, a step of determining, as a first candidate road centerline, the road line which has the smallest horizontal distance from the reference vehicle and is located on the side of the vehicle on which the on-vehicle imaging device is located with respect to the reference vehicle, and determining, as a second candidate road centerline, the road line which has the smallest horizontal distance from the reference road direction marking and is located on the side of the vehicle on which the on-vehicle imaging device is located with respect to the reference road direction marking; and a step of determining, of the first candidate road centerline and the second candidate road centerline, the one which has the smaller horizontal distance from the vehicle on which the on-vehicle imaging device is located with respect to the reference road direction marking, as the road centerline.

[0087] In another aspect of the embodiment of the present invention, the road centerline recognition device of embodiment 1 may be respectively arranged with the processor 901, for example, the road centerline recognition device may be a chip connected to the processor 901, and may be configured to realize the functions of the road centerline recognition device under the control of the processor 901.

[0088] The electronic device 900 in this embodiment does not need to include all of the components shown in FIG.

[0089] As shown in FIG. 9, processor 901, also referred to as a controller or operation control unit, may include a microprocessor or other processing device and / or logic device, and processor 901 accepts input and controls the operation of each part of electronic device 900.

[0090] The memory 902 may be, for example, one or more of a buffer, a flash memory, a hard disk, a removable medium, a volatile memory, a non-volatile memory, or other suitable devices. The processor 901 may execute a program stored in the memory 902 to realize information storage or processing, etc. The other members are similar to the prior art, and therefore the description thereof is omitted here. Each part of the electronic device 700 may be realized by dedicated hardware, firmware, software, or a combination thereof without departing from the scope of the present invention.

[0091] According to this embodiment, the first neural network is used to directly detect each object in the image, a reference vehicle and / or a reference road direction marking is determined based on the road lines in the object and the opposite direction vehicle and / or the opposite direction road direction marking, and the road centerline is determined based on the distance and positional relationship between the detected road lines and the reference vehicle and / or the reference road direction marking, thereby making it possible to quickly and accurately recognize the road centerline with a simple device structure, and to provide useful information for various possible applications.

[0092] <Example 3> The embodiment of the present invention further provides a road centerline recognition method corresponding to the road centerline recognition device of embodiment 1. Figure 10 is a schematic diagram of a road centerline recognition method of embodiment 3 of the present invention. As shown in Figure 10, the method includes the following steps:

[0093] Step 1001: An object is detected from an image captured by an on-vehicle imaging device using a first neural network.

[0094] Step 1002: For roads with left-hand traffic, if the detected objects include an opposite-direction vehicle and / or an opposite-direction road direction sign located on the right side of the vehicle on which the on-board camera is located, as well as a road line, or, for roads with right-hand traffic, if the detected objects include an opposite-direction vehicle and / or an opposite-direction road direction sign located on the left side of the vehicle on which the on-board camera is located, as well as a road line, determine the position of each detected object.

[0095] Step 1003: Based on the position of each object, the opposite direction vehicle and / or the opposite direction road direction sign having the smallest horizontal distance from the vehicle on which the vehicle-mounted camera is located is determined as a reference vehicle and / or a reference road direction sign.

[0096] Step 1004: Based on the distance and positional relationship between the road lines in the detected object and the reference vehicle and / or the reference road direction marking, a road centerline of the road lines in the detected object is determined.

[0097] In one aspect of the embodiment of the present invention, the specific implementation methods of each of the above steps are the same as those described in the first embodiment, and the description thereof is omitted here.

[0098] According to this embodiment, the first neural network is used to directly detect each object in the image, a reference vehicle and / or a reference road direction marking is determined based on the road lines in the object and the opposite direction vehicle and / or the opposite direction road direction marking, and the road centerline is determined based on the distance and positional relationship between the detected road lines and the reference vehicle and / or the reference road direction marking, thereby making it possible to quickly and accurately recognize the road centerline with a simple device structure, and to provide useful information for various possible applications.

[0099] An embodiment of the present invention further provides a computer-readable program that, when executed in a road centerline recognition device or electronic device, causes a computer to execute the road centerline recognition method described in Example 3 above in the road centerline recognition device or electronic device.

[0100] An embodiment of the present invention further provides a storage medium storing a computer-readable program for causing a computer to execute the road centerline recognition method described in the above embodiment 3 in a road centerline recognition device or electronic device.

[0101] The road centerline recognition device or the road centerline recognition method executed in the electronic device described with reference to the embodiments of the present invention may be implemented in hardware, a software module executed by a processor, or a combination of both. For example, one or more of the functional block diagrams shown in FIG. 1, or one or more combinations of the functional block diagrams, may correspond to each software module of a computer program flow or each hardware module. These software modules may correspond to each step shown in FIG. 10. These hardware modules may be realized by implementing these software modules in hardware using, for example, a field programmable gate array (FPGA).

[0102] The software module may be located in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a mobile hard disk, a CD-ROM, or any other form of storage medium known to those skilled in the art. The storage medium may be connected to the processor so that the processor reads information from the storage medium and writes information to the storage medium, or the storage medium may be a component of the processor. The processor and the storage medium are located in an ASIC. The software module may be stored in the memory of the mobile terminal, or in a memory card inserted in the mobile terminal. For example, if the device (e.g., a mobile terminal) uses a MEGA-SIM card or a large capacity flash memory device with a relatively large capacity, the software module may be stored in the MEGA-SIM card or the large capacity flash memory device.

[0103] One or more of the functional blocks and / or one or more combinations of functional blocks depicted in Figure 1 may be implemented in a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, or any suitable combination thereof to perform the functions described herein. One or more of the functional blocks and / or one or more combinations of functional blocks depicted in Figure 1 may be implemented in, for example, a combination of computing devices, such as a combination of a DSP and a microprocessor, a combination of multiple microprocessors, one or more microprocessors in combination with a DSP communication, or any other configuration.

[0104] Although the present invention has been described above with reference to specific embodiments, the above description is merely illustrative and does not limit the scope of protection of the present invention. Various modifications and changes may be made to the present invention without departing from the spirit and principles of the present invention, and these modifications and changes also fall within the scope of the present invention.

[0105] Furthermore, the following supplementary notes are disclosed regarding the embodiments including the above-mentioned examples. (Appendix 1) A road centerline recognition device, comprising: a first detection unit that detects an object from an image acquired by an in-vehicle image capturing device using a first neural network; a first determination unit that determines the position of each detected object when the detected object includes a vehicle in the opposite direction located on the right side of the vehicle on which the on-board camera is located and / or a road direction sign in the opposite direction, and a road line, on a road with left-hand traffic, or when the detected object includes a vehicle in the opposite direction located on the left side of the vehicle on which the on-board camera is located and / or a road direction sign in the opposite direction, and a road line, on a road with right-hand traffic; a second determination unit that determines, based on the position of each object, the opposite direction vehicle and / or the opposite direction road direction sign that has the smallest horizontal distance from the vehicle where the vehicle-mounted photographing device is located as a reference vehicle and / or a reference road direction sign; and a third determination unit that determines a road center line among the road lines in the detected object based on a distance and positional relationship between the road lines in the detected object and the reference vehicle and / or the reference road direction marking. (Appendix 2) 2. The apparatus of claim 1, wherein the first neural network is a feature pyramid network. (Appendix 3) a second detection unit that determines a traveling direction of the vehicle at the detected object using a second neural network; and a third detector configured to determine a direction of road direction markings on the detected object using a third neural network. (Appendix 4) The second determination unit, a fourth determination unit that determines, for a road with left-hand traffic, the leftmost opposite direction vehicle and / or the opposite direction road direction sign as a reference vehicle and / or a reference road direction sign; and a fifth determination unit that, for a road with right-hand traffic, determines the rightmost opposite direction vehicle and / or the opposite direction road direction sign as a reference vehicle and / or a reference road direction sign. (Appendix 5) The third determination unit, and a sixth determination unit that, when either a reference vehicle or a reference road direction sign is detected, determines as the road center line the road line that has the smallest horizontal distance from the reference vehicle or the reference road direction sign and is located on the vehicle side on which the on-board photography device is located with respect to the reference vehicle or the reference road direction sign. (Appendix 6) The sixth determination unit, a seventh determination unit that determines the center of a bottom side of a detection frame of the reference vehicle or the reference road direction marking as a vehicle reference point or a road marking reference point; a drawing unit that draws each road line based on a diagonal line of each detected road line detection frame; an eighth determination unit that determines a position of a road line reference point based on a vertex of each road line detection frame; a first calculation unit that calculates a difference in horizontal coordinate between the road line reference point and the vehicle reference point or the road marking reference point; and a ninth determination unit that determines, for roads with left-hand traffic, a road line where the difference is less than zero and the absolute value of the difference is the smallest as a road centerline, and that determines, for roads with right-hand traffic, a road line where the difference is greater than zero and the absolute value of the difference is the smallest as a road centerline. (Appendix 7) The sixth determination unit, The device described in Appendix 6, further comprising an extension unit that, if the horizontal coordinate of the vehicle reference point or the road marking reference point is not included in the horizontal coordinate range of the drawn road line, extends the drawn road line so that the horizontal coordinate of the vehicle reference point or the road marking reference point is included in the horizontal coordinate range of the extended road line. (Appendix 8) The third determination unit, a tenth determination unit that, when both the reference vehicle and the reference road direction marking are detected, determines a road line that has a smallest horizontal distance from the reference vehicle and is located on the vehicle side where the vehicle-mounted photographing device is located with respect to the reference vehicle as a first candidate road center line, and determines a road line that has a smallest horizontal distance from the reference road direction marking and is located on the vehicle side where the vehicle-mounted photographing device is located with respect to the reference road direction marking as a second candidate road center line; and an eleventh determination unit that determines, as the road centerline, the one of the first candidate road centerline and the second candidate road centerline which has a shorter horizontal distance from the vehicle on which the on-board photographing device is located. (Appendix 9) 2. An electronic device comprising the device according to claim 1. (Appendix 10) A method for recognizing a road centerline, comprising the steps of: Detecting an object from an image captured by an on-vehicle imaging device using a first neural network; determining the position of each detected object when the detected objects include a vehicle in the opposite direction and / or a road direction sign in the opposite direction located on the right side of the vehicle on which the on-board camera is located, and / or a road line on the road on which the on-board camera is located, or when the detected objects include a vehicle in the opposite direction and / or a road direction sign in the opposite direction located on the left side of the vehicle on which the on-board camera is located, and / or a road line on the road on which the on-board camera is located; determining, based on the position of each object, the opposite direction vehicle and / or the opposite direction road direction sign that has the smallest horizontal distance from the vehicle where the vehicle-mounted photographing device is located, as a reference vehicle and / or a reference road direction sign; and determining a road centerline among the road lines in the detected object based on the distance and positional relationship between the road lines in the detected object and the reference vehicle and / or the reference road direction marking. (Appendix 11) 11. The method of claim 10, wherein the first neural network is a feature pyramid network. (Appendix 12) determining a direction of travel of the vehicle at the detected object using a second neural network; 11. The method of claim 10, further comprising determining an orientation of road direction markings at the detected object using a third neural network. (Appendix 13) The step of determining, based on the position of each object, the opposite direction vehicle and / or the opposite direction road direction sign having the smallest horizontal distance from the vehicle where the vehicle-mounted photographing device is located as a reference vehicle and / or a reference road direction sign, For a road with left-hand traffic, determining the leftmost opposite direction vehicle and / or the opposite direction road direction sign as a reference vehicle and / or a reference road direction sign; and for roads with right-hand traffic, determining the rightmost opposite direction vehicle and / or opposite direction road direction marking as a reference vehicle and / or reference road direction marking. (Appendix 14) The step of determining a road center line among the road lines in the detected object based on a distance and a positional relationship between the road lines in the detected object and the reference vehicle and / or the reference road direction marking, The method described in Appendix 10, comprising a step of, when either a reference vehicle or a reference road direction marking is detected, determining as the road center line the road line which has the smallest horizontal distance from the reference vehicle or the reference road direction marking and is located on the vehicle side on which the on-board photographing device is located with respect to the reference vehicle or the reference road direction marking. (Appendix 15) The step of determining, as a road center line, a road line that has a minimum horizontal distance from the reference vehicle or the reference road direction marking and is located on the vehicle side on which the vehicle-mounted photographing device is located with respect to the reference vehicle or the reference road direction marking, determining a center of a bottom side of a detection frame of the reference vehicle or the reference road direction marking as a vehicle reference point or a road marking reference point; drawing each road line based on a diagonal of each detected road line detection frame; determining a location of a road line reference point based on the vertices of each road line detection frame; calculating the difference in horizontal coordinates between said road line reference point and said vehicle reference point or road marking reference point; determining, for left-hand traffic roads, the road line where the difference is less than zero and the absolute value of the difference is the smallest as the road centerline, and for right-hand traffic roads, determining, for right-hand traffic roads, the road line where the difference is greater than zero and the absolute value of the difference is the smallest as the road centerline. (Appendix 16) The step of determining, as a road center line, a road line that has a minimum horizontal distance from the reference vehicle or the reference road direction marking and is located on the vehicle side on which the vehicle-mounted photographing device is located with respect to the reference vehicle or the reference road direction marking, 16. The method of claim 15, further comprising the step of: if the horizontal coordinate of the vehicle reference point or the road marking reference point is not included in the range of horizontal coordinates of the drawn road line, extending the drawn road line so that the horizontal coordinate range of the extended road line includes the horizontal coordinate of the vehicle reference point or the road marking reference point. (Appendix 17) The step of determining a road center line among the road lines in the detected object based on a distance and a positional relationship between the road lines in the detected object and the reference vehicle and / or the reference road direction marking, when both the reference vehicle and the reference road direction marking are detected, determining a road line which has a smallest horizontal distance from the reference vehicle and is located on the vehicle side on which the vehicle-mounted photographing device is located with respect to the reference vehicle as a first candidate road center line, and determining a road line which has a smallest horizontal distance from the reference road direction marking and is located on the vehicle side on which the vehicle-mounted photographing device is located with respect to the reference road direction marking as a second candidate road center line; The method described in Appendix 10, comprising a step of determining, as the road centerline, one of the first candidate road centerline and the second candidate road centerline which has a smaller horizontal distance from the vehicle on which the on-board photographing device is located.

Claims

1. A road centerline recognition device, comprising: a first detection unit that detects an object from an image acquired by an in-vehicle image capturing device using a first neural network; a first determination unit that determines a position of each detected object when the detected object includes a vehicle in the opposite direction located on the right side of the vehicle on which the on-vehicle image capture device is located and / or a road direction sign in the opposite direction, and a road line, on a road with left-hand traffic, or when the detected object includes a vehicle in the opposite direction located on the left side of the vehicle on which the on-vehicle image capture device is located and / or a road direction sign in the opposite direction, and a road line, on a road with right-hand traffic; a second determination unit that determines, based on the position of each object, the opposite direction vehicle and / or the opposite direction road direction sign that has the smallest horizontal distance from the vehicle on which the vehicle-mounted image capture device is located as a reference vehicle and / or a reference road direction sign; and a third determination unit that determines a road center line among the road lines in the detected object based on a distance and positional relationship between the road lines in the detected object and the reference vehicle and / or the reference road direction marking.

2. The apparatus of claim 1 , wherein the first neural network is a feature pyramid network.

3. a second detection unit that determines a traveling direction of the vehicle at the detected object using a second neural network; 2. The apparatus of claim 1, further comprising a third detector that uses a third neural network to determine an orientation of road direction markings on the detected object.

4. The second determination unit, a fourth determination unit that determines, for a road with left-hand traffic, the leftmost opposite direction vehicle and / or the opposite direction road direction sign as a reference vehicle and / or a reference road direction sign; and a fifth determination unit that, for a road with right-hand traffic, determines the rightmost opposite direction vehicle and / or opposite direction road direction sign as a reference vehicle and / or a reference road direction sign.

5. The third determination unit, 2. The device according to claim 1, further comprising: a sixth determination unit that, when either a reference vehicle or a reference road direction marking is detected, determines as a road center line a road line that has a smallest horizontal distance from the reference vehicle or the reference road direction marking and is located on a vehicle side on which the vehicle-mounted imaging device is located with respect to the reference vehicle or the reference road direction marking.

6. The sixth determination unit, a seventh determination unit that determines the center of a bottom side of a detection frame of the reference vehicle or the reference road direction marking as a vehicle reference point or a road marking reference point; a drawing unit that draws each road line based on a diagonal line of each detected road line detection frame; an eighth determination unit that determines a position of a road line reference point based on a vertex of each road line detection frame; a first calculation unit that calculates a difference in horizontal coordinate between the road line reference point and the vehicle reference point or the road marking reference point; a ninth determination unit that determines, for left-hand traffic roads, the road line where the difference is less than zero and the absolute value of the difference is the smallest as the road centerline, and that determines, for right-hand traffic roads, the road line where the difference is greater than zero and the absolute value of the difference is the smallest as the road centerline.

7. The sixth determination unit, 7. The device of claim 6, further comprising an extension unit that, if the horizontal coordinate range of the drawn road line does not include the horizontal coordinate of the vehicle reference point or the road marking reference point, extends the drawn road line so that the horizontal coordinate range of the extended road line includes the horizontal coordinate of the vehicle reference point or the road marking reference point.

8. The third determination unit, a tenth determination unit that, when both the reference vehicle and the reference road direction marking are detected, determines a road line that has a smallest horizontal distance from the reference vehicle and is located on the vehicle side of the vehicle on which the vehicle-mounted photographing device is located with respect to the reference vehicle as a first candidate road center line, and determines a road line that has a smallest horizontal distance from the reference road direction marking and is located on the vehicle side of the vehicle on which the vehicle-mounted photographing device is located with respect to the reference road direction marking as a second candidate road center line; and an eleventh determination unit that determines, as the road centerline, one of the first candidate road centerline and the second candidate road centerline that has a shorter horizontal distance from the vehicle on which the vehicle-mounted imaging device is located.

9. 13. An electronic device comprising the device of claim 1.

10. A method for recognizing a road centerline, comprising the steps of: Detecting an object from an image captured by an on-vehicle imaging device using a first neural network; determining the position of each detected object when the detected objects include a vehicle in the opposite direction located on the right side of the vehicle on which the on-board camera is located and / or a road direction sign in the opposite direction, and a road line, for a road with left-hand traffic, or when the detected objects include a vehicle in the opposite direction located on the left side of the vehicle on which the on-board camera is located and / or a road direction sign in the opposite direction, and a road line, for a road with right-hand traffic; determining, based on the position of each object, the opposite direction vehicle and / or the opposite direction road direction sign that has the smallest horizontal distance from the vehicle on which the vehicle-mounted photographing device is located, as a reference vehicle and / or a reference road direction sign; and determining a road centerline among the road lines in the detected object based on the distance and positional relationship between the road lines in the detected object and the reference vehicle and / or the reference road direction marking.

Images