Method and device for recognizing a line in a camera image for a machine or a transmitting system, and recognition system for a machine or a transmitting system

Abstract

The approach presented here relates to a method for recognizing a line (110) in a camera image for a machine (100). The method comprises a reading step, a recognizing step and a generating step. In the reading step, a camera signal (140) is read from an interface to a camera (120), the camera signal (140) representing the camera image of the camera (120). In the recognizing step, the presence of the line (110) in the camera image is recognized using at least one anchor (145) having at least one base anchor point (150). In the generating step, a line regression (155) of the line (110) is generated using at least the base anchor point (150), the line regression (155) having a first regression point which was shifted relative to the base anchor point (150) and having at least one additional regression point which was generated relative to a previously generated regression point, in order to recognize the line (110).

Description

[0001] Description

[0002] title

[0003] Method and device for detecting a line in a camera image for a machine or a transmission system and detection system for a machine or a transmission system

[0004] State of the art

[0005] The approach is based on a device or method according to the class of the independent claims. The present approach also relates to a computer program.

[0006] Driver assistance systems, for example, use lane detection for lane assist.

[0007] Disclosure of the invention

[0008] Against this background, the approach presented here presents a method for detecting a line in a camera image for a machine or a transmission system, a device using this method, a corresponding computer program, and finally a detection system for a machine or a transmission system according to the main claims. The measures listed in the dependent claims enable advantageous developments and improvements of the method specified in the independent claim.

[0009] The advantages achievable with the presented approach are that a particularly uniform prediction of a line is enabled. A method for detecting a line in a camera image for a machine or a transmission system is presented. The method comprises a reading step, a detection step, and a generation step. In the reading step, a camera signal is read in from an interface to a camera, wherein the camera signal represents the camera image of the camera. In the detection step, a presence of the line in the camera image is detected using at least one anchor, according to one embodiment using a plurality of anchors, each with at least one base anchor point.In the generating step, a line regression of the line is generated using at least the base anchor point, the line regression comprising a first regression point shifted relative to the base anchor point and at least one further regression point generated relative to a previously generated regression point to detect the line.

[0010] This method can be implemented, for example, in software or hardware or in a mixed form of software and hardware, for example in a control unit.

[0011] The machine can be a fully or partially computer-controlled machine, for example a vehicle. The transmission system can be a system for transmitting information, such as a surveillance system or a medical, for example imaging, system. The camera can be a vehicle camera, for example a vehicle camera of an environment detection system of the vehicle. In the detection step, for example, a grid can be superimposed over the camera image, and the presence of the line can be detected if the line is detected in at least one cell of the grid. In the detection step, for example, a detected object can be classified in order to detect the line, for example using a feature extractor. The line can represent a road marking or an edge of an object. For example, the line can represent part of a lane.The anchor(s) can have a template or pattern that is searched for in the camera image. For example, different anchors can be used to detect different lines in the camera image. In the generation step, line regression can be generated to compensate for an offset. For example, line regression can be generated in the generation step if the base anchor point was detected in the center of a cell in the grid and / or on a line.For example, in the generation step, the line regression can be generated, wherein the first regression point is shifted relative to the base anchor point on an x-axis, and the first further regression point is generated shifted relative to the first regression point on the x-axis and a y-axis perpendicular to the x-axis, and subsequent further regression points are generated shifted relative to the previously generated regression point on the x-axis and the y-axis in order to detect the line. Such a method enables line regression, which can also be referred to as "regression representation," in lane detection by predicting the offsets / deviation of each point along the line / lane relative to the previous point and not relative to the anchor. This advantageously leads to a particularly smooth prediction of the line.The prediction describes the offset in the image coordinates that must be applied to the anchor to fit the line into the image.

[0012] The method may further comprise an outputting step in which, using the line regression, a control signal is output to control a function of the machine, in particular a vehicle, or of the transmission system. Thus, using the line regression, which can predict a course of the line, for example, in the form of a lane, a vehicle can be controlled, for example, to keep the vehicle within the lane.

[0013] According to one embodiment, in the detection step, the line can be detected in an edge region of the camera image using the at least one anchor and / or in a cell of the camera image divided into a plurality of cells using the anchor of the respective cell. For example, in the detection step, the line can be detected in the center of a cell. This allows the edge region of the camera image in individual cells to be searched for lines quickly and easily. It is also advantageous if, according to one embodiment, in the detection step, the line in the camera image is detected if at least the base anchor point of the anchor runs through the line. However, in the detection step, the line in the camera image can also be detected if only the base anchor point runs through the line or runs close to the line.For example, an anchorless method can be used to detect the line, where only the base anchor point is required as an anchor.

[0014] Alternatively, the line can be detected in the camera image during the detection step if the entire anchor passes through or near the line. This allows an anchor-based method to detect the line. An entire anchor is defined as an anchor that has more than one base anchor point, for example, a linear anchor.

[0015] For example, “near” means a deviation of a few pixels.

[0016] The approach presented here further provides a device configured to perform, control, or implement the steps of a variant of a method presented here in corresponding devices. This variant of the approach in the form of a device also allows the underlying problem to be solved quickly and efficiently.

[0017] For this purpose, the device can have at least one computing unit for processing signals or data, at least one memory unit for storing signals or data, at least one interface to a sensor or an actuator for reading sensor signals from the sensor or for outputting data or control signals to the actuator, and / or at least one communication interface for reading or outputting data embedded in a communication protocol. The computing unit can be, for example, a signal processor, a microcontroller, or the like, wherein the memory unit can be a flash memory, an EEPROM, or a magnetic storage unit.The communication interface can be designed to read in or output data wirelessly and / or wired, wherein a communication interface that can read in or output wired data can read this data, for example, electrically or optically from a corresponding data transmission line or output it to a corresponding data transmission line.

[0018] In this case, a device can be understood as an electrical device that processes sensor signals and outputs control and / or data signals depending on them. The device can have an interface, which can be implemented in hardware and / or software. In a hardware implementation, the interfaces can, for example, be part of a so-called system ASIC, which contains a wide variety of functions of the device. However, it is also possible for the interfaces to be separate integrated circuits or to consist at least partially of discrete components. In a software implementation, the interfaces can be software modules that are present, for example, on a microcontroller alongside other software modules.

[0019] Also advantageous is a computer program product or computer program with program code that can be stored on a machine-readable carrier or storage medium such as a semiconductor memory, a hard disk memory or an optical memory and is used to carry out, implement and / or control the steps of the method according to one of the embodiments described above, in particular when the program product or program is executed on a computer or a device.

[0020] A recognition system for a machine or a transmission system comprises the device described above and a camera for providing the camera signal. Such a recognition system can serve as a comprehensive system for the automated recognition of lines such as lanes, whereby the device allows lines such as lanes to be predicted particularly consistently. The recognition system can be part of a driver assistance system for a vehicle. Exemplary embodiments of the approach presented here are illustrated in the drawings and explained in more detail in the following description. It shows:

[0021] Fig. 1 is a schematic representation of a machine with a device according to an embodiment for detecting a line in a camera image;

[0022] Fig. 2 is a schematic representation of a line regression generated using a generating device of an apparatus according to an embodiment;

[0023] Fig. 3 is a schematic representation of a camera image for use with a device according to an embodiment; and

[0024] Fig. 4 is a flowchart of a method according to an embodiment for detecting a line in a camera image for a machine or a transmission system.

[0025] In the following description of advantageous embodiments of the present approach, the same or similar reference numerals are used for the elements shown in the various figures and having a similar effect, whereby a repeated description of these elements is omitted.

[0026] Fig. 1 shows a schematic representation of a machine 100 with a device 105 according to an embodiment for detecting a line 110 in a camera image.

[0027] For example only, the device 105 according to this exemplary embodiment is arranged on or in the machine 100, which is embodied here, for example, as a vehicle. According to one exemplary embodiment, the device 105 is implemented in a control unit of the machine 100. According to this exemplary embodiment, the machine 100 further comprises a camera 120, which is embodied here, for example, as an environment detection camera for detecting the environment of the machine 100. According to an alternative exemplary embodiment, the device 105 can be used accordingly with a transmission system.

[0028] The device 105 has a read-in interface 125, a recognition device 130, and a generation device 135. The read-in interface 125 is configured to read a camera signal 140 from an interface to the camera 120, wherein the camera signal 140 represents the camera image of the camera 120. The recognition device 130 is configured to detect the presence of the line 110 in the camera image using an anchor 145 with at least one base anchor point 150.The generation device 135 is configured to generate a line regression 155 of the line 110 using at least the base anchor point 150, wherein the line regression 155 comprises a first regression point shifted relative to the base anchor point and a second regression point generated relative to the first regression point, and corresponding to the second regression point, at least one further regression point generated relative to a previously generated regression point to detect the line 110. A more detailed representation of the line regression 155 is shown in Fig. 2.

[0029] According to this exemplary embodiment, the camera 120 is designed as a vehicle camera, for example, a vehicle camera of an environment detection system of the vehicle. The anchor 145 comprises a template or pattern that is searched for in the camera image by the recognition device 130 in order to recognize the line 110. The generation device 135 is designed according to this exemplary embodiment to generate the line regression 155, which compensates for an offset. The device 105 presented here enables the line regression 155, which can also be referred to as a "regression representation," during lane detection by predicting the offsets / deviations of each point along the line 110, which is a lane according to this exemplary embodiment, relative to the previous point and not relative to the anchor 145. This advantageously leads to a particularly consistent prediction of the line 110, see also Fig. 2.Optionally, the device 105 according to this exemplary embodiment further comprises an output device 160, which is configured to output a control signal 165 for controlling a function 170 of the machine 100, here the vehicle, or alternatively the transmission system, using the line regression 155. Thus, using the line regression 155, which can predict a course of the line 110, for example in the form of a lane, the machine 100 can be controlled, for example, the vehicle can be kept within the lane.

[0030] According to this embodiment, the detection device 130 is designed to detect the line 110 in an edge region of the camera image and / or in a cell of the camera image divided into a plurality of cells using the anchor 145, see also Fig. 3.

[0031] According to this exemplary embodiment, the recognition device 130 is designed to recognize the line 110 in the camera image if at least the base anchor point 150 of the anchor 145 runs through the line 110. Or, according to one exemplary embodiment, the recognition device 130 is designed to recognize the line 110 in the camera image if only the base anchor point 150 runs through the line 110 or runs in the vicinity of the line 110. Thus, according to one exemplary embodiment, an anchor-free method is used to recognize the line 110, in which only the base anchor point 150 is necessary as the anchor 145. According to an alternative exemplary embodiment, the recognition device 130 is designed to recognize the line 110 in the camera image if the entire anchor runs through the line 110 or runs in the vicinity of the line 110. Thus, according to another exemplary embodiment, an anchor-based method is used to recognize the line 110.A complete anchor 145 is understood to mean an anchor 145 that has more than the base anchor point 150, for example, a linear anchor. "Near" is understood to mean a deviation of, for example, a maximum of 5 millimeters.

[0032] Together with the camera 120, the device 105 presented here can also be referred to as a recognition system 175. According to one embodiment, the recognition system 175 is part of a driver assistance system 180 of the vehicle. The device 105 presented here enables delta regression / regression for lane detection.

[0033] Traditionally, deep object detectors used in computer vision are based on a concept called "anchors," meaning so-called "prior boxes" are used as prototypes for detection. In this context, the detectors predict displacements / offsets of the boxes relative to the respective anchors. Lane detection is a special case of object detection, where objects are not labeled by boxes but by line segments. Therefore, anchor-based object detection methods are used alongside traditional instance segmentation methods.

[0034] The device 105 presented here modifies the regression representation of the lane detection task by predicting the offsets / deviation of each point along the lane relative to the previous point, rather than relative to the anchor 145. The approach presented here can be formulated as either an anchor-based or an anchorless method. In both cases, only the base anchor point 150 of the anchors 145 is used (cell centers); the difference lies solely in the fitting step.

[0035] This formulation leads to a more consistent prediction of line 110 compared to a similarly possible regression representation, in which all regression points are generated relative to the anchor. Since the anchors 145 become redundant, this also results in a smaller neural network.

[0036] The device 105 presented here operates with data of the following types, which according to one embodiment are obtained by receiving digital images and / or videos, to calculate the control signal 165 for controlling a physical system, such as a computer-controlled machine 100, such as a robot, a vehicle, a household appliance, a power tool, a manufacturing machine, a personal assistant or an access control system, or a system for transmitting information, such as a monitoring system or a medical, for example, imaging, system.

[0037] The device 105 does this by detecting the presence of objects in the sensor data, particularly lanes or other types of line-based road markings.

[0038] The device 105 works with images, therefore an image recording in the form of the camera image must be available, which serves as input for the approach presented here.

[0039] Fig. 2 shows a schematic representation of a line regression 155 generated using a generation device of a device according to an embodiment. This may be the device described in Fig. 1.

[0040] The line regression 155 has the first regression point 200, which was shifted relative to the base anchor point 150 of the anchor. Starting from the first regression point 200, further regression points 210 were generated, each relative to a regression point 200, 210 generated directly beforehand. For example, the generation device according to this exemplary embodiment is designed to generate the line regression 155, wherein the first regression point 200 was shifted relative to the base anchor point 150 on an x-axis, the first further regression point 210 was generated shifted relative to the previously generated first regression point 200 on the x-axis and the y-axis, and the subsequent further regression points 210 were generated shifted relative to the previously generated further regression point 210 on the x-axis and the y-axis in order to detect the line 110.According to this exemplary embodiment, the line regression 155 has at least five further regression points 210, each of which was generated shifted relative to the previously generated regression point 200, 210 on the x-axis and the y-axis. In known regressions, the coordinates are predicted relative to the anchors, i.e., the prediction describes an offset in the image coordinates that must be applied to the anchor in order to fit the line into the image only on the x-axis. Unlike the known regression method, the approach presented here is implemented in such a way that not only the x-coordinates are predicted, but also the y-coordinates are not fixed. The approach presented here differs from known regression methods in that the references for the regression are not lines, but points 150, 200, 210, and for each cell, see Fig.3, by which a line was detected in the image, the regression coordinates are predicted relative to the previous coordinate. That is, for the first point 200 in the line, the prediction is relative to the base anchor point 150, for the first subsequent point 210, the prediction is relative to the first point 200, and so on. In contrast to known approaches, this regression is formulated by predicting not only the x-coordinate but also the y-coordinate.

[0041] The approach described here can be formulated in two ways, depending on how the classification goal is chosen.

[0042] First, an anchorless method is feasible, in which only the base anchor points 150 are used for both the line regression 155 and the classification. This means that a line 110 in the form of a ground truth line matches an anchor if it passes through or near its base anchor point 150, and that there is no "match" of an anchor line with a ground truth line, or

[0043] On the other hand, an anchor-based method is feasible, in which only the base anchor points 150 are used for line regression 155, but the entire anchor with more than one base anchor point 150 is used for classification. That is, a line 110 in the form of a ground truth line is matched with an anchor if the similarity between the anchor and the line 110 is high enough.

[0044] Unlike regression predictions, which describe a linear displacement relative to a linear anchor, where the application of the offset to the anchor coordinates results in a line prediction, the present approach changes this in that the anchor is only a point 150 and not a line, and the predictions describe a displacement relative to the previous point, as opposed to the corresponding anchor point. Thus, the line regression 155 does not represent a straight line overall.

[0045] Fig. 3 shows a schematic representation of a camera image 300 for use with a device according to an embodiment. This may be the device described in Fig. 1 or 2.

[0046] According to this exemplary embodiment, the recognition device of the device is designed to superimpose a grid 305 over the camera image 300 and to recognize the presence of the line 110 if the line 110 is detected in at least one cell 310, 311, 312 of the grid 305. According to this exemplary embodiment, the recognition device is designed to classify a detected object in order to recognize the line 110, for example, using a feature extractor. According to this exemplary embodiment, the line 110 represents a road marking or, according to an alternative embodiment, an edge of an object. For example, according to this exemplary embodiment, the line 110 represents part of a lane.For example, the generating device according to this embodiment is designed to generate the line regression when the base anchor point was detected in a cell center 315 of a cell 310, 311, 312 of the grid 305 on a line 110.

[0047] In summary, according to one embodiment, corresponding to the anchor 145 in the cell 310 or merely the base anchor point of the anchor 145 in the cell 310, further anchors are defined in cells 310, 311, 312 along the boundaries of the image 300, and the classification output predicts for each cell 310, 311, 312 whether or not the cell 310, 311, 312 contains a corresponding line 110, for example, a trace, that is similar to one of the anchors 145 on that cell 310, 311, 312. In Fig. 3, for the sake of clarity, only one corresponding anchor 145 is drawn through one of the cells 310. In other words, according to one embodiment, a network predicts for each cell 310, 311, 312 whether there is a line 110 in the cell 310, 311, 312 that is similar to an anchor 145 associated with the respective cell 310, 311, 312.According to this exemplary embodiment, the grid 305 comprises eight uniformly sized cells 310, 311, 312 arranged one above the other along the edge of two opposite image sides of the camera image 300. Along the edge of a bottom side of the camera image 300, the grid 305 according to this exemplary embodiment has twelve adjacently arranged uniformly sized cells 310, 311, 312. Cells 310 represent a positive prediction of a line 110, cells 311 represent a negative prediction, and cells 312 represent a borderline case.

[0048] Fig. 4 shows a flowchart of a method 400 according to an embodiment for detecting a line in a camera image for a machine or a transmission system. This may be a method 400 that can be executed by one of the devices described with reference to the preceding figures.

[0049] The method 400 comprises a reading step 405, a detection step 410, and a generation step 415. In the reading step 405, a camera signal is read from an interface to a camera, the camera signal representing the camera image of the camera. In the detection step 410, the presence of the line in the camera image is detected using an anchor with at least one base anchor point. In the generation step 415, a line regression of the line is generated using at least the base anchor point, the line regression having a first regression point that has been shifted relative to the base anchor point and at least one further regression point that has been generated relative to a previously generated regression point in order to detect the line.

[0050] If multiple anchors are present, they can be used to detect the line or to detect different lines in a corresponding manner. Optionally, the method 400 according to this exemplary embodiment further comprises an output step 420. In the output step 420, a control signal for controlling a function of the machine, in particular a vehicle, or the transmission system is output using line regression.

Claims

Claims 1. A method (400) for detecting a line (110) in a camera image (300) for a machine (100) or for a transmission system, the method (400) comprising the following steps: Reading (405) a camera signal (140) from an interface to a camera (120), wherein the camera signal (140) represents the camera image (300) of the camera (120); Detecting (410) a presence of the line (110) in the camera image (300) using at least one anchor (145) with at least one base anchor point (150); and Generating (415) a line regression (155) of the line (110) using at least the base anchor point (150), wherein the line regression (155) comprises a first regression point (200) that has been moved relative to the base anchor point (150) and at least one further regression point (210) that has been generated relative to a previously generated regression point to detect the line (110).

2. Method (400) according to one of the preceding claims, comprising an outputting step (420) in which a control signal (165) for controlling a function (170) of the machine (100), in particular of a vehicle, or of the transmission system is output using the line regression (155).

3. Method (400) according to one of the preceding claims, wherein in the step (410) of recognizing the line (110) in an edge region of the camera image (300) using the at least an anchor (145) is detected and / or is detected in a cell (310) of the camera image (300) divided into a plurality of cells (310, 311, 312) using the anchor (145) of the respective cell (310). Method (400) according to one of the preceding claims, wherein in the detection step (410) the line (110) is detected in the camera image (300) if at least the base anchor point (150) of the anchor (145) runs through the line (110). Method (400) according to claim 4, wherein in the detection step (410) the line (110) is detected in the camera image (300) if the entire anchor runs through the line (110). Method (400) according to one of the preceding claims, wherein the line (110) represents a lane marking or an edge of an object.A device (105) configured to execute and / or control the steps (405, 410, 415, 420) of the method (400) according to one of the preceding claims in corresponding units (125, 130, 135, 160). A recognition system (175) for a machine (100) or a transmission system, wherein the recognition system (175) comprises a device (105) according to claim 7 and a camera (120) for providing the camera signal (140). A computer program configured to execute and / or control the steps (405, 410, 415, 420) of the method (400) according to one of claims 1 to 6. A machine-readable storage medium on which the computer program according to claim 9 is stored.

Images